is it wireless, can a neighbor be uploading stuff through your network? Check your router's logs and see if there is anyone else that's on/been on your network other than you and your family.
Also, I've rarely seen this but sometimes the wireless router will wig out and you'll get a roller coaster ride in download/upload speed.
Go to
http://www.speakeasy.net/speedtest/ and take the test. If it changes drastically (like going from 1500kbps to 3500kbps, then down to 500kbps) then you have a wireless router issue. There are some connection optimizers that can solve this issue, but i don't know which ones are good.