who's your ISP?
i use comcast, and i'm seriously beginning to wonder if they're throttling updates, as an unintended consequence of trying to stop illegal torrents - or of throttling their network in general, just to be dicks. i don't download anything major but geek stuff - no music or movies or any kind of bittorrent stuff but stuff in the repos. and i have no speed issues at all, except when getting updates, & etc.
apt-get uses some kind of torrent, doesn't it?
i can play HD video off youtube all day long without a stutter, no problems with any other large files.
but as soon as i start apt-get, my speed goes down to 30-50kbps. it's very annoying.