Sounds like a BIOS thing.
First, shut down the computer. Then remove the USB stick.
Reboot, and go into the BIOS (more than likely F2), and make sure it's set to boot from your HDD before your removable/USB devices.
If that doesn't solve it, you may have to re-setup GRUB.
"People should better think of their computing devices as facilities lended by the DHS." -Berke Durak, Linux wipe (secure file deletion) development.