Well, depending on how much noise your hard drive makes, you can probably hear it. Maybe a slight click and a rising frequency whirring sound as the motor spins up the drive.
I'll tell you what I am thinking. As a power saving feature, if the drive is not needed for a period of time the head is parked and the power is cut off to the drive. As your ram fills up or the kernel needs information from the drive, power is reapplied to the drive and it spins back up so it can be accessed. The system can, to some extent, forecast when the drive will be needed and spins up or restarts the drive so it can be accessed. It is possible that the ata controller is not anticipating the need for spin up properly, or that the drive is slow spinning up because of a drive motor going bad or a spindle dragging in a bearing. The cpu then has to wait for the drive to spin up before it can move information into or out of ram. If I had to guess, I would say it is probably a drive problem. It is more likely that than the controller.
Of course I could be way out in left field too.