This would be a kernel problem or a module problem.
Linux does not handle powering down/up USB devices all that well.
an application may load a specific module to better handle the power requirements of a device.
otherwise if the kernel does not know exactly what device has been connected it may take the safe option an use a low power generic device, maybe like a keyboard.
USB 2.5inch HDD need a max of 520mA so they need 2 USB dongles for power (only one is data).
another possibility is it is the device (phone or speakers) being safe since it is the device that draws power, it is not pushed in. this could still be kernel related as the phone needs some signal to draw max power (amps).
Now where was i going? Oh yes, crazy!