I did some more test with my hard drive and it seems a lot faster than it used to be.
It took me only 30 sec to transfer almost 1Go (26MB/sec) on my fat32 partition.
This may be due to my new bios setting (legacy option disabled) I will perform some more test and let you know.
I thought a USB stick and a USB hard drive would have the same transfer rate, but I was wrong.
My USB stick is definitly the cheapest one on the market therefore it will never achieve fast transfer. Thanks to Quantumboredom for pointing this out.
I guess everything is fine now, thank you all for your help