ZFS hangs up

Questions about applications and software
Forum rules
Before you post please read how to get help
Post Reply
mintybits
Level 6
Level 6
Posts: 1123
Joined: Fri Jan 27, 2012 5:09 pm

ZFS hangs up

Post by mintybits » Tue Feb 06, 2018 6:20 pm

Has anyone else seen this?
I have a 2 drive ZFS mirror with WD Green 3TB drives. A few days ago my whole system became terribly slow. Today I discovered one of the drives has very slow performance. I ran the Disks program and did a read benchmark...it struggled to do 2MB/s.
Fine, I thought, I'll tell ZFS to put the drive offline so that my system would work quickly again and I could do a backup. But when I issued the zpool offline command on the affected drive, zfs never came back. I could not even do a zpool status nor zfs list. Each command just hung up. After waiting 15 minutes I decided to reboot...Linux Mint would not complete its shutdown and also hung up. Eventually, I resorted to pulling the plug. I disconnected the bad drive and the system rebooted ok in a degraded raid state. I'm now backing it up.

It is extremely rare in my experience for Linux to hang up like this.

Has anyone else seen this?
Thx.

User avatar
catweazel
Level 17
Level 17
Posts: 7761
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: ZFS hangs up

Post by catweazel » Wed Feb 07, 2018 12:51 am

Has anyone else seen this?
What you're experiencing is typical of both dying hardware and disk corruption. There's actually no telling what kind of symptoms might be experienced in such a situation, except that experience will tell you that something has gone belly up.
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

mintybits
Level 6
Level 6
Posts: 1123
Joined: Fri Jan 27, 2012 5:09 pm

Re: ZFS hangs up

Post by mintybits » Wed Feb 07, 2018 12:29 pm

The voice of experience. :)
I am sure Oracle didn't overlook this when designing ZFS so I reckon it must be something to do with the Linux variant and the OS. The hanging up that is. Imagine if this happened in a data centre.
My Linux OS is running off a separate, SSD.

mintybits
Level 6
Level 6
Posts: 1123
Joined: Fri Jan 27, 2012 5:09 pm

Re: ZFS hangs up

Post by mintybits » Fri Feb 09, 2018 5:22 am

I discovered something interesting that I wasn't aware of. Both of my WD Green drives, which have been in my ZFS mirror for a year or so, have huge load/unload cycle counts: 475000 and 525000. I used "hdparm -B 255" and "hdparm -S 0" to try to disable all power saving activity but the good drive still racked up 335 in 9 hours (about 1 every 100 seconds). Some research suggests Green drives are only warranted for 300000 cycles. The new Red drive I installed yesterday did only 3 cycles in the same time.
A fix I found http://www.storagereview.com/node/1031 describes a WD tool WDIDLE3 that can disable the Green's power saving. I'll try this and post the result.

mintybits
Level 6
Level 6
Posts: 1123
Joined: Fri Jan 27, 2012 5:09 pm

Re: ZFS hangs up

Post by mintybits » Fri Feb 09, 2018 8:50 am

Fixed it.
I downloaded and compiled the idle3 tool here: http://idle3-tools.sourceforge.net/
I found my WD Green 3TB was set to 8s timeout. I set it to 36 mins and power-cycled the drive.
Now it is not clocking up the load/unload cycles.
I imagine my other drive died because of an excess of cycles. Which would be a pointless waste of a drive.

Clearly this has been a known problem for at least 7 years. I didn't know. So that's a problem in itself.
The other problem was the way ZFS and my OS just hanged up when the drive failed.

I'll bet a lot of naive people like me use WD Green drives on their Linux systems. How can they find out before trashing a perfectly good drive?

BTW my new WD Red has a default timeout of 300s.

Post Reply

Return to “Software & Applications”