Mint13-14 Latest Kernel Version Causing EXT4-fs Errors

[viking777]

Whilst talking about Manjaro, I run that distro as well, using a release candidate of the 3.8 kernel and I see no evidence of the error in the log files for that.

[viking777]

Something else strange I noticed about this, I have syslog entries going back to Jan 19th but none of them have any evidence of this error until it first occurred on Jan 22nd after which it is happening frequently (looks like every boot). The other strange thing about it is that on each occasion the error occurs the same two inodes are deleted every time - inode 30 and inode 57. Others are deleted as well (and these vary) but those two occur every time

I am off to do a file system check.

Absolutely nothing from fsck. But now I see something even more worrying.

Because I hate boot splash screens and always disable them, I can see the boot messages as they are printed. During my last boot of Mint I saw a whole page of 'orphaned inodes' being deleted, yet, this is the worrying thing, not one of them is recorded in syslog!. The boot messages go by too quickly to read, but there must have been at least 50 of them maybe more. This makes you wonder how bad the problem really is and exactly what it is doing to your file system.

I am going back to the older kernel now for sure.

[karlchen]

Hello, AlbertP.

Here you can file a bug against Ubuntu's Linux kernel: https://bugs.launchpad.net/ubuntu/+sour ... x/+filebug

Hm, filing a bug against the Ubuntu kernel does not seem to be too trivial in case you found it on a genuine Linux Mint 14 system. E.g. the apport-bug programme refuses to do its job because apport-bug is not a genuine LinuxMint software. Hm, I guess what it really complains about is that ls-release states the system is LinuxMint, but apport-bug only wants to check and report on genuine Ubuntu systems.

In short words, I will have to boot my Ubuntu 12.04.1 x64 with kernel 3.2.0-36 and start all over again running apport-bug. This should yield an acceptable bug report for Ubuntu Linux kernel 3.2.0-36.
Wonder how I can file the same bug against Ubuntu kernel 3.5.0-22 without replacing Linux Mint 14 by Ubuntu 12.10 first.
I am afraid in the end I will have to replace Mint 14 by Ubuntu 12.10 in order to be able to file an acceptable bug report for Ubuntu Linux kernel 3.5.0-22.

Kind regards,
Karl

[karlchen]

Hello, viking777.

With respect to Mint 14, kernel 3.5.0-x, things seem to get worse here instead of better, even though I downgraded from K3.5.0-22 to 3.5.0-21 and finally to K3.5.0-19. They all seem to want to recover the read-only filesystem on every startup.
With respect to Mint 13 and Ubuntu 12.04.1, going back from K3.2.0-36 to K.3.2.0-35, however, seems to have made the orphaned inodes go away for good.

Note that there is one machine where
+ Mint 13, K3.2.0-35, no more EXT-fs complaints after downgrading, and
+ Mint 14, K3.5.0-22 -> K3.5.0-21 -> K3.5.0-19, EXT-fs error messages on every reboot
live on the same machine, just on different disk partitions.

Kind regards,
Karl
--
Linux Mint 14: inxi -Fx:

Code: Select all

System:    Host: unimatrix0 Kernel: 3.5.0-22-generic i686 (32 bit, gcc: 4.7.2) Desktop: Gnome Distro: Linux Mint 14 Nadia
Machine:   Mobo: N/A model: N/A Bios: American Megatrends version: 080015 date: 10/09/2009
CPU:       Dual core Intel Atom CPU 330 (-HT-MCP-) cache: 512 KB flags: (lm nx sse sse2 sse3 ssse3) bmips: 6399.92
           Clock Speeds: 1: 1599.984 MHz 2: 1599.984 MHz 3: 1599.984 MHz 4: 1599.984 MHz
Graphics:  Card: NVIDIA ION VGA bus-ID: 03:00.0
           X.Org: 1.13.0 drivers: (unloaded: fbdev,vesa,nouveau) FAILED: nvidia Resolution: 1280x1024@60.0hz
           GLX Renderer: ION/integrated/SSE2 GLX Version: 3.3.0 NVIDIA 304.64 Direct Rendering: Yes
Audio:     Card: NVIDIA MCP79 High Definition Audio driver: snd_hda_intel bus-ID: 00:08.0 Sound: ALSA ver: 1.0.25
Network:   Card-1: Atheros AR9285 Wireless Network Adapter (PCI-Express) driver: ath9k bus-ID: 04:00.0
           IF: wlan0 state: down mac: 00:25:d3:eb:5e:c8
           Card-2: NVIDIA MCP79 Ethernet driver: forcedeth port: d080 bus-ID: 00:0a.0
           IF: eth0 state: up speed: 100 Mbps duplex: full mac: 00:01:2e:2b:26:c2
Drives:    HDD Total Size: 167.8GB (5.4% used) 1: id: /dev/sda model: SAMSUNG_HM160HI size: 160.0GB
           2: USB id: /dev/sdb model: Premium size: 7.8GB
Partition: ID: / size: 48G used: 5.3G (12%) fs: ext4 ID: swap-1 size: 5.42GB used: 0.00GB (0%) fs: swap
RAID:      No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors:   System Temperatures: cpu: 79.0C mobo: N/A gpu: 0.0:56C
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 160 Uptime: 40 min Memory: 362.2/1761.3MB Runlevel: 2 Gcc sys: 4.7.2 Client: Shell inxi: 1.8.4

Linux Mint 14 Upgrade to K3.5.0-22:

Code: Select all

Commit Log for Sat Jan 19 12:57:55 2013

Die folgenden Pakete wurden aktualisiert:
linux-generic (3.5.0.17.19) to 3.5.0.22.28
linux-headers-generic (3.5.0.17.19) to 3.5.0.22.28
linux-headers-generic-pae (3.5.0.17.19) to 3.5.0.22.28
linux-image-generic (3.5.0.17.19) to 3.5.0.22.28
linux-libc-dev (3.5.0-17.28) to 3.5.0-22.34

Die folgenden Pakete wurden installiert:
linux-headers-3.5.0-22 (3.5.0-22.34)
linux-headers-3.5.0-22-generic (3.5.0-22.34)
linux-image-3.5.0-22-generic (3.5.0-22.34)
linux-image-extra-3.5.0-22-generic (3.5.0-22.34)

Linux Mint 14: grep "EXT-fs" syslog.1 syslog:

Code: Select all

syslog.1:Jan 19 11:30:11 unimatrix0 kernel: [    1.721742] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 11:30:11 unimatrix0 kernel: [    4.632774] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    1.674595] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    1.674607] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    2.792754] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    2.802863] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 262167
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    2.813109] EXT4-fs (sda6): 1 orphan inode deleted
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    2.813115] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    2.900072] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 12:16:03 unimatrix0 kernel: [    5.604054] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 12:51:47 unimatrix0 kernel: [    1.676858] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 12:51:47 unimatrix0 kernel: [    1.676867] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 12:51:47 unimatrix0 kernel: [    3.680198] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 12:51:47 unimatrix0 kernel: [    3.688199] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 12:51:47 unimatrix0 kernel: [   20.579759] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 13:59:00 unimatrix0 kernel: [    1.654648] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 13:59:00 unimatrix0 kernel: [    1.654657] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 13:59:00 unimatrix0 kernel: [    3.420429] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 13:59:00 unimatrix0 kernel: [    3.428535] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 13:59:00 unimatrix0 kernel: [   32.468828] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    1.695103] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    1.695114] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715672] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715695] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786610
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715763] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786514
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715879] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788262
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715922] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788307
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.715963] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788293
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.736286] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788199
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.736346] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786861
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.746478] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 787669
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.746540] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788038
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.746582] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788092
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.746622] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788090
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762685] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786591
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762721] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786486
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762766] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788069
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762809] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788058
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762918] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 788037
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.762959] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 787743
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779277] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 787744
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779321] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 787767
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779363] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786864
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779449] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786669
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779558] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786621
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779631] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786614
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779673] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1310730
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779711] EXT4-fs (sda6): 24 orphan inodes deleted
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.779716] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    2.956403] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 20:46:59 unimatrix0 kernel: [    6.707078] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    1.667823] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    1.667831] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159002] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159031] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051400
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159171] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051392
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159196] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051399
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159240] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786561
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159267] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786486
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159355] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786613
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159463] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786612
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159498] EXT4-fs (sda6): 7 orphan inodes deleted
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.159503] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [    3.267253] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 21:08:11 unimatrix0 kernel: [   30.343105] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    1.679298] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    1.679310] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.460910] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.460931] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786614
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.461000] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786476
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.461104] EXT4-fs (sda6): 2 orphan inodes deleted
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.461110] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [    3.547174] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 22:03:26 unimatrix0 kernel: [   18.962632] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    1.701820] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    1.701832] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858339] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858363] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051402
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858508] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051401
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858533] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 1051392
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858577] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786561
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858605] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786476
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858687] EXT4-fs (sda6): 5 orphan inodes deleted
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.858693] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [    3.955665] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 23:28:59 unimatrix0 kernel: [   19.756449] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    1.630520] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    1.630529] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.805477] EXT4-fs (sda6): orphan cleanup on readonly fs
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.805506] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786615
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.805572] EXT4-fs (sda6): ext4_orphan_cleanup: deleting unreferenced inode 786476
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.805676] EXT4-fs (sda6): 2 orphan inodes deleted
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.805681] EXT4-fs (sda6): recovery complete
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [    2.891672] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 19 23:52:26 unimatrix0 kernel: [   18.394362] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 22 21:12:46 unimatrix0 kernel: [    1.813835] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 22 21:12:46 unimatrix0 kernel: [   17.855693] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 22 21:27:34 unimatrix0 kernel: [    1.741682] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 22 21:27:34 unimatrix0 kernel: [   17.232514] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 22 22:18:18 unimatrix0 kernel: [    1.708217] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 22 22:18:18 unimatrix0 kernel: [   17.750726] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 22 22:31:16 unimatrix0 kernel: [    1.854258] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 22 22:31:16 unimatrix0 kernel: [   17.797095] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Jan 23 22:11:13 unimatrix0 kernel: [    1.684727] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Jan 23 22:11:13 unimatrix0 kernel: [   17.428563] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog:Jan 23 22:37:11 unimatrix0 kernel: [    1.762216] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Jan 23 22:37:11 unimatrix0 kernel: [    1.762225] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Jan 23 22:37:11 unimatrix0 kernel: [    2.454815] EXT4-fs (sda6): recovery complete
syslog:Jan 23 22:37:11 unimatrix0 kernel: [    2.464101] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 23 22:37:11 unimatrix0 kernel: [   18.453886] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog:Jan 23 22:52:50 unimatrix0 kernel: [    1.720077] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Jan 23 22:52:50 unimatrix0 kernel: [    1.720088] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Jan 23 22:52:50 unimatrix0 kernel: [    4.394645] EXT4-fs (sda6): recovery complete
syslog:Jan 23 22:52:50 unimatrix0 kernel: [    4.403857] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 23 22:52:50 unimatrix0 kernel: [   20.029038] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    1.670490] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    1.670516] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    2.646816] EXT4-fs (sda6): recovery complete
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    2.656118] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 23 22:57:15 unimatrix0 kernel: [   18.258962] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro

Linux Mint 14 palimpsest controller details (actually got from Mint 13, because "gnome disks" does no longer display controller details):

Linux Mint 14 palimpsest controller details (actually got from Mint 13, because "gnome disks" does no longer display controller details)

Mint14_through_mint13_diskutility.png (39.11 KiB) Viewed 997 times

[viking777]

Well I haven't had anything on 3.5.0-21 YET (mint or ubuntu). But if you remember, I was a couple of days behind you getting the error on 3.5.0-22 as well, so I still watch out for it on every boot.

[karlchen]

Good morning, viking777.

Oh well. As we still do not know what the root cause is, in fact it might be a combination of two or more factors, it is imaginable that on one machine going back to K3.5.0-21 does make the problem go away, whereas on another machine the problem persists.
Though on my Mint14 system there seems to be a difference:
Running on K3.5.0-22 there were EXT4-fs error messages about orphaned inodes. Running on K3.5.0-21 or K3.5.0-19 the EXT4-fs messages on every restart only state:

Code: Select all

syslog:Jan 23 22:57:15 unimatrix0 kernel: [    1.670490] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    1.670516] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    2.646816] EXT4-fs (sda6): recovery complete
syslog:Jan 23 22:57:15 unimatrix0 kernel: [    2.656118] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 23 22:57:15 unimatrix0 kernel: [   18.258962] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro

The last 2 lines are normal and do not inidcate any problem. The first 3 lines should not be there. At least, no more complaints about cleaning up unreferenced inodes.

Kind regards,
Karl

[viking777]

Morning.

I get that alright:

Code: Select all

cat /var/log/syslog | grep -i recovery
Jan 24 11:24:41 fujitsu kernel: [    1.776901] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
Jan 24 11:24:41 fujitsu kernel: [    1.777752] EXT4-fs (sda6): write access will be enabled during recovery
Jan 24 11:24:41 fujitsu kernel: [    2.982859] EXT4-fs (sda6): recovery complete

That after my last reboot of MInt 14 -21 kernel. Strangely enough that is not present on Ubuntu -21 kernel just yet, but I have only shut it down once this morning so far, maybe that makes a difference. That error doesn't bother me at all, it shouldn't occur for sure but if it is recovering a lost or broken journal, then that is what journalling file systems are made to do. Deleting inodes on every start up is something very different and far more sinister (especially the same inodes every time! ).

I don't get shutdown messages displayed in Mint (I have never worked out why) but I do get them in Ubuntu and almost every shutdown I am getting messages something like this:

Code: Select all

Asking all remaining processes to stop .............FAIL
Killing all remaining processes ..............OK


I just wonder if this has anything to do with the orphaned inodes error?

[AlbertP]

I'm just cloning the git repository of the Ubuntu kernels to check if there has been any ext4 or ahci change between these kernel versions.

[AlbertP]

I can't find a commit affecting the ext4 filesystem or the ahci driver between 3.2.0-35 and 3.2.0-36. Nothing appears to have changed in fs/ext4/ in the source as well as in drivers/ata/ahci.c.

[karlchen]

Hello, AlbertP.

Thanks a lot for all your investigation. - Sorry for having been away for 2 days. Sometimes your job will simply eat you up, uhm, eat me up in this case.
Well, without having digged into the kernel as deep as you did, but based on a few years working in IT, I could imagine that the root cause of the EXT4-fs errors, created during shutdown, may have been there for quite a while. But some change inside K3.2.0-36 and K3.5.0-22 now trigger the error, without being to be blamed themselves.

Anyway, back on genuine Ubuntu 12.04.1 x64, running K3.2.0-36, it will be possible to create a Ubuntu bug report and upload it to Ubuntu Launchpad, without being rejected because a too Minty desktop environment has been installed.

Cf. Latest Kernel Version 3.2.0-36 Causing EXT4-fs Errors

Cheers,
Karl

[viking777]

Thanks Karl. I have added my report to that as well.

[karlchen]

Hello, viking777.

Great. Thanks a lot for adding your report. - Let's see what the outcome of the reports will be ....

Cheers,
Karl

[karlchen]

Hello, viking777.

Update on the Launchpad bug report based on my tests:

• No real improvement seen on Ubuntu 12.10/Mint 14 32-bit using proposed kernel 3.5.0-23: cf. comment #10 here.
• There seems to be hope for Ubuntu 12.04.1/Mint 13 x64 using proposed kernel 3.2.0-37, however: cf. comment #11 here.

Kind regards,
Karl

[viking777]

Thanks for updating me here karlchen. I thought I was subscribed to that bug thread, but I wasn't (I am now). I added to the thread as you are probably already aware.

[karlchen]

Good evening, viking777.

Yes, I read your posts.
The behaviour which you see using kernel 3.5.x seems to be the same that I see using the 3.5x kernels on Mint 14, 32-bit, as well.
In my Mint 14 syslog files I found such lines like the one you posted

Jan 29 11:04:35 fujitsu kernel: [ 1.827272] EXT4-fs (sda2): INFO: recovery required on readonly filesystem

too.
I used Mint 13, 32-bit, on /dev/sda1, to run a forced fsck in verbose mode on the Mint14 partition, which is /dev/sda6. The verbose fsck mode revealed this:

Code: Select all

sudo fsck -v -f /dev/sda6
fsck von util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
lm14cinn: stelle das Journal wieder her <============== EXT4 file system journal will be recovered
Durchgang 1: Prüfe Inodes, Blocks, und Größen
Durchgang 2: Prüfe Verzeichnis Struktur
Durchgang 3: Prüfe Verzeichnis Verknüpfungen
Durchgang 4: Überprüfe die Referenzzähler
Durchgang 5: Überprüfe Gruppe Zusammenfassung
Die Anzahl freier Blöcke ist falsch (10905020, gezählt=10904942). <===== incorrect free block count
Repariere<j>? ja

Die Anzahl freier Inodes ist falsch (2864274, gezählt=2864270). <======= incorrect free inode count
Repariere<j>? ja


lm14cinn: ***** DATEISYSTEM WURDE VERÄNDERT ***** <==== filesystem has been changed

281458 Inodes sind in Benutzung (8.95%)
     155 nicht zusammenhängende Dateien (0.1%)
     178 nicht zusammenhängende Verzeichnisse (0.1%)
         # von Inodes mit ind/dind/tind Blöcken: 0/0/0
         Erweiterungstiefe Histogramm: 246374/41
 1677970 Blöcke werden benutzt (13.34%)
0 ungültige Blöcke
       1 große Datei

  207356 reguläre Dateien
   36374 Verzeichnisse
      58 zeichenorientierte Gerätedateien
      25 Blockgerätedateien
       2 Fifos
      20 Verknüpfungen
   37605 symbolische Verknüpfungen (34921 schnelle symbolische Verknüpfungen)
      29 Sockets
--------
  281469 Dateien

Sorry for the German text, translated the 3 relevant pieces into English.
The free block count and the free inode count will not be wrong on every reboot. But the EXT4 journal seems to be needing recovery on every single reboot.

This problem does not happen oon Ubuntu 12.04.1 x64, running on kernel 3.2.0-37. It does not want to recover the EXT4 journal on every reboot so far.

Cheers,
Karl

[viking777]

Since switching to the 3.5.0-23 kernel on Ubuntu 12.10 the 'orphan' errors have gone completely.

The 'recovery' errors have returned this morning after 12 extra mounts.

I will carry out another fsck and post this information on the bug report later.

[karlchen]

Hello, viking777. Hello, AlbertP.

<Speculation on>

As you, AlbertP, reported there were not any changes in the kernel sources of K3.2.0-36 which were directly related to AHCI or the EXT4 filesystem, I suspect the root cause is less obvious. It might be something which causes a minimal slowdown somewhere in the last phase of the shutdown right before the root file system gets umounted. Under unlucky circumstances something now takes slightly more time. The shutdown procedure, however, is like a stubborn mule. It does not check whether the root filesystem has been umounted, instead it goes straight ahead and shuts down the system (or reboots).

In short words, personally I suspect the root cause to be a timing issue. Yet, this is just an assumption, based on the symptoms.

</Speculation off>

By the way, no speculation, but the result of another

Code: Select all

grep "EXT4-fs" /var/log/syslog

So far Unbuntu 12.04.1 x64 running on K3.2.0-37 must have shutdown cleanly every time. No recovery of the EXT4 journal, no orphaned inodes. (Keep my fingers crossed.)

Kind regards,
Karl

[karlchen]

Have just received the mintUpdater notification on my Mint 13 xfce 32-bit that kernel 3.2.0-37 were waiting in the regular repositories. - Hm, I guess I will allow Ubuntu 12.04.1 x64 to be my only system running K3.2.0-37 for a few more days ...

Karl

[viking777]

I updated Mint14 to the 3.5.0-23 kernel this morning. The 'recovery' message occurred after just 1 mount!!

No orphans yet though, and the recovery error was cleared by running fsck even though it didn't repair anything.

So for me it is exactly the same behaviour as ubuntu so far.

[karlchen]

Hello, viking777.

Current state from my point of view:

• Ubuntu 12.04.01/12.04.02 x86/x64, running kernel 3.2.0-37: EXT4-fs error messages have gone away and not returned, yet. So, the problem seems to have been fixed.
• Ubuntu 12.10 x86/x64, running kernel 3.5.0-23: No orphaned inodes for a couple of days. But EXT4 journal gets recovered on every reboot. So, the problem has not been completely fixed, yet.

Kind regards,
Karl