Mint 18.3 64 bit AMD E2+Radeon R2 filesystem corruption soon after install. HP 245 G6 laptop.

[ewblen]

Greetings,

Installing Mint 18.3 Cinnamon 64 bit on to a HP 245 G6 Notebook PC https://support.hp.com/hk-en/document/c05480842
It is the version with AMD E2-9000e processor and Radeon R2 GPU.

The install goes fine into free space created from shrunk Win 10 partition. UEFI boot as usual these days,

Within a few minutes of starting to use the installation, the GUI completely freezes. Nothing works to reboot except magic sysrq.
After reboot, the system drops into the initrd busybox. The root ext4 file system is extensively damaged.

I have redone the installation a few times. One time I managed to install ssh-server, and could connect to the machine while the gui was frozen.
There were a lot of errors in the system log. Sadly I couldn't capture that log before the SSH session also hung.

I have seen some suggestion that there is some problem to do with AMD iommu or the GPU.

* Questions:
1) Is a known problem?
2) Is it worth trying a distro with newer kernel (don't advise me to upgrade, as it is hit and miss whether the system will crash and burn before an upgrade can be downloaded and applied)
3) Any other ideas for debugging or documenting this problem.
4) Is there a more appropriate forum for this info/question

thanks

Eliot

[zcot]

Did you verify the installation media in the first place? You do say the installation goes fine but is that part verified?

Other than that how about put up some info to at least get a good idea of the basics. You can do terminal command: upload-system-info

[ewblen]

Yes I have verified the download.

I now have a 700K log file gathered over ssh.

After a while, get this kind of thing...
[ 1721.024680] AMD-Vi: Completion-Wait loop timed out
[ 1721.204878] AMD-Vi: Completion-Wait loop timed out
[ 1721.392804] AMD-Vi: Completion-Wait loop timed out
[ 1721.580857] AMD-Vi: Completion-Wait loop timed out
[ 1721.768884] AMD-Vi: Completion-Wait loop timed out
[ 1721.852795] AMD-Vi: Event logged [
[ 1721.852802] IOTLB_INV_TIMEOUT device=00:01.0 address=0x00000001f59afce0]

Then the kernel stack dumps start happening...

Here is the end of it before the machine froze *again*

[ 1931.719383] ------------[ cut here ]------------
[ 1931.719390] WARNING: CPU: 1 PID: 15490 at /build/linux-hwe-lyR8gz/linux-hwe-4.10.0/drivers/iommu/amd_iommu.c:1256 __domain_flush_pages+0x1de/0x200
[ 1931.719392] Modules linked in: bnep binfmt_misc nls_iso8859_1 edac_mce_amd edac_core kvm irqbypass hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc uvcvideo aesni_intel aes_x86_64 crypto_simd videobuf2_vmalloc glue_helper videobuf2_memops snd_hda_codec_hdmi cryptd snd_hda_codec_realtek snd_hda_codec_generic videobuf2_v4l2 snd_hda_intel videobuf2_core snd_hda_codec videodev snd_hda_core media snd_hwdep btusb joydev btrtl btbcm snd_pcm btintel snd_seq_midi snd_seq_midi_event bluetooth snd_rawmidi input_leds serio_raw i2c_piix4 fam15h_power snd_seq snd_seq_device snd_timer snd soundcore shpchp mac_hid hp_wireless i2c_designware_platform i2c_designware_core rfkill_gpio tpm_crb 8250_dw parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log amdkfd
[ 1931.719465] amd_iommu_v2 amdgpu i2c_algo_bit ttm psmouse drm_kms_helper syscopyarea sysfillrect ahci libahci sysimgblt fb_sys_fops r8169 drm mii wmi i2c_hid hid fjes video
[ 1931.719490] CPU: 1 PID: 15490 Comm: gdbus Tainted: G W 4.10.0-38-generic #42~16.04.1-Ubuntu
[ 1931.719492] Hardware name: HP HP 245 G6 Notebook PC/8324, BIOS F.20 11/07/2017
[ 1931.719493] Call Trace:
[ 1931.719496] <IRQ>
[ 1931.719501] dump_stack+0x63/0x90
[ 1931.719505] __warn+0xcb/0xf0
[ 1931.719509] warn_slowpath_null+0x1d/0x20
[ 1931.719513] __domain_flush_pages+0x1de/0x200
[ 1931.719517] ? queue_flush_all+0x90/0x90
[ 1931.719520] __queue_flush+0x46/0xc0
[ 1931.719524] queue_flush_all+0x77/0x90
[ 1931.719527] queue_flush_timeout+0x18/0x20
[ 1931.719531] call_timer_fn+0x35/0x140
[ 1931.719535] run_timer_softirq+0x222/0x4d0
[ 1931.719540] ? ktime_get+0x3c/0xb0
[ 1931.719545] ? lapic_next_event+0x1d/0x30
[ 1931.719550] ? clockevents_program_event+0x82/0x120
[ 1931.719554] __do_softirq+0xed/0x297
[ 1931.719559] irq_exit+0xb6/0xc0
[ 1931.719563] smp_apic_timer_interrupt+0x3d/0x50
[ 1931.719566] apic_timer_interrupt+0x89/0x90
[ 1931.719570] RIP: 0033:0x7f4680ff8503
[ 1931.719573] RSP: 002b:00007f4675e28140 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 1931.719577] RAX: 0000000000000001 RBX: 00007f4675e286c0 RCX: 0000000000000001
[ 1931.719580] RDX: 00007f46824d43a2 RSI: 00007f46824d43a1 RDI: 00007f46680408f1
[ 1931.719582] RBP: 00007f4675e286b0 R08: 0000000000000000 R09: 0000000000000001
[ 1931.719584] R10: 0000000000000030 R11: 0000000000000022 R12: 00007f46824d43a1
[ 1931.719587] R13: 00007f4675e28820 R14: 0000000000000000 R15: 00007f46824d43a2
[ 1931.719589] </IRQ>
[ 1931.719592] ---[ end trace 9db9f3fa6f14b74a ]---
[ 1931.940342] AMD-Vi: Completion-Wait loop timed out
[ 1931.940347] ------------[ cut here ]------------
[ 1931.940354] WARNING: CPU: 1 PID: 15490 at /build/linux-hwe-lyR8gz/linux-hwe-4.10.0/drivers/iommu/amd_iommu.c:1256 __domain_flush_pages+0x1de/0x200
[ 1931.940355] Modules linked in: bnep binfmt_misc nls_iso8859_1 edac_mce_amd edac_core kvm irqbypass hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc uvcvideo aesni_intel aes_x86_64 crypto_simd videobuf2_vmalloc glue_helper videobuf2_memops snd_hda_codec_hdmi cryptd snd_hda_codec_realtek snd_hda_codec_generic videobuf2_v4l2 snd_hda_intel videobuf2_core snd_hda_codec videodev snd_hda_core media snd_hwdep btusb joydev btrtl btbcm snd_pcm btintel snd_seq_midi snd_seq_midi_event bluetooth snd_rawmidi input_leds serio_raw i2c_piix4 fam15h_power snd_seq snd_seq_device snd_timer snd soundcore shpchp mac_hid hp_wireless i2c_designware_platform i2c_designware_core rfkill_gpio tpm_crb 8250_dw parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log amdkfd
[ 1931.940428] amd_iommu_v2 amdgpu i2c_algo_bit ttm psmouse drm_kms_helper syscopyarea sysfillrect ahci libahci sysimgblt fb_sys_fops r8169 drm mii wmi i2c_hid hid fjes video
[ 1931.940452] CPU: 1 PID: 15490 Comm: gdbus Tainted: G W 4.10.0-38-generic #42~16.04.1-Ubuntu
[ 1931.940454] Hardware name: HP HP 245 G6 Notebook PC/8324, BIOS F.20 11/07/2017
[ 1931.940456] Call Trace:
[ 1931.940458] <IRQ>
[ 1931.940462] dump_stack+0x63/0x90
[ 1931.940466] __warn+0xcb/0xf0
[ 1931.940470] warn_slowpath_null+0x1d/0x20
[ 1931.940473] __domain_flush_pages+0x1de/0x200
[ 1931.940477] ? queue_flush_all+0x90/0x90
[ 1931.940480] __queue_flush+0x46/0xc0
[ 1931.940483] queue_flush_all+0x77/0x90
[ 1931.940487] queue_flush_timeout+0x18/0x20
[ 1931.940491] call_timer_fn+0x35/0x140
[ 1931.940495] run_timer_softirq+0x222/0x4d0
[ 1931.940499] ? ktime_get+0x3c/0xb0
[ 1931.940504] ? lapic_next_event+0x1d/0x30
[ 1931.940509] ? clockevents_program_event+0x82/0x120
[ 1931.940513] __do_softirq+0xed/0x297
[ 1931.940518] irq_exit+0xb6/0xc0
[ 1931.940522] smp_apic_timer_interrupt+0x3d/0x50
[ 1931.940525] apic_timer_interrupt+0x89/0x90
[ 1931.940528] RIP: 0033:0x7f4680ff8503
[ 1931.940530] RSP: 002b:00007f4675e28140 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 1931.940535] RAX: 0000000000000001 RBX: 00007f4675e286c0 RCX: 0000000000000001
[ 1931.940537] RDX: 00007f46824d43a2 RSI: 00007f46824d43a1 RDI: 00007f46680408f1
[ 1931.940539] RBP: 00007f4675e286b0 R08: 0000000000000000 R09: 0000000000000001
[ 1931.940542] R10: 0000000000000030 R11: 0000000000000022 R12: 00007f46824d43a1
[ 1931.940544] R13: 00007f4675e28820 R14: 0000000000000000 R15: 00007f46824d43a2
[ 1931.940546] </IRQ>
[ 1931.940550] ---[ end trace 9db9f3fa6f14b74b ]---
[ 1932.161305] AMD-Vi: Completion-Wait loop timed out
[ 1932.262375] AMD-Vi: Event logged [
[ 1932.262383] IOTLB_INV_TIMEOUT device=00:01.0 address=0x00000001f59afe80]
[ 1932.498405] AMD-Vi: Completion-Wait loop timed out
[ 1932.734537] AMD-Vi: Completion-Wait loop timed out
[ 1932.970548] AMD-Vi: Completion-Wait loop timed out
[ 1933.206564] AMD-Vi: Completion-Wait loop timed out
[ 1933.264259] AMD-Vi: Event logged [
[ 1933.264266] IOTLB_INV_TIMEOUT device=00:01.0 address=0x00000001f59aff90]

[ewblen]

System: Host: mint Kernel: 4.10.0-38-generic x86_64 (64 bit) Desktop: Cinnamon 3.6.6
Distro: Linux Mint 18.3 Sylvia
Machine: System: HP (portable) product: HP 245 G6 Notebook PC
Mobo: HP model: 8324 v: 22.24 Bios: American Megatrends v: F.20 date: 11/07/2017
CPU: Dual core AMD E2-9000e RADEON R2 4 COMPUTE CORES 2C+2G (-MCP-) cache: 2048 KB
clock speeds: max: 1500 MHz 1: 1300 MHz 2: 1300 MHz
Graphics: Card: Advanced Micro Devices [AMD/ATI] Device 98e4
Display Server: X.Org 1.18.4 drivers: ati,amdgpu (unloaded: fbdev,vesa,radeon)
Resolution: 1366x768@60.01hz
GLX Renderer: Gallium 0.4 on AMD STONEY (DRM 3.9.0 / 4.10.0-38-generic, LLVM 4.0.0)
GLX Version: 3.0 Mesa 17.0.7
Audio: Card-1 Advanced Micro Devices [AMD] Device 157a driver: snd_hda_intel
Card-2 Advanced Micro Devices [AMD/ATI] Device 15b3 driver: snd_hda_intel
Sound: Advanced Linux Sound Architecture v: k4.10.0-38-generic
Network: Card-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller driver: r8169
IF: eno1 state: down mac: ac:e2:d3:5b:bd:6b
Card-2: Realtek Device d723
IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A
Drives: HDD Total Size: 1032.2GB (1.3% used) ID-1: USB /dev/sda model: USB_3.0_JET_16GB size: 15.9GB
ID-2: /dev/sdb model: TOSHIBA_MQ01ABD1 size: 1000.2GB
ID-3: USB /dev/sdc model: USB_Flash_Drive size: 16.0GB
Partition: ID-1: swap-1 size: 8.01GB used: 0.00GB (0%) fs: swap dev: /dev/sdb7
RAID: No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors: System Temperatures: cpu: 37.0C mobo: N/A gpu: 0.0
Fan Speeds (in rpm): cpu: N/A
Info: Processes: 194 Uptime: 5 min Memory: 593.3/7504.8MB Client: Shell (bash) inxi: 2.2.35

[rene]

You should try booting with the "amd_iommu=off" kernel parameter. Not sure moreover why this wouldn't be applicable to the Live/installer session, so I'd boot also that one with that parameter.

As to the Live/installer session: at the boot-menu, with the selection at "Start Linux Mint", hit "e" to edit the "linux" line and add "amd_iommu=off" at the end; press F10 to boot. As to the installed system: try tapping Esc while booting to get into the grub menu and do the same. To add it permanently after install edit /etc/default/grub and add it to the GRUB_CMDLINE_LINUX_DEFAULT line, save, and run sudo update-grub.

[EDIT] The HP site is buggy in current Firefox so can't check but also make sure you have the newest BIOS flashed and/or no possibility for IOMMU tweakage exists in its settings. It may be referred to as AMD-V or some other generic "virtualization" naming.

[ewblen]

Ah well, it looks like this issue was discussed on LKML in April last year:
Thread start: "AMD IOMMU causing filesystem corruption" https://www.spinics.net/lists/kernel/msg2480782.html

[rene]

Installing a 4.13 kernel will then supposedly fix things for real -- but note that as to installation and/or workaround with the standard 4.10 kernel indeed amd_iommu=off would be the thing to do.

[ewblen]

Thanks Rene,

Good news:using amd_iommu=off is working (so far).
I'm slightly scared to turn it off for 4.13 kernel. If it is easy for you can you point out how you're sure it is OK. (otherwise I can search through LKML...)

Reason I'm scared is that the final time I booted without iommu off to gather detailed logs, the EFI boot partition was damaged along with the root partition. Causing much pain retaining access to the installed other OS!

--
Eliot

[rene]

I'm slightly scared to turn it off for 4.13 kernel. If it is easy for you can you point out how you're sure it is OK. (otherwise I can search through LKML...)

I am not; am in fact after checking sure it is not. That's pretty bad: a filesystem destroying bug that was identified during 4.11 the fix for which has made it into mainline only in 4.14 and does not seem to have been backported either. The level of confusion in the relevant thread, https://patchwork.kernel.org/patch/9669611/, is something I feel does not do justice to the issue's severity.

As to Ubuntu's kernels specifically, you in its 4.15 series find the ATS quirk present:

http://kernel.ubuntu.com/git/ubuntu/ubu ... edge#n4847

but not in its 4.13 series, http://kernel.ubuntu.com/git/ubuntu/ubu ... ks.c?h=hwe. You should be able to verify directly whether or not the quirk is present in your kernel with a simple dmesg | grep ATS. If you are safe without disabling all of amd_iommu then you should be seeing "Disabling ATS (broken on this device)".

Also note by the way: even an EFI partition is a simple bag of bytes: you may wish to image it with dd or alike (while booted into a amd_iommu=off kernel) for easy restoration when things do still go south at some point.

[ewblen]

Thanks René,

I'll play it safe for now. I took your advice to add amd_iommu=off to grub defaults.

Adding to the badness of the problem, I'm pretty sure that the filesystem on a backup drive that I was restoring files from got damaged as well.
Fortunately the laptop I'm migrating from is still working, and has a fairly new hard drive so I dodged that bullet.

The insidious thing about this problem is that the system doesn't just die on startup, but can run for tens of minutes before failing, giving ample time for e.g.connecting backup drive.

On EFI partition image - yes, caution in hindsight is a wonderful thing. I should have generated recovery media before embarking on this project!

--
E

[ewblen]

For the record, I've documented my experience with this laptop here: https://bigblen.wordpress.com/2018/02/2 ... -problems/

[gianluca.brizi]

Hi, I had the same issue on Lenovo v110 with AMD E2 7th generation. Processor 9010. I solved with the 19th version of Linuxmint. It has the right kernel for this issue and the cmd: "dmesg | grep ATS" gave me the right retourn. After the installation I inserted the suggested line at the end of the /etc/default/grub file. Thanks.

[rene]

Note, if dmesg | grep ATS says "Disabling ATS (broken on this device)" then you don't in fact additionally need the "amd_iommu=off" kernel parameter; the latter is only for working around the issue in pre-4.15 kernels; better, kernels that don't implement the ATS quirk already.