SOLVED Kworker/USB maxing out single core

Questions about applications and software
Forum rules
Before you post please read how to get help
Post Reply
gessel
Level 1
Level 1
Posts: 34
Joined: Thu Sep 17, 2015 5:01 am

SOLVED Kworker/USB maxing out single core

Post by gessel » Sun Feb 26, 2017 6:15 pm

This seems to be a recurring issue and most recent solutions seem to center around unplugging USB devices. The only USB device attached to my laptop is my mouse (my keyboard is PS/2). Detaching my mouse didn't resolve the CPU load, though it seems to be USB-related based on stack traces. While I'm happy to mess with BIOS settings per https://askubuntu.com/questions/33640/k ... o-much-cpu, before making more work for myself I had hoped there might be some useful wisdom here:

I just updated to 18.1 and a new kernel, but the problem predates the update and wasn't resolved either with 18.1 or the 4.8 kernel: Linux iku 4.8.0-39-generic #42~16.04.1-Ubuntu SMP Mon Feb 20 15:06:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
htop shows 100% load on core 8 (at the moment, this of course varies).

Debugging with

Code: Select all

# echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
# cat /sys/kernel/debug/tracing/trace_pipe > out.txt
Yields a whole lot of

Code: Select all

     kworker/7:3-2528  [007] d... 41788.273239: workqueue_queue_work: work struct=ffff9757f17a79e0 function=pm_runtime_work workqueue=ffff9757f950fc00 req_cpu=512 cpu=7
     kworker/7:3-2528  [007] d... 41788.273281: workqueue_queue_work: work struct=ffff9757f1016208 function=hub_event workqueue=ffff9757f90afc00 req_cpu=512 cpu=7
(334,876 entries in about 10 seconds, seems like a lot to me)

Running # sudo perf record -g -a sleep 10
implicates xhci_hub_control

Code: Select all

 #  lspci | grep USB
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
3f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
Per this post https://askubuntu.com/questions/645/how ... mmand-line I tried # echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind and CPU usage on core 8 dropped to zero - totes a USB issue. of course my mouse died, but my PS2 keyboard and mouse button work fine (I disable the touchpad as it is pretty much unusable due to hypersensitivity to palm proximity).

Executing # echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind brings the mouse back. Might be a suboptimal solution if you have a USB keyboard, and definitely not a recommended tactic with USB drives attached as they won't be dismounted gracefully at all. Is there any better solution?
Last edited by gessel on Wed Mar 01, 2017 3:12 pm, edited 1 time in total.

User avatar
phd21
Level 19
Level 19
Posts: 9312
Joined: Thu Jan 09, 2014 9:42 pm
Location: Florida

Re: Kworker/USB maxing out single core

Post by phd21 » Mon Feb 27, 2017 12:33 am

Hi "gessel",

It would help to know more about your system setup. If you run "inxi -Fxzd" and "lsusb" and "df" from the console terminal prompt, highlight the results, copy and paste them back here, that should provide enough information.

I am assuming that you are using version 18.1 of Linux Mint KDE because of "kworker". I did have 100% CPU reading on my Linux Mint 18 KDE all the time on a single core computer until I upgraded to 18.1 Plasma 5.8.x; I figured it must be a bug in memory reporting because the same computer using Linux Mint 17.3 KDE did not do that.

Interesting reading those links you provided, and that when someone's system is really low on space, that will increase CPU usage and create other problems; of course that is pretty true with any computer operating system.

I am currently running Linux Mint 17.3 KDE and version 18.1, with USB Keyboard and mouse, on ancient 7 years old underpowered computer with only 3 gb system memory (Ram), and I am not having the problems you are having. At least it is 64-bit and dual core which does indeed make a positive difference.


Hope this helps ...
Phd21: Mint KDE 18.3 & 19, 64-bit Awesome OS, Ancient Dell OptiPlex 780 Core2Duo E8400 3GHz,4gb Ram,256gb SDD, Video: Intel 4 Graphics, DVD Lightscribe. Why I use KDE?:https://opensource.com/life/15/4/9-reasons-to-use-kde

gessel
Level 1
Level 1
Posts: 34
Joined: Thu Sep 17, 2015 5:01 am

Re: Kworker/USB maxing out single core

Post by gessel » Mon Feb 27, 2017 3:08 am

The "k" in kworker stands for kernel, not KDE. I'm running XFCE. There's some good info in The Linus' bug report at https://lkml.org/lkml/2010/8/15/111. It is a long standing bug: https://bugs.launchpad.net/ubuntu/+sour ... bug/717919 and others here have seen it viewtopic.php?f=47&t=231930&p=1228813&h ... r#p1228813. Mine is definitely xHCI dependent and returns over time to an idle machine - that is if I do the unbind/bind trick the runaway process dies for a while, but eventually comes back even if I'm not using the computer - just idle (not even a screen saver). I'm not quite sure how long that takes, but but between the first post and this one it spontaneously came back and was cured again with unbind/bind and I haven't touched the computer nor changed the USB config in any way during that time.

Code: Select all

# inxi -Fxzd
System:    Host: iku Kernel: 4.8.0-39-generic x86_64 (64 bit gcc: 5.4.0) Desktop: Xfce 4.12.3 (Gtk 2.24.28)
           Distro: Linux Mint 18.1 Serena
Machine:   System: Hewlett-Packard (portable) product: HP ZBook 15 G2 v: A3009DD10203
           Mobo: Hewlett-Packard model: 2253 v: KBC Version 03.10
           Bios: Hewlett-Packard v: M70 Ver. 01.08 date: 03/16/2015
CPU:       Quad core Intel Core i7-4910MQ (-HT-MCP-) cache: 8192 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 23147
           clock speeds: max: 3900 MHz 1: 3750 MHz 2: 3015 MHz 3: 3186 MHz 4: 2900 MHz 5: 3762 MHz 6: 2950 MHz
           7: 3215 MHz 8: 2900 MHz
Graphics:  Card: NVIDIA GK106GLM [Quadro K2100M] bus-ID: 01:00.0
           Display Server: X.org 1.18.4 driver: nvidia tty size: 224x80 Advanced Data: N/A for root
Audio:     Card-1 Intel 8 Series/C220 Series High Definition Audio Controller
           driver: snd_hda_intel bus-ID: 00:1b.0
           Card-2 NVIDIA GK106 HDMI Audio Controller driver: snd_hda_intel bus-ID: 01:00.1
           Sound: Advanced Linux Sound Architecture v: k4.8.0-39-generic
Network:   Card-1: Intel Ethernet Connection I217-LM driver: e1000e v: 3.2.6-k port: 7040 bus-ID: 00:19.0
           IF: enp0s25 state: up speed: 1000 Mbps duplex: full mac: <filter>
           Card-2: Intel Wireless 7260 driver: iwlwifi bus-ID: 3e:00.0
           IF: wlp62s0 state: down mac: <filter>
Drives:    HDD Total Size: 3128.6GB (54.9% used) ID-1: /dev/sda model: Samsung_SSD_850 size: 2000.4GB temp: 0C
           ID-2: /dev/sdb model: Samsung_SSD_850 size: 1000.2GB temp: 0C
           ID-3: /dev/sdc model: SanDisk_SD6PP4M size: 128.0GB temp: 0C
           Optical: No optical drives detected.
Partition: ID-1: / size: 86G used: 13G (16%) fs: ext4 dev: /dev/dm-0
           ID-2: /home size: 1.8T used: 1.6T (90%) fs: ext4 dev: /dev/sda1
           ID-3: /boot size: 236M used: 112M (51%) fs: ext2 dev: /dev/sdc1
           ID-4: swap-1 size: 4.19GB used: 0.00GB (0%) fs: swap dev: /dev/dm-1
           ID-5: swap-2 size: 34.27GB used: 0.00GB (0%) fs: swap dev: /dev/dm-3
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 55.0C mobo: 51.0C gpu: 0.0:51C
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 334 Uptime: 21:17 Memory: 4443.0/32090.0MB Init: systemd runlevel: 5 Gcc sys: 5.4.0
           Client: Shell (bash 4.3.461) inxi: 2.2.35 

Code: Select all

# lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp. 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 0424:5434 Standard Microsystems Corp. Hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 008: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 003 Device 006: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 003 Device 005: ID 04f2:b466 Chicony Electronics Co., Ltd 
Bus 003 Device 003: ID 138a:003f Validity Sensors, Inc. VFS495 Fingerprint Reader
Bus 003 Device 007: ID 8087:07dc Intel Corp. 
Bus 003 Device 004: ID 046d:c042 Logitech, Inc. G3 Laser Mouse
Bus 003 Device 002: ID 0424:5434 Standard Microsystems Corp. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
and

Code: Select all

# df
Filesystem                 1K-blocks       Used Available Use% Mounted on
udev                        16407984          0  16407984   0% /dev
tmpfs                        3286020      10064   3275956   1% /run
/dev/mapper/mint--vg-root   89747632   12820680  72344964  16% /
tmpfs                       16430080      82576  16347504   1% /dev/shm
tmpfs                           5120          4      5116   1% /run/lock
tmpfs                       16430080          0  16430080   0% /sys/fs/cgroup
/dev/sda1                 1918698544 1629812116 191399172  90% /home
/dev/sdc1                     240972     114590    113941  51% /boot
cgmfs                            100          0       100   0% /run/cgmanager/fs
tmpfs                        3286020         36   3285984   1% /run/user/1000

User avatar
phd21
Level 19
Level 19
Posts: 9312
Joined: Thu Jan 09, 2014 9:42 pm
Location: Florida

Re: Kworker/USB maxing out single core

Post by phd21 » Mon Feb 27, 2017 11:23 am

Hi "gessel"

I did not realize that "Kworker" was common to other Linux Mint editions & versions, my bad. I have been using Linux Mint KDE almost exclusively for a long time, so I think anything that starts with a "K" means KDE.

I'll assume you have already applied the Mint Updates including Kernel updates, and checked to see if there are any recommended drivers in the "Driver Manager" or "Drivers" option ...

According to the results of the "Inxi-Fxzd" console terminal command, you certainly have more than enough computer power. Did you run that as root user with sudo or su, no need for that, because it did not show all the graphics details (Advanced Data: N/A for root)?

After you have booted up, try removing the mouse USB receiver, waiting 10-12 seconds, an reinserting it... Are you using the "Solaar" application for the Logitech mouse?

I know there is a "Grub" command parameter, and other configuration options, to prevent USB devices from "going to sleep" (auto suspend) which might help. Also, do not use passive USB hubs without their own power supply, as they do not work well or at all under Linux Mint, or even worse, they work intermittently.

https://forums.gentoo.org/viewtopic-t-994070.html

http://forums.funtoo.org/topic/582-ques ... arguments/

Grub boot parameter option to disable (turn off) auto suspend of USB devices

Code: Select all

'usbcore.autosuspend=-1'
I have also read where some USB 1 and USB 2 devices will sometimes not work properly if they are plugged into USB 3.x ports. And apparently on some newer computers, there are USB ports that are dedicated ? to specific devices like keyboards or mice, so you might want to check that.

High CPU usage of kworker/ksoftirqd
https://bugs.launchpad.net/ubuntu/+sour ... ug/1488426

Hope this helps ...
Phd21: Mint KDE 18.3 & 19, 64-bit Awesome OS, Ancient Dell OptiPlex 780 Core2Duo E8400 3GHz,4gb Ram,256gb SDD, Video: Intel 4 Graphics, DVD Lightscribe. Why I use KDE?:https://opensource.com/life/15/4/9-reasons-to-use-kde

gessel
Level 1
Level 1
Posts: 34
Joined: Thu Sep 17, 2015 5:01 am

Re: Kworker/USB maxing out single core

Post by gessel » Mon Feb 27, 2017 4:47 pm

I made the same error, conditioned to associate "k" with KDE.

Updated the OS/kernel and am using the recommended drivers for video (nvidia-367) and the intel microcode driver. There aren't any others suggested.

I did run it from a su terminal. If helpful, i can run it from a normal user.

I'm not using Solaar - do you suggest I should?

I've tried adding the auto-suspend disable. That seems like it might be on the right path. It might also help with some other USB issues. I have to restart and will update in a few hours if the problem hasn't returned. THANKS!!!

(more details on the necessary steps to make the mod, should it work: https://unix.stackexchange.com/question ... 0-or-above)

User avatar
phd21
Level 19
Level 19
Posts: 9312
Joined: Thu Jan 09, 2014 9:42 pm
Location: Florida

Re: Kworker/USB maxing out single core

Post by phd21 » Mon Feb 27, 2017 8:20 pm

Hi "gessel",

You are welcome.

Since you seem comfortable with computers, just re-run the "inxi -Fxzd" as a regular user to see that you do not have any video errors. If you feel like posting it back, then go for it.

If you have not rebooted since updating the Kernel, and or Grub, then do so.

Yes, for anyone using wireless Logitech keyboards and or mice, I would install "Solaar" even though they seem to work without it.

I will check out that link you gave as well.

Hope this helps ...
Phd21: Mint KDE 18.3 & 19, 64-bit Awesome OS, Ancient Dell OptiPlex 780 Core2Duo E8400 3GHz,4gb Ram,256gb SDD, Video: Intel 4 Graphics, DVD Lightscribe. Why I use KDE?:https://opensource.com/life/15/4/9-reasons-to-use-kde

gessel
Level 1
Level 1
Posts: 34
Joined: Thu Sep 17, 2015 5:01 am

Re: Kworker/USB maxing out single core

Post by gessel » Wed Mar 01, 2017 3:10 pm

Hi phd21,

Thank you for your help. This seems to be solved.

I've left the computer on for a few days now and the issue hasn't returned - CPU use is normal and I credit the auto-suspend disable change.

Summary:
  • CPU usage on one core maxes out, seemingly at random, but most likely after the machine has been left idle for some time.
  • top shows kworker (plus a thread indicator) as the culprit.
  • digging deeper (see thread) shows indication of endless loop of tasks associated with USB activity such as pm_runtime_work and hub_event
  • digging even deeper (with perf) implicates a USB subsystem (in my case xhci_hub_control
  • executing # echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind && echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind to restart the USB service in question (see details in thread on finding the correct address and hub subsystem) restores normal CPU load.
  • disabling auto-suspend (of USB devices) solves the problem (see details in thread).

User avatar
phd21
Level 19
Level 19
Posts: 9312
Joined: Thu Jan 09, 2014 9:42 pm
Location: Florida

Re: Kworker/USB maxing out single core

Post by phd21 » Wed Mar 01, 2017 3:29 pm

Hi "gessel",

That's good news !

You are very welcome. Glad to help ...
Phd21: Mint KDE 18.3 & 19, 64-bit Awesome OS, Ancient Dell OptiPlex 780 Core2Duo E8400 3GHz,4gb Ram,256gb SDD, Video: Intel 4 Graphics, DVD Lightscribe. Why I use KDE?:https://opensource.com/life/15/4/9-reasons-to-use-kde

Post Reply

Return to “Software & Applications”