I am a Java developer, pretty good with the shell, but really could use some expert advice right now. I'm pretty novice in this peer-group, but I can read a man page. I've been using Mint for a few years now, and LMDE for about 6 months. First, the formalities:
LMDE Cinnamon 64, latest everything.
Acer Aspire, Sandy Bridge I5 (dual core), 8GB RAM, 500GB hybrid SSD (I replaced the old drive and upgraded the memory myself).
Oracle Java 7u51
Unless someone recognizes this immediately, I think what I need is some ideas on how to see the problem. I've used top, System Monitor, and tried some other tools that I could quite not figure out, but I just am not seeing the problem.
Here is the description:
I am executing a rather long process in which I upload a bunch of randomly generated documents into an XML database (I see a similar issue on both EXist and xDB). This actually involves 2 processes talking over HTTP. There are a significant number of small files being created and closed continuously. The sockets are being managed however Java manages sockets.
So here is what I am seeing. After about 5 minutes of continuous imports, the CPU drops from about 2 cores fully utilized to near 0% and hangs there for maybe 20 to 40 seconds. Then it starts importing again, like normal, and after a bit, it hangs at 0% again. This keeps happening for hours, until the process completes. That is, the process completes normally, but it just takes a lot longer than it should.
While the CPU is at 0%, the desktop is only partially responsive, and almost unusable. I notice that apps like FireFox will hang during this time. No refresh. But after a bit, it catches up and things work until the next cycle.
What I have done so far:
My first suspicion is that the OS is running out of some kind of resource, and after a bit, it recovers enough of that resource to continue. I set ulimit -n 65535 (or the equivalent setting in limits.conf) and verified that the system was seeing this correctly. No change in behavior. The process is creating and deleting lots of small temporary files, so I thought this might be a good place to look.
I could not figure out how to see if I was running out of socket ports, something simple like that.
I saw something about an entropy collection problem, searching around, but usually if you move the mouse around a lot, that changes the behavior (entropy is collected automatically from mouse motion). I didn't see that moving the mouse affected the behavior/symptoms. Also, most of my randomness comes from a Java Random object, which doesn't use system entropy after the first time you create it.
I am pretty sure it is not Java memory settings (though it still could be) because of the way this affects the entire system. The process needs about 500MB of heap to run comfortably, and I gave it 3GB. Memory usage is stable, both heap and non-heap, as reported by the JVM.
So are there other type of resources that I could be looking at (that are limited)? Other things I should be aware of? it feels to me like a resource limitation (this is a desktop install, not a server install, after all). In the past I have run into the ulimit problem, but this doesn't seem to be that. What are some other types of resources, and how do I know they've run out?
Oh yeah. I do very large Maven builds on this machine (30 minutes, very heavy), and they seem to work okay. I may be seeing the same problem, but it isn't as apparent.
I am hoping to learn something new and exciting from this. Linux is a great teacher. Can anyone help me get on the path? Thanks!