I just signed up for a Linode 360 last night, to host a Linux distribution site ( www.ultimalinux.com ). It's on dallas58, UML, and running a custom distribution - bet you can't guess which one Everything's running fine, except I've gotten at least two of these messages today:


It's been a while since I used Linode (or UML in general), so I'm not sure what would be causing this. My installation currently runs a whole bunch of servers - Apache, MySQL, Sendmail, OpenSSH - but I don't think that's the problem; a nearly identical configuration on a Pentium-3 box at home doesn't have the same load issues:

load average: 2.88, 2.90, 2.94

on the Linode, vs.

load average: 0.08, 0.02, 0.01

on the P-3. Both of these are with a heavily optimized configuration, there haven't been any memory problems, and overall the Linode's performance has been great, much better than a couple years ago when it was still a Linode-80.

The Linode hasn't been running any CPU-intensive tasks, and I'm not even sure how much network activity it's seen apart from my working on the configuration. top is showing some extremely bizarre fluctuations, one minute it's around 5 user/10 sys, next it's 20 user/80 sys.

updatedb and all those other tasks are disabled on the machine, in fact I didn't even install the locate package

The only thing I can think of is that the installation uses an NPTL-only glibc, which I believe is how newer Slackware releases are also built (my code's originally forked from a 2006-ish Slackware-current, and has since been independently developed). I saw that's a bit of a problem with Xen, could that be impacting UML as well?

~Martin Ultima

How does your CPU graph on the platform manager look?

Also, NPTL libc isn't a problem with Xen on a 64-bit host (as linode uses), or on UML at all.

Hello Martin,

There are no incompatibilities or issues with NPTL in either our Xen or UML setups. UML has supported NPTL for quite some time. Also, our Xen config doesn't even require a modified libc to run at full speed (the hypervisor runs in 64bits).

I can verify that your Linode is spinning CPU on the host.

-Chris

Thoguht I'd give my 5 cents, I had unusual high loads as well recently, and the solution was that mysqld_safe was running twice.

Stop mysql, use killall mysqld_safe and the nstart mysql again and see if that helps.

It looks something like this:


By the way, it's running kernel 2.6.23.17-linode43 (although I don't think there were any other choices).


I'll have to keep that in mind - I've never been good with libc-related stuff, I'm happy to do whatever so long as it works

I have MySQL completely stopped, and it's still giving 80% sys in top. While it was running there was only one mysqld and one mysqld_safe process.


Well, I certainly couldn't hope for anything more definitive than that.


As long as you're aware, that's my main concern - I actually came back to Linode after my previous hosting was compromised and subsequently deleted. (Learned my lesson about third-party shared hosting, and settling for anything less than SSH...) I certainly wouldn't want my new site getting removed after all that

Edit I did a bit of experimentation, and the CPU usage appears to have subsided after shutting down inetd. I have a feeling it might have been some program acting up because I was re-configuring the Linode's hostname. Now I have 0% user, 0-5% sys, but 100% id (offhand I have no idea what that even is).

Edit Oops... sorry for double-posting, thought someone else had already replied again

id is idle.

from man top:

Slackware doesn't work in Xen? My UML and Xen Linodes both beg to differ.

Another thing I'll have to remember... You can probably tell I haven't used top much before (I tend to prefer a combination of ps -A, killall -9, and free -m, and for everything else I just eyeball it...)

I'm assuming that 'idle' is what we're going for, or...? On my machines here, I've dabbled a bit, but never seriously monitored CPU utilization much - they're either idling, or else running builds, usually two or three simultaneously, at which point starting the CPU monitor frequently crashes the machine.


I don't think I meant it that way, but I will freely admit I have little UML-specific experience, and none whatsoever with Xen.

At any rate, CPU usage appears to be much better now:


I'll test it on one of my machines here, just to be sure, but I think it was some errant inetd process that didn't like my hostname changes, judging from (a) the CPU usage calming after restarting inetd, and (b) the fact that I saw a few "inetd, 100% usage" processes flash on my screen...

If everything checks OK, I guess I'll consider this one solved - my Linode's running fine now otherwise, but I may be back...

inetd may have started a child that was burning your CPU.
Next time, try "ps faux" if your system has that. It gives you a tree of processes with memory and CPU usage. (It helps to have a wide terminal for this.)

That's what I suspected. I reached that particular conclusion independently, about three days ago.


I'm aware of the ps command; I'm not particularly big into the -f option, or -aux, or anything that involves horizontally resizing my terminal window. (I generally have two or three running at any given moment, and even dual 1280x1024 displays only have so much screen space available.) I prefer a simple ps -A myself, since it's less cluttered, and it's easier to tell at a glance what should/shouldn't be running. But thanks for the tip.

I've currently kludged it by setting a cron job to restart inetd every morning around 4 am; this has worked nicely enough the last several days. I'm planning to investigate the cause of the problem once I have more time, possibly this weekend, but meanwhile I'm still working to rebuild the Web site - I had a copy of its database, but it was deleted before I could get a good backup of the files - and not to mention, there's also that whole issue of developing the actual product...

UPDATE: I should have posted this almost a month ago, but I've been a bit busy. Oh well.

Anyway, a glance through /var/log/secure later, it turns out that inetd was trying to spawn an in.identd process, but the executable in question wasn't on the system. (I forgot so many services were enabled by default...) I have since disabled that particular service, as well as several others that I don't use, and the CPU usage problems have not come up since.

I also discovered in syslog a rather absurd number of attempts on the SSH port - I guess just like everyone else here, judging from all the other topics That was around April 13. My new sshd_config has put an end to all that, although I don't think it's quite draconian enough yet, even considering - 15 minutes grace time is still a lot, and I should probably disable interactive password entry since I always use SSH keys (and I'm the only current user on the site).

_________________
Hi, I'm li25-188.members.linode.com!