Running Latest 32 bit (3.8.4-linode50)

Guess when I rebooted or migrated to take advantage of the upgrades...



This is for a machine that's 99% idle (or, as top puts it, 100% )

top - 22:14:55 up 1 day, 23:50,  1 user,  load average: 0.02, 0.18, 0.13
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024976k total,   800344k used,   224632k free,   124216k buffers
Swap:   263164k total,      284k used,   262880k free,   531736k cached

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)

That's a two-hour average -- you sat there watching top the whole time? Maybe you have some CPU-intensive cronjob that runs for a few seconds. Longview?

_________________
Matt Nordhoff (aka Peng on IRC)

Workload on the server hasn't changed. In March the graph said 5%. After rebooting for upgrade 1 the graph jumped to 6.5%. After rebooting for upgrade 2 the graph jumped to 9%. And yet the workload is unchanged...

So it doesn't matter if I have spikes in activity (yes, once an hour there's a job that spikes to 40% on the 5 minute graph); the fact is that the change in baseline as measured by these graphs corresponds exactly with the reboots and upgrades.

_________________
Rgds

Stephen

(Linux user since kernel version 0.11)

I've noticed the same thing. After the first 4 -> 8 core upgrade, MySQL spun up a lot more processes which explained the jump in CPU and RAM usage I saw. I haven't looked into accounting for the latest jump due to the memory increase, at percentages that low I'm not sure I'll bother though .

Yeah, I think something does seem to be going on with the CPU stats after the upgrade. In one of my accounts, out of 7 upgrades, 3 show a unambiguous step up to a new level in CPU. If anything else, on the faster CPUs I'd have thought they'd shrink a little (assuming 100% is still 1 core).

In one case, a web server that has been static in terms of work load for a long time, CPU graphs have been essentially flat lines say back as far as Jan 2012, with an average on the monthly (2-hr) graphs of 3.2-3.7%. After the upgrade, and right after the upgrade, the average (also on the 2-hr graph) has jumped to around 6.4. That can't just be a coincidence. Active processes and the work being done remain the same as before the upgrade. No database scaling by cores or anything, and the web site is static file serving. I've got also munin running on the same node (with a finer granularity) and don't appear to be seeing the same sort of change in those graphs.

For another data point, in a different account I have a Linode that for most intents and purposes has been completely idle a lot recently. Just logging into it can cause a noticeable spike in the graph and most graphs have an entire scale of maybe 0.03-0.05%. Its graph looks like it ran into Mt. Everest after the upgrade, with a 10x increase, albeit from ~0.05% to ~0.5%.

It may be coincidental but it seemed to affect my more lightly loaded nodes. Of course, the changes are small in terms of absolute value, but certainly large relatively speaking. Maybe it's just a small difference in collection or calculation/precision on the new hosts, but something seems different. It doesn't appear to be a fixed percentage or fixed overhead (given the difference in my two samples).

Maybe statistics gathering on new hosts is allocating more of the host time spent on behalf of a guest to that guest or something, but you're only likely to notice it particularly on lower average CPU nodes. That might also explain why it's hard to see in with purely guest-based monitoring (whether top or something like munin).

-- David

PS: mnordhoff, the fact that it's a 2 hour average should actually make it easier to catch the cause. A long average like a 2 hour window will tend to completely hide short, bursty CPU-intensive tasks, so probably wouldn't move that much if that was the reason. To move a 2 hour average requires the extra CPU usage be getting used for a good fraction of each interval and likely at higher levels than the average ends up reflecting. So given a change that is stable over a long period of time, it should actually be possible to pick up interactively. Although, of course, picking up a move of only a few percent may be tricky no matter how long the period.