That happens because the counter has overflowed into gibberish.

In any case, the problem you're having is obviously the one I've recently filed a support case for (closed URL).

I'm actually surprised that this has come up now. I've had this issue for many months with the paravirt kernels. I just assumed that it was some odd Xen-ism that I didn't bother to pursue, because having wonky user/sys times wasn't that large of an issue for me.

Why do people keep suggesting this is a Linode issue?

Here are my numbers from Bjarmeson's test and this is on my 10.04 desktop (upgraded through 7.04 to 10.04)

All tests successful.
Files=100, Tests=500, 6 wallclock secs ( 0.58 usr 0.37 sys + 2.73 cusr 3.59 csys = 7.27 CPU)
Result: PASS

real 0m7.395s
user 0m3.696s
sys 0m4.020s

This is on my Linode Ubuntu 10.04:

All tests successful.
Files=100, Tests=500, 5 wallclock secs ( 0.57 usr 0.70 sys + 1608.09 cusr 8.69 csys = 1618.05 CPU)
Result: PASS

real 0m6.297s
user 26m48.963s
sys 0m9.449s

Not sure what it proves?

It's a Linode issue if you run specific kernel versions. It's happening for me on at least x86_64 2.6.32..2.6.34. But probably earlier versions, I've been encountering this issue for ~6 months or so through a few reboots.

It doesn't happen on e.g. the stable kernels.

Which distro/version are you running? I'm wondering if (in some cases) the userland might be incompatible with newer kernels, in much the same way some userlands are incompatible with the 2.6.18.8 kernel.

The 64-bit architecture throws another interesting variable into the mix, as well.

Also, probably splitting hairs here, but any bugs in Linode kernels are imported verbatim from kernel.org

I'm running a Debian that tracks testing.

I'm getting pretty much the same problem with Fedora 13, 32 bit. Kernel
is 2.6.33-linode24.

I've tried 2.6.34 kernels with the same issue. This has only happened
since deploying Fedora 13 a couple of weeks ago.

I have older Ubuntu and Arch linode distros that do not have this
problem, even with the same kernel. Fedora 13 on direct hardware servers
here does not have this problem.

All tests successful.
Files=100, Tests=500,  4 wallclock secs (541.74 usr 662.00 sys + 4075.79
cusr 13341.57 csys = 18621.10 CPU)
Result: PASS

real   0m4.997s
user   76m58.004s
sys   233m23.644s

I'm running rails on the Fedora 13 distro and all 4 rails instances show
about 10 minutes CPU time when the linode has only been up about a
minute!

I could not reproduce any of this with 64-bit Debian lenny or testing, running a variety of kernels:

Deployed linode54174 97.107.134.59 with Debian 5.0 64bit
Assigned host: newark111

*** TEST 1: debian 5.0 64-bit, Latest 2.6 Paravirt (2.6.32.12-x86_64-linode12)
$ time perl many-digs.pl ; cat /proc/uptime ; uname -a
Files=100, Tests=500, 50 wallclock secs ( 2.08 cusr +  5.41 csys =  7.49 CPU)

real   0m49.498s
user   0m2.468s
sys   0m5.747s
996.11 7839.08
Linux li64-59 2.6.32.12-x86_64-linode12 #1 SMP Wed Apr 28 18:47:37 UTC 2010 x86_64 GNU/Linux

*** TEST 2: debian 5.0 64-bit, 2.6.34-x86_64-linode13
Files=100, Tests=500, 34 wallclock secs ( 1.90 cusr +  5.36 csys =  7.26 CPU)

real   0m34.727s
user   0m2.300s
sys   0m5.689s
75.46 577.85
Linux li64-59 2.6.34-x86_64-linode13 #2 SMP Wed Jun 23 18:54:41 UTC 2010 x86_64 GNU/Linux

*** TEST 3: debian 5.0 64-bit, Latest 2.6 Stable (2.6.18.8-x86_64-linode10)
Files=100, Tests=500, 46 wallclock secs ( 1.74 cusr +  4.02 csys =  5.76 CPU)

real   0m46.570s
user   0m2.013s
sys   0m4.146s
82.00 79.52
Linux li64-59 2.6.18.8-x86_64-linode10 #1 SMP Tue Nov 10 16:29:17 UTC 2009 x86_64 GNU/Linux

*** TEST 4: debian testing 64-bit, Latest 2.6 Paravirt (2.6.32.12-x86_64-linode12)
Files=100, Tests=500,  5 wallclock secs ( 0.56 usr  0.73 sys +  2.38 cusr  5.74 csys =  9.41 CPU)
Result: PASS

real   0m5.337s
user   0m3.166s
sys   0m6.538s
28.65 193.43
Linux (none) 2.6.32.12-x86_64-linode12 #1 SMP Wed Apr 28 18:47:37 UTC 2010 x86_64 GNU/Linux

*** TEST 5: debian testing 64-bit, 2.6.34-x86_64-linode13
Files=100, Tests=500,  9 wallclock secs ( 0.56 usr  0.68 sys +  2.08 cusr  5.47 csys =  8.79 CPU)
Result: PASS

real   0m9.794s
user   0m2.867s
sys   0m6.209s
34.11 240.37
Linux (none) 2.6.34-x86_64-linode13 #2 SMP Wed Jun 23 18:54:41 UTC 2010 x86_64 GNU/Linux

*** TEST 6: debian testing 64-bit, Latest 2.6 Stable (2.6.18.8-x86_64-linode10)
Files=100, Tests=500,  4 wallclock secs ( 0.37 usr  0.33 sys +  1.94 cusr  4.31 csys =  6.95 CPU)
Result: PASS

real   0m5.260s
user   0m2.491s
sys   0m4.694s
99.42 97.16
Linux (none) 2.6.18.8-x86_64-linode10 #1 SMP Tue Nov 10 16:29:17 UTC 2009 x86_64 GNU/Linux

*** TEST 7: debian testing 64-bit, pvgrub linux-image-xen-amd64
Files=100, Tests=500,  8 wallclock secs ( 0.54 usr  0.72 sys +  2.36 cusr  5.74 csys =  9.36 CPU)
Result: PASS

real   0m8.570s
user   0m3.124s
sys   0m6.536s
24.60 161.65
Linux (none) 2.6.32-5-xen-amd64 #1 SMP Tue Jun 1 06:38:33 UTC 2010 x86_64 GNU/Linux

Methodology:
Change repo from ftp.debian.org to mirror.anl.gov
apt-get update, apt-get upgrade (still on lenny)
apt-get install locales, dpkg-reconfigure locales (generate en_US.UTF-8)
apt-get install git-core dnsutils
cd /tmp;git clone git://gist.github.com/449825.git;cd 449825;git pull
note: edited many-digs.pl to use grep -q linode.com instead of grep -q theshore.net

for upgrade to testing:
edit /etc/apt/sources.list to replace all "lenny" with "testing"
apt-get update, apt-get dist-upgrade, hit "Enter" whenever required, reboot

for pvgrub stock kernel testing:
apt-get install linux-image-xen-amd64 grub
 configuring grub-pc: selected /dev/xvda (not a good idea, perhaps)
apt-get purge grub
apt-get install grub-legacy
cd /boot/grub
update-grub
set kernel for pv-grub-x86_64

Note: I was going to try it with 2.6.35-rc3 from kernel.org as well, but ended up bricking it and couldn't be arsed to fix it

While I have the spare node, I think I'll try Fedora 13 too... I will report.

Fedora's clean, too:

Deployed linode54174 97.107.134.59 with Fedora 13
Assigned host: newark111

*** TEST 1: Fedora 13 32-bit, 2.6.32.12-linode25
$ time perl many-digs.pl ; cat /proc/uptime ; uname -a
Files=100, Tests=500,  3 wallclock secs ( 0.47 usr  0.55 sys +  1.82 cusr  5.17 csys =  8.01 CPU)
Result: PASS

real   0m3.519s
user   0m2.544s
sys   0m5.758s
563.34 4309.10
Linux li64-59 2.6.32.12-linode25 #1 SMP Wed Apr 28 19:25:11 UTC 2010 i686 i686 i386 GNU/Linux

*** TEST 2: Fedora 13 32-bit, 2.6.34-linode26
Files=100, Tests=500,  4 wallclock secs ( 0.47 usr  0.55 sys +  1.65 cusr  5.12 csys =  7.79 CPU)
Result: PASS

real   0m4.792s
user   0m2.365s
sys   0m5.718s
28.88 197.81
Linux li64-59 2.6.34-linode26 #2 SMP Wed Jun 23 18:47:16 UTC 2010 i686 i686 i386 GNU/Linux

*** TEST 3: Fedora 13 32-bit, 2.6.33-linode24
Files=100, Tests=500,  9 wallclock secs ( 0.45 usr  0.53 sys +  1.57 cusr  4.93 csys =  7.48 CPU)
Result: PASS

real   0m9.166s
user   0m2.264s
sys   0m5.521s
21.44 139.16
Linux li64-59 2.6.33-linode24 #1 SMP Wed Feb 24 22:02:52 UTC 2010 i686 i686 i386 GNU/Linux

*** TEST 4: Fedora 13 32-bit, 2.6.32-linode23
Files=100, Tests=500,  4 wallclock secs ( 0.42 usr  0.55 sys +  1.82 cusr  5.01 csys =  7.80 CPU)
Result: PASS

real   0m4.211s
user   0m2.491s
sys   0m5.609s
12.74 71.70
Linux li64-59 2.6.32-linode23 #1 SMP Sat Dec 5 16:04:55 UTC 2009 i686 i686 i386 GNU/Linux

*** TEST 5: Fedora 13 32-bit, Latest 2.6 Stable (2.6.18.8-linode22)
<Did not boot (not unexpected)>


Methodology:
yum update
yum install git-core
yum install perl-Test-Harness
git clone git://gist.github.com/449825.git;cd 449825
note: edited many-digs.pl to use grep -q linode.com instead of grep -q theshore.net

I dont think anybody has explicity said it's a Linode issue, just that it's likely a kernel + Xen issue.

OverlordQ and Azathoth: Since you're having this issue too can you post more info about what distribution version / versions of critical software you're running? Here are my versions:

# A paravirt kernel
$ uname -r

2.6.32.12-x86_64-linode12

# Tracking Debian testing
$ cat /etc/debian_version 
squeeze/sid

$ dpkg -l |grep libc6
ii  libc6                                2.11.2-2                       Embedded GNU C Library: Shared libraries
ii  libc6-dev                            2.11.2-2                       Embedded GNU C Library: Development Libraries and Hea

I'm able to say that this is limited to x86_64 kernels. Thoughts?

I will be reaching out to the Xen folks today, and seeing if it's an issue there or an issue from upstream. Any anecdotes welcome.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

Might want to poke that kernel bug I opened as well, let them know

Much has gone on behind the curtain! There was testing, and bisecting, and finally an anticlimactic three-line commit. We're thinking this is the one \o/. Kudos to Ævar for making my commit message moar better and help with testing the patch.

We're rolling a 2.6.34 with the change applied to see if it does the trick. I'd ask that you all give it a go when we have it ready...I haven't been able to make the problem appear on 2.6.35-rc5+ with the patch applied.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

Jed did a great job tracking this down. I compiled a custom kernel on my Debian testing machine with his patch, it completely solved the issue.

I wrote a little howto for myself explaining how to update the kernel on Debian testing. It compliments the existing pv-grub documentation. But you should probably not use it unless you're really excited to try this out, otherwise just wait for the Linode kernel.

2.6.34-x86_64-linode14 is out, which contains the potential fix for the time accounting weirdness. Please give it a try and let us know!

Thanks,
-Chris