Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Did you survive the leapsecondapocalypse?
Poll ended at Tue Jul 03, 2012 2:26 pm
Nope, crashed  5%  [ 1 ]
Kinda, MySQL went berserk  11%  [ 2 ]
Yeah, all good  84%  [ 16 ]
Other  0%  [ 0 ]
Total votes : 19
Author Message
 Post subject: Leapsecondapocalypse!
PostPosted: Sun Jul 01, 2012 2:26 pm 
Offline
Senior Member

Joined: Mon Dec 07, 2009 6:46 am
Posts: 331
I was gloating because RHEL 6 seems to be unaffected by the "bug", and I run only CentOS 6, but I spoke too soon. Seems MySQL CPU went through the roof on a node running that db. What's your experience?


Top
   
PostPosted: Sun Jul 01, 2012 5:50 pm 
Offline
Senior Member

Joined: Wed Oct 20, 2010 12:35 pm
Posts: 111
Location: United Kingdom
I have a node running Debian 6 and was expecting to see some emails from Pingdom saying my site was down but it has been rock solid.


Top
   
PostPosted: Sun Jul 01, 2012 7:59 pm 
Offline
Senior Member

Joined: Sun May 23, 2010 1:57 pm
Posts: 315
Website: http://www.jebblue.net
My Ubuntu desktop and laptop both went haywire at the same time yesterday afternoon. MySQL, Google Chrome and home Minecraft Server all started using max CPU at the same time, had to reboot both. Also checked my Linode and MySQL was fine but I rebooted it anyway.


Top
   
PostPosted: Sun Jul 01, 2012 8:16 pm 
Offline
Senior Member

Joined: Sat Jun 12, 2010 4:53 pm
Posts: 77
Ubuntu 10.04 linode, no problems

Code:
$ uname -a
Linux li166-66 3.0.18-linode43 #1 SMP Mon Jan 30 11:44:09 EST 2012 i686 GNU/Linux


Code:
$ uptime
 17:15:38 up 111 days, 12:04,  1 user,  load average: 0.00, 0.01, 0.05


Top
   
PostPosted: Sun Jul 01, 2012 9:36 pm 
Offline
Senior Member
User avatar

Joined: Tue Apr 13, 2004 6:54 pm
Posts: 833
My Ubuntu 11.04 desktop needed rebooting 'cos VboxSVC was chewing up 110% of CPU. My CentOS 6 machines that ran qpidd. The CentOS 6 machines _without_ qpidd and the CentOS 5 machines were all fine.
Code:
linode.pts/0% uptime
 21:36:33 up 20 days,  8:11,  1 user,  load average: 0.00, 0.01, 0.05

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)


Top
   
PostPosted: Tue Jul 03, 2012 1:22 am 
Offline
Senior Member
User avatar

Joined: Tue Nov 24, 2009 1:59 pm
Posts: 362
My bunch of random Debian 6 boxes, both linodes and vmware, were OK. Two of them were running mysql.

_________________
rsk, providing useless advice on the Internet since 2005.


Top
   
PostPosted: Tue Jul 03, 2012 10:06 am 
Offline
Senior Member

Joined: Sat Nov 27, 2010 8:21 pm
Posts: 63
Had no issues here, but Hetzner reported some interesting info:

Quote:
During the night of 30.06.2012 to 01.07.2012 our internal
monitoring systems registered an increase in the level of
IT power usage by approximately one megawatt.

The reason for this huge surge is the additional switched
leap second which can lead to permanent CPU load on Linux
servers.


Top
   
PostPosted: Tue Jul 03, 2012 10:06 am 
Offline
Senior Member
User avatar

Joined: Tue May 26, 2009 3:29 pm
Posts: 1691
Location: Montreal, QC
Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)


Top
   
PostPosted: Tue Jul 03, 2012 1:12 pm 
Offline
Senior Member
User avatar

Joined: Thu Nov 24, 2011 12:46 pm
Posts: 139
Location: Mesa AZ
Completely clean across all platforms. Non-issue.

_________________
Kevin a.k.a. Dweeber


Top
   
PostPosted: Tue Jul 03, 2012 7:41 pm 
Offline
Senior Member

Joined: Sun May 23, 2010 1:57 pm
Posts: 315
Website: http://www.jebblue.net
Guspaz wrote:
Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)


I run all the latest, home and Linode. Even on the Linode kernel 3.4.2-linode44 I saw MySQL go max
CPU, same on my two home machines a desktop and a laptop, all running Ubuntu 12.04.

The only thing I can think of that was in common is that I was running something on Java on all 3. Why would _that_ impact MySQL? I read something about the JVM and MySQL both use a Linux thread thing called a futex. I don't program that low usually so I never used it.


Top
   
PostPosted: Tue Jul 03, 2012 9:04 pm 
Offline
Senior Member

Joined: Wed May 13, 2009 1:18 am
Posts: 681
Guspaz wrote:
Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)

Of course, ironically, this is a case where running an ancient kernel could have been more protective than the most current, since it could predate the introduction of the issue, which I believe was as of 2.6.22.

-- David


Top
   
PostPosted: Wed Jul 04, 2012 2:08 am 
Offline
Junior Member

Joined: Mon Jan 30, 2012 3:21 am
Posts: 29
Location: Glendale, CA
Interesting.

In my case I did not have an issue at the exact time this was supposed to happen (midnight GMT), or 5PM PST; yet I did have my CPU on my linodes and on my local dev/test box go really high [90%-95%] at midnight between 6-30-12 and 7-1-12 for about 90 seconds. I checked the graphs for the previous 4 months and did not see that happen previously. Running Fedora 15 3.4.2-x86_64-linode25 with the linodes and fedora 15 2.6.43 on the devs. MySQL version is "mysql Ver 14.14 Distrib 5.5.23, for Linux (x86_64) using readline 5.1"


Top
   
PostPosted: Wed Jul 04, 2012 2:22 am 
Offline

Joined: Wed Jul 04, 2012 2:21 am
Posts: 1
Yeah, both my linodes held up well. one arch one debian 6. my hetzner dedi started doing some weird things to it's gigabit connection however, two srcds instances and the minecraft server went haywire.


Top
   
PostPosted: Thu Jul 05, 2012 4:18 pm 
Offline
Junior Member

Joined: Sun May 27, 2012 6:35 am
Posts: 25
The Centos 6.2 box that we run cPanel websites on did not appear to suffer any problems, however the Munin graphs did show a small increase in average system load and a huge jump in committed_as memory occurring exactly at midnight. Over the next few days our external monitoring tool also showed that webpages were taking on average 50ms longer to load than the week before.

Setting the time manually from the command line immediately cleared the problem - average load and web page load times went back to the levels from the week before. (committed_as memory remains high, however).

So you may want to take a closer look at your system, even though you think everything is ok.


Top
   
PostPosted: Sat Jul 07, 2012 4:58 am 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
dovecot decided to commit suicided which required simply starting it again apart from that nothing crapped out...bugger of it was the leap second occurred at 1am where I am so pingdom work me up :(

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group