Linode just did a reboot of one of my Ubuntu servers as they needed to upgrade the underlying virtual server. However, the machine spent about 3.5 hours starting up afterwards (with a lot of io).
Just wondering how I might investigate what it was up to. The logfiles I've looked at seem to be a bit quiet on the matter, but I'm probably not looking at the right ones.

Here's the IO graph:

BTW, here's a bit from Syslog:

May 20 21:07:54 li347-225 init: cron main process (1975) killed by TERM signal
May 20 21:07:54 li347-225 init: hvc0 main process (2077) killed by TERM signal
May 20 21:07:54 li347-225 init: hwclock-save main process (21639) terminated with status 1
May 20 21:07:54 li347-225 kernel: Kernel logging (proc) stopped.
May 21 05:30:53 li347-225 kernel: imklog 4.2.0, log source = /proc/kmsg started.
May 21 05:30:53 li347-225 rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="3746" x-info="http://www.rsyslog.com"] (re)start
May 21 05:30:53 li347-225 rsyslogd: rsyslogd's groupid changed to 103
May 21 05:30:53 li347-225 rsyslogd: rsyslogd's userid changed to 101
May 21 05:30:53 li347-225 rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
May 21 05:30:53 li347-225 kernel: rack: expectation table full
May 21 05:30:53 li347-225 kernel: nf_ct_sip: dropping packetIN=eth0 OUT= MAC=f2:3c:91:df:6b:a7:88:43:e1:a3:fa:7f:08:00 SRC=86.135.1.25 DST=178.79.178.225 LEN=578 TOS=0x00 PREC=0x00 TTL=50 ID=31359 PROTO=UDP SPT=63171 DPT=5060 LEN=558
May 21 05:30:53 li347-225 kernel: nf_conntrack: expectation table full

An 8 hour gap in your syslog? What?

the underlying virtual server? There's a rumour that linode runs on a grid of 386s emulating a xeon, I think you just confirmed it.

There are two things I can think of, 1 fsck and 2. mysql in ubuntu/debian does some sort of database repair on start. They seem unlikely to take 4 hours though, but you never know with 40 linodes booting up at similar times.

Firstly, not sure where that TERM signal would have come from at that time, but that was suspiciously at the beginning of a period of silence - and it may well be that it wasn't really at 21:07 on 20th May (the clock sync seems to be correct, though - both now and at the end of last week).

Secondly, fsck (or similar) is high on my list of suspects, but /var/log/fsck/checkfs and /var/log/fsck/checkroot are empty. Having said that, it wouldn't surprise me if logging to a device you're trying to fix is a little on the tricky side...

BTW, this was the announcement I received:

The server where your Linode currently resides requires a reboot for maintenance. The following maintenance window has been scheduled:

Sunday, May 20th 2012 09:00PM - 12:00AM EDT
(Monday, May 21st 2012 1:00AM - 4:00AM GMT)

During this window your Linode will be gracefully shut down and rebooted. Downtime should be less than 30 minutes. Please ensure your services are correctly configured to start up on boot.

This maintenance is required due to a software issue, which our vendor suggested will be made public by US-CERT <https://www.us-cert.gov/> in a few weeks. Although we strive to keep disruptions to a minimum, this is a mandatory security update required to protect you and our other customers. We apologize for the inconvenience.

I think they're updating a bunch of servers. I also got that e-mail, however mine was for May 25.

Maybe a vulnerability they cannot tell us about yet?

I got a similar e-mail. Although the wording was slightly different in certain places.



The Linode this e-mail affects is in London. I also have two other Linodes in Dallas. I asked if they would be affected by this at some point, and support said no (interestingly enough).

maybe preparations for this year's Linode birthday upgrades.

Given the reference to CERT, it's almost certainly the former.

It is possible that your Linode is not setup properly to shutdown gracefully and when the server went down, you had your Linode go down uncleanly. With all the other Linodes starting up at the same time, it may have taken a lot longer to correct itself.

The only way that I know of you can test for that is to do another shutdown of your own and see what happens when it comes back up.

You might want to archive your log files first so that you can see what is getting reported when you do the shutdown etc...

_________________
Kevin a.k.a. Dweeber