At some point today, which I'll try to narrow down in a bit, host56 will be rebooted to bring it into sync with the rest of our Xen beta boxes.

I'll let you know when as we close in on getting ready.

Sorry for the short notice.

-Chris

I'm going to do this at 2:00 PM Eastern (about 1.5 hours from now).

-Chris

Upgrade complete, however there's one more fix we're working on. Should only be a few minutes more.

-Chris

Argh, up, down, up, down again

Same here. Good to know it's not just me, at least.

Ah, and now I see the "Lassie intiated boot" jobs with the xen_linode_boot error messages in my dashboard. Godspeed, caker.

Yeah I don't know exactly what those xen errors mean but I'm thinking that's not a good thing.

Uh-oh?

xen_linode_boot: failed to get domid
xen_linode_boot: warning - li-network might not have ran

I am apparently experiencing the same problem as everyone else:

Repeating "Lassie initiated boot" jobs with "xen_linode_boot: failed to get domid / xen_linode_boot: warning - li-network might not have ran" messages.

It appears that the each Lassie job is entered about 10 minutes after the previous startup attempt was entered. It looks like the server is caught in a state where server start up fails but is logged as successful. This state then causes Lassie to issue a restart command, which continues the cycle of startup failures.

Or caker boots host56, sees the xen_linode_boot errors, says "Damn!", takes it down to try for a fix, then boots again, sees the same error message...

(and by "try for a fix" I mean he checks the eth cord and kicks the box a couple times )

Obviously this did not go as anticipated. Xen is refusing to boot guests on this host. It worked the first time around, but no more -- still digging through it.

Meanwhile, I'm migrating accounts to a working Xen box.

Updates here in a bit.

-Chris

Thanks for the update.

Nailed it. xenstore database corruption -- go figure.

Boot away, if I haven't beaten you to it.

-Chris