Hey everybody,
I (tried) to upgrade my linode today from 9.10 to 10.04 Lucid. Unfortunately after the upgrade, most of the services aren't coming up. When I check the runlevel, it comes back with 'unknown'. I had to issue "telinit 2" to get the services to come up. However, things are still not 100%. Has anyone else seen this? A quick google search shows that a handful of people out there have experienced the same thing.

I am particularly aggravated at the fact that I can't boot into my backup linode disk image. Right before I started the upgrade process, I made a duplicate image of the root (xvda) image. When things went sideways, I changed the config profile and tried to boot into the backup image. The networking comes up (I can ping), but no SSH access. Has anyone observed that?

And my final rant/frustration is Lish. I was able to see the console via ssh and also AJAX in the linode manager. However, I can't get a login prompt. It has been a while since I used lish, but I thought it was interactive?

Any thoughts, suggestions, and/or nice words are appreciated. Thanks.

As Ubuntu's using some new funky sysvinit replacement (I think?), ca't help you with this... but lack of login prompt in LISH usually means your system isn't configured to launch getty on the Xen serial console (/dev/hvc0, IIRC. It's often remapped into /dev/tty1 by kernel boot params, but I think that doesn't work with paravirt kernels.)

Did you have only a "root" and "my data" partitions, or did you have /usr and friends separately too? Because in the latter case, it's quite probably the reason why you can't boot from the backup - services' binaries got upgraded too, after all.

I just had root and "my data" partitions. In theory, if you boot from a backup "root" image, it should just work right? I don't see what else would be required...

I deployed a new 10.04 to experiment. The console and came up and I was able to login. But once I configurated the hostname, firewall, and /etc/network/interfaces and then rebooted, again no console. So I am not sure what is going on there.

I would hate to rebuild this thing from scratch... but that may be the best way to handle it. I have done several upgrades with no problem in the past. Maybe its time to start "clean." But the console thing is really bothering me. I have no "out of band" access as it stands now.

I had a pretty bad experience with upgrading to Lucid too - the initial upgrades seemed to work, and other than forgetting to edit my fstab on the second upgrade, everything appeared fine.

Except that nothing seemed to work anymore, and for some reason beyond fathom they are now forcing you to use the 'service' command and denying /etc/init.d/ completely. Everytime I tried to reload, restart, start or stop a service it would just sit there. Even when using the switch to 'not wait for a response', the command just hung.

This was about the worst upgrade experience I've ever had with Ubuntu.

+1. I'm not yet willing to say I didn't screw it up somehow. However, I am starting to think this is related to the "Upstart" changes. I don't have a lot of background, but I guess it is supposed to handle the "old" init.d type stuff. I found some info on ubuntuforums that seems to point to a solution. We'll see what happens.

I have decided that I will not rebuild the server though. That will take too damn long. I have to get this fixed so that when I boot the server, everything comes up and its listening.

Thanks for the help so far.

I commented out the "post-up" lines in /etc/network/interfaces:

 
auto lo
iface lo inet loopback
#       post-up iptables-restore < /etc/iptables.up.rules
#       post-up echo 1 > /proc/sys/net/ipv4/ip_forward
#       post-up echo 1 > /proc/sys/net/ipv4/conf/tun0/forwarding

auto eth0
iface eth0 inet static

        address [REMOVED]
        netmask 255.255.255.0
        gateway [REMOVED]
        post-up ifconfig eth0:0 [REMOVED] netmask 255.255.255.0 up

That has gotten be back into runlevel 2 upon bootup. So apparently some of that post-up code is breaking the boot (upstart?) process. I still don't have a console login prompt though. And, obviously, I still need to make those lines of code work somehow...

Alright I am a bad system administrator.



Interface tun0 doesn't exist anymore. That was causing the problem. Actually, neither of those lines is needed for me anymore. They were required when I was using openvpn. But now that I am running openvpn-as, everything is automagic. So this seemed like a problem due to the upgrade, but it is actually my mistake. I just never reboot my server.... so it didn't show until the upgrade.

HOWEVER. I still don't have console!

I'm assuming you have the helpers enabled in the configuration profile, right? Barring that, on a "ps aux | grep getty", where is getty listening? It should be hvc0, I believe. Also make sure /dev/hvc0 does exist, just in case

Yes, helpers are enabled. ps aux gives me the following:

matt@host:~$ ps aux | grep getty
matt      2864  0.0  0.1   3324   796 pts/0    S+   14:28   0:00 grep getty

so it looks like init is once again failing to execute certain stuff. when i manually run the command in /etc/init/hvc0.conf

exec /sbin/getty 38400 hvc0

the console comes up. so something in the init process seems to not be going smoothly. any idea on where i can find some logging for this?

Ok I fixed the console problem the other day by changing /etc/init/hvc0.conf like so:

# hvc0 - getty for Linode console

start on (virtual-filesystems
          or starting rcS
          or starting mountall-shell)

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 hvc0

Apparently the original "start on runlevel 2" idea wasn't working out. I kindly suggest that the linode distro deployment guy look at testing this out. I am pretty sure this is broken out of the box.

This is now officially the worst OS upgrade I have ever experienced. I am finding bugs constantly. Now I am spending Saturday night trying to reinstate some system logging...