This is the console response when I try a Lish Ajax console login:

TCP cubic registered                                                                                
Initializing XFRM netlink socket                                                                    
NET: Registered protocol family 10                                                                  
ip6_tables: (C) 2000-2006 Netfilter Core Team                                                       
IPv6 over IPv4 tunneling driver                                                                     
NET: Registered protocol family 17                                                                  
NET: Registered protocol family 15                                                                  
Bridge firewalling registered                                                                       
Ebtables v2.0 registered                                                                            
Registering the dns_resolver key type                                                               
registered taskstats version 1                                                                      
XENBUS: Device with no driver: device/console/0                                                     
md: Waiting for all devices to be available before autodetect                                       
md: If you don't use raid, use raid=noautodetect                                                    
md: Autodetecting RAID arrays.                                                                      
md: Scanned 0 and added 0 devices.                                                                  
md: autorun ...                                                                                     
md: ... autorun DONE.                                                                               
REISERFS warning (device xvda): super-6502 reiserfs_getopt: unknown mount option "nobarrier"        
EXT3-fs: barriers not enabled                                                                       
kjournald starting.  Commit interval 5 seconds                                                      
EXT3-fs (xvda): mounted filesystem with writeback data mode                                         
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.                                       
devtmpfs: mounted                                                                                   
Freeing unused kernel memory: 668k freed                                                            
Write protecting the kernel read-only data: 10240k                                                  
Freeing unused kernel memory: 84k freed                                                             
Freeing unused kernel memory: 1356k freed                                                           
init: udevtrigger main process (1203) terminated with status 1                                      
init: udevtrigger post-stop process (1204) terminated with status 1                                 
init: udevmonitor main process (1202) killed by TERM signal                       

SOMEONE must know what that means SURELY? especially the last three messages relating to terminated processes?

What can be causing the processes to be terminated during a login - that dosen't look right (even to a neophyte like me)

The upgrade was interupted while it was making changes to your filesystem, so your server was left in an undefined state and now it doesn't boot -- hence no access.

udev, the device manager that creates the device nodes in /dev, is hosed. Try this:

Boot into rescue mode, then run:

mkdir -p /mnt/rescue
mount /dev/xvda /mnt/rescue
chroot /mnt/rescue /bin/bash
dpkg --configure -a

Reboot the server normally.

_________________
/ Peter

I didn't post that morpheous did your hvc0.conf file is exactly how it should be.

You're running an old linode kernel but a linode one none the less (you should switch to 3.2 when this is all fixed)

Also you should create a backup plan for the future just in case something like this happens again.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Peter, I am forever indebted to you!. Thanks for your clear, simple and straight forward explanation and solution. The server is up and running again, I am beside myself with joy, and have gained more fear (awe?) of the mysterious inner workings of the Linux OS and its sysadmins!

Thank you so much for helping me out of the hole I inadvertently dug myself into!.

I'm pleased that you're fixed. You should run (most of) the rest of the code from drpks' solution to finish cleaning up the mess:

sudo apt-get -f install
sudo apt-get --fix-missing install
sudo apt-get update
sudo apt-get upgrade

Reboot again. (Don't issue reboots from within your Linode -- they don't work as expected because there is code on the host that doesn't get invoked unless you reboot from the control panel or Lish.)

Save that trick in your toolbox -- chrooting into a busted system is one of the most powerful ways of fixing things I know.

_________________
/ Peter

It's good to hear that you were able to recover. You may find that some additional preparation eases the risk in future upgrades, since there's always a risk of failure notwithstanding your "pulling the plug", so to speak, in this case :-)

For example, if you have enough spare disk space in your Linode, you can clone your system image before attempting something like the upgrade, so worst case you just revert and start over. If you don't have enough space, you can clone to a temporary Linode to act as backup during the process. In fact, before doing significant upgrades (even application ones), I'll sometimes clone to a new Linode and experiment with the upgrade on that Linode (while the production system keeps running) just to feel good about the process.

If you have backups enabled on your Linode, you can also use snapshots for this purpose, or a snapshot/restore to set up the clone Linode for testing without needing any downtime on the production system.

I'd probably also run any large system upgrade beneath "screen" just in case you lose your network connection, there won't be any risk of blocking or interrupting the process.

-- David

That sounds like a smart thing to do. I don't as yet have backup installed, but I do have some spare disk space and I very much like the idea of cloning a system or even cloning a linode to act as a backup during an install or upgrade.

I remember investigating cloning a while back, but it seemed quite complicated/convoluted to me at the time. Could you please point me to some docs or a gentle tutorial that shows how I may do that. I suspect that I may be more "motivated" to learn how to do that after the debacle today!.

I don't know if there's any documentation in the library, but it's awfully simple. If you have the disk space, just give it a shot.

(edit: There are some documents in the library - try http://library.linode.com/linode-platfo ... isk-images and http://library.linode.com/linode-platfo ... one-linode)

If you just want to duplicate a disk image locally, click on the image and click the duplicate button.

To clone part or all of a Linode to another Linode, use the Clone tab, then select a profile (which includes the associated disk images) or selected images. You can select what other Linode to duplicate to.

Neither operation is destructive to the original profile/disk image, so safe to experiment with. You should, however, have your Linode shut down while doing the duplication/clone to ensure the integrity of the copy (something that isn't as necessary with a backup snapshot since that uses filesystem snapshots, though you may still need to worry about things such as databases).

-- David