Linode Forum

[morpheous]

I am running a headless install of Ubuntu 10.0.4 LTS on a linode. I was carrying out an OS upgrade earlier on today. I SSH'd into the sever and typed the following commands:

sudo aptitude update
sudo aptitude safe-upgrade

Whilst the second command (sudo aptitude safe-upgrade) was running, I (foolishly) decided to shut the server down (to bring down Apache and any other daemons that may be running on the server), whilst the OS and other software was being upgraded.

I lost my SSH connection to the server (unsuprisingly), and the install was aborted by the server being shut down. After rebooting the server, I have been unbale to log into the server again.

Here is the console output when I attempt to login remotely using the Lish Ajax Console:

Code: XENBUS: Device with no driver: device/console/0
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
REISERFS warning (device xvda): super-6502 reiserfs_getopt: unknown mount option "nobarrier"
EXT3-fs: barriers not enabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (xvda): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
devtmpfs: mounted
Freeing unused kernel memory: 668k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 84k freed
Freeing unused kernel memory: 1356k freed
init: udevtrigger main process (1203) terminated with status 1
init: udevtrigger post-stop process (1205) terminated with status 1
init: udevmonitor main process (1202) killed by TERM signal

I thought there maybe something messed up with the filesystem, so I run fsck. Here is the output:

Code: root@hvc0:~# fsck -fy /dev/xvdb
fsck from util-linux 2.19.1
e2fsck 1.42-WIP (02-Jul-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb: 111349/1286144 files (6.5% non-contiguous), 1798314/5120000 blocks
root@hvc0:~#

AFAIK, this means that there is nothing wrong with the filesystem - so I have no idea what else to do. My server is currently unreachable, and I can't SSH in to try to rerun the install in case that was the cause of all of this.

Does anyone have any idea on what the issue could be and how I may resolve this and get the server back online?

[Guspaz]

You don't need to stop any daemons to do an upgrade; the package manager will handle all that for you.

You can use lish to access the linode directly:

http://library.linode.com/troubleshooting/using-lish-the-linode-shell

[morpheous]

Guspaz wrote: You don't need to stop any daemons to do an upgrade; the package manager will handle all that for you.

You can use lish to access the linode directly:

http://library.linode.com/troubleshooting/using-lish-the-linode-shell

Thanks for the link. I managed to login using Lish shell. However, the console output is exactly the same as it was for the AJAX console. i.e. I am logged in as root@hvc0. Normally, I am logged in as myself@linode123 - so I still can't login into my server - or at least, I don't know how to get access to my server (as either one of my accounts on the server or root on that server) - so the server is still down.

[drpks]

Try:
Code: sudo dpkg --configure -a
sudo apt-get -f install
sudo apt-get --fix-missing install
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo reboot

Source: https://answers.launchpad.net/ubuntu/+source/update-manager/+question/154945

[morpheous]

Just to clarify, when I attempt to access the server, I don't even get a login prompt, so I CAN'T LOGIN TO THE SERVER.

This is the console output:

Code: md: ... autorun DONE.                                                                               
REISERFS warning (device xvda): super-6502 reiserfs_getopt: unknown mount option "nobarrier"       
EXT3-fs: barriers not enabled                                                                       
kjournald starting.  Commit interval 5 seconds                                                     
EXT3-fs (xvda): mounted filesystem with writeback data mode                                         
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.                                       
devtmpfs: mounted                                                                                   
Freeing unused kernel memory: 668k freed                                                           
Write protecting the kernel read-only data: 10240k                                                 
Freeing unused kernel memory: 84k freed                                                             
Freeing unused kernel memory: 1356k freed                                                           
init: udevtrigger main process (1203) terminated with status 1                                     
init: udevtrigger post-stop process (1206) terminated with status 1                                 
init: udevmonitor main process (1202) killed by TERM signal                                         


I am surprised at how difficult it is proving to be just to get access to the server after an aborted upgrade...

[obs]

Are you running a linode kernel or a distro/custom kernel?

If you're running a distro/custom kernel you may need to put
Code:
# hvc0 - getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -8 38400 hvc0
In /etc/init/hvc0.conf

[morpheous]

drpks wrote: Try:
Code: sudo dpkg --configure -a
sudo apt-get -f install
sudo apt-get --fix-missing install
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo reboot

Source: https://answers.launchpad.net/ubuntu/+source/update-manager/+question/154945

Thanks for the info. However, in order to type those command, I first need to login to the server. At the moment, I am not even being presented with a login screen, so I can't login to the server in order to type those commands.

HTH

[obs]

Reboot into rescue mode, mount the drive and then edit it there see here http://library.linode.com/troubleshooting/finnix-rescue-mode

[morpheous]

obs wrote: Reboot into rescue mode, mount the drive and then edit it there see here http://library.linode.com/troubleshooting/finnix-rescue-mode

Hi, thanks for your input. The problem is that I am not familiar with a lot of the terminology being used. Although I'm a software architect/developer, I have ZERO sysadmin skills, so the statement "mount the drive" unfortunately, dosen't relay much information to me.

I am aware that the 'mount' command is used to mount devices/drives, but thats about it. More specifically, I dont know which drive it is I am supposed to mount or indeed how to find out the devices/drives on my linode.

I did spend a fair bit of time this afternoon (several hours actually) in the finnix rescue mode (the console output is shown in one of my earlier messages). The net result is that when I login the rescue mode, I am logged in as root@hdvc0 (or something similar).

I typed ls, and it reported 0 files. The server has been down since early this morning (over 10 hrs ago) and I am still unable to even log into the server.

To say that I am getting slightly frustrated would be a gross understatement. Having said that, I fully appreciate that you are all trying to help me - of your own volition, so I will try my best not antagonize anyone.

Thank you all for your helpful feedback.

[obs]

I assume you're using xvda for your root drive you probably are, boot into rescue mode and type
Code: mkdir -p /mnt/rescue
mount /dev/xvda /mnt/rescue
nano /mnt/rescue/etc/init/hvc0.conf

Then paste what I posted and press ctrl+x then y

This will make the directory /mnt/rescue, mount /dev/xvda on /mnt/rescue then edit /etc/init/hvc0.conf in the mounted volume

[Mr Nod]

Have you raised a ticket with support?

I know this is an unmanaged service, but I should imagine it's something that support will be familiar with and could help you correct it very quickly.

[morpheous]

Hi obs,

Thanks for your help. I typed the commands you suggested. However when attempting to mount, the system complained that the device looked like a swap disk and faield to mount. This is correct however, as I remembered that I set /dev/xvda as my swap space. I tried the command then with /dev/xvdb instead (which is where my data resides), and I was able to proceed to open the .conf file with the nano editor.

However, the (hvc0.conf) file is not empty. It contains the following lines:

# hvc0 - getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -8 38400 hvc0


I thought it best to ask whether to:

1. Overwrite the contents of the file entirely with the new commands
2. Add the new commands to the BEGINING of the file OR
3. Add the new commands to the END of the file

I look forward to your response, and once again, thanks for your help. At least, now I feel I am making some progress.

[morpheous]

Mr Nod wrote: Have you raised a ticket with support?

I know this is an unmanaged service, but I should imagine it's something that support will be familiar with and could help you correct it very quickly.

Yes, I raised a support ticket early this morning. Support made a few suggestions (Reboot in Rescue mode and check with fsck). After that failed to resolve the issue, support suggested that I come in here and see if I could get some help from the community.

[obs]

The contents are the same, sounds like you're using a distro/custom kernel, reboot using a Linode kernel see if that boots.

[morpheous]

obs wrote: The contents are the same, sounds like you're using a distro/custom kernel, reboot using a Linode kernel see if that boots.

We may be talking at cross purposes here. The 'content' I was about to insert in the .conf file is:

Code: sudo dpkg --configure -a
sudo apt-get -f install
sudo apt-get --fix-missing install
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo reboot

Which is what you previously suggested. This is however different from the contents of the hvc0.conf file. Quite clearly, I had misunderstood your previous instruction - I'm glad I decided to double check before going ahead with replacing the file contents.

On the matter of a custum distro, I don't think this is the case. I am running a 64bit version of the Ubuntu 10.0.4 LTS. I made this choice so that it is compatable with my local dev machine, and so that C++ applications I have written locally, can be deployed to run on the server.

Last but not the least, I assume that the statement "reboot using a Linode kernel see if that boots" means to stop the server (running in safe mode) and rebooting normally (since I don't have a custom distro).

I double checked my linode setting configuration, and it seems I may have given you incorrect information about the device mappings. I include a snapshot of my configuration below (hopefully, it helps someone notice why this situation has arisen).

[img]http://dug.im/af94c[/img]