[
I'm posting this a bit after the event, so will add comments like this.
]

I have managed to trash my Debian operating system and I hope to be able to get out of my predicament.

I can still get at it and look at and look around (using ssh myname@host40.linode.com), but it is unlikely that I will be able to do anything as sophisticated as setting up an ftp server ever again. My Debian packages are messed up and the root cause of this seems to be that once a low-level package has got itself into the state where it is "not fully installed or removed", then very few programs can run. And the more you try to fix the problem with various hard core dpkg commands, the worse it gets.

For other people that have got themselves into all sorts of multi-sourced debian pickles take a look at this pitiful collection:

http://www.google.com/search?hl=en&lr=&as_qdr=all&q=debian+%22packages+not+fully+installed+or+removed%22+fix&btnG=Search

In the above list you won't find any magic solutions. I know this even though I only read the first two links. Sometimes you just know when to give up!

I first noticed I had a problem when trying to install the latest version of Tomcat (not as a debian packge). Working through the install I noticed that autoconf was missing. Trying to 'apt-get install' it, I realised that the state of the packaging system had got into a tangle.

From that point on I can't point out just one command that really hosed my operating system, but this one stands out as making a significant contribution:

http://linux.derkeiler.com/Mailing-Lists/Debian/2003-11/2599.html

I discovered a little bit late a few comments around the place to the effect that that "once you go from a stable system to a mixed system, then it is very difficult to get back".

I did a number of silly things in a row. You will have to read to near the end to get to the sillyest. I started off well enough by following Harry Sufehmi's instructions from here:

http://www.harrysufehmi.com/phpwiki/index.php/SettingUpLinuxServer

I also used these instructions:

http://wiki.gednet.com/DebianServerSetup

All good. Unfortunately stable didn't seem to me to have some pretty standard stuff, and it looked like I would have to be installing many programs by hand. I don't remember what the catalyst for leaning towards unstable was. One thing was that there seemed to be a derth of stable Java packages. Another was that I was trying desperately to get hold of a decent package for qmail, to avoid an arduous looking manual install. Anyway I happened upon this little discovery:

http://jaqque.sbih.org/kplug/apt-pinning.html

For a while it seemed to do the trick. I got qmail working exactly as I needed it to without reading everything in 'qmail Rocks' and/or 'Life with qmail'. However it did take some time, perhaps because I didn't read what the package maintainer had to say in the file /usr/doc/qmail/README.debian, or perhaps because I didn't get the latest version.

Currently 1.03-24 is the stable version of qmail:

http://packages.qa.debian.org/q/qmail.html

On my Linode /var/lib/dpkg/status has the entry 'Config-Version: 1.03-25'. Thus I must have got the package I installed from a non-stable (ie. other than from stable, in fact most certainly from testing) set of debian packages. Had I known what I was doing I could have done better. Not really understanding my environment I missed a much more recent version number, 1.03-36, which is the version that exists in testing. Unfortunately I have only just discovered this link to the Debian Package Tracking System. In the future I will be able to manually confirm that apt-get has actually found me the most recent stable/testing package.
[
In fact 1.03-25 came from this line in sources.list
deb http://packages.dotdeb.org ./

1.03-36 comes quite orthodoxically from a sarge package called qmail-src. Once you've got the .deb file it works reasonably well with the packaging system, and it sets up a few things for you, so its basically as easy to work with as 1.03-25 was.
]

When I get Debian up and running again it will be with one source only, and I think that testing should fit the bill. I had noticed that alot of people were using Sarge. Another link I wish I had found a little earlier is this one:

http://www.linode.com/forums/viewtopic.php?t=1179&highlight=mail+server

Now that I have had time to reflect, I can see that mixing different sources (especially stable and unstable) together was always going to end up with an unholy mess. One network of dependencies is enough to manage, even for software on its own. Once you add people to the system, both package maintainers and end users like myself, then you are asking for trouble.

Part of the definition of unstable is that the package dependencies have not yet been tested much. The 'apt-pinning' link was possibly a little bit irresponible by the author: there were no caveats on that page. The title of this one, however, takes it into the genre of oxymoronic jokes:

http://lists.debian.org/debian-user/2002/02/msg01764.html

Having stuffed everything up, what is the worst thing I could do next? (This is the sillyest part I promised you). I was frustrated and very tired and I .......... rebooted the machine. Before reboot I had a friendly outward-going Linode that was completely unaware of what was going to happen to it after its next sleep. After reboot I had an autistic, very confused stranger on my hands. In fact it would be fair to say that it never really properly woke up again.

I can go in and see what a mess my machine is, but I just can't make it communicate. I can go in for a backed up tar of the important configuration parts of the machine (/etc etc.), but I have no way of sending the information outward.

Just in case you haven't had enough, I have one last mistake to admit to. When I first got the Linode I partitioned the whole of my disk image, even though the machine was only ever going to use a small portion of it.

My current thinking is that I will need to hire just enough disk space to install a fresh distribution. From this new fully functional machine I will be able to mount the trashed disk, and simply copy across whatever I need. When the new machine is running happily for a while I will be able to really trash the disk, burrying the past behind me.
[
See postscript.
]

I will finish with a few commands from the console:

mydomainname:/var/lib/dpkg# apt-get install ftp
Reading Package Lists... Done
Building Dependency Tree... Done
You might want to run `apt-get -f install' to correct these:
Sorry, but the following packages have unmet dependencies:
  ttf-bitstream-vera: Depends: defoma but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

mydomainname:/var/log# apt-get -f install
Reading Package Lists... Done
Building Dependency Tree... Done
Correcting dependencies... Done
The following extra packages will be installed:
  dialog
The following packages will be REMOVED:
  ttf-bitstream-vera
The following NEW packages will be installed:
  dialog
0 packages upgraded, 1 newly installed, 1 to remove and 0  not upgraded.
7 packages not fully installed or removed.
Need to get 0B/104kB of archives. After unpacking 445kB will be freed.
Do you want to continue? [Y/n] Y
/usr/sbin/dpkg-preconfigure: ff: command not found
(Reading database ... 8193 files and directories currently installed.)
Removing ttf-bitstream-vera ...
/usr/bin/defoma-font: ff: command not found
dpkg: error processing ttf-bitstream-vera (--remove):
 subprocess pre-removal script returned error exit status 127
Errors were encountered while processing:
 ttf-bitstream-vera
E: Sub-process /usr/bin/dpkg returned an error code (1)
mydomainname:/var/log#

All comments appreciated.

- Chris Murphy

POSTSCRIPT
I had no trouble creating new Configuration Profiles (which you do through the Distribution Wizard) and working with different Hard Drive Images depending on what I was doing at the time. For example after I had decided to go with Sarge I effectively created a new machine that could still look at the old machine mounted at /oldimage. And then when I felt that I was about to do something potentially dangerous (Debian package wise) I just made a copy of the disk in its good state so I could go back to it if needed. In this respect Debian and Linode are made for each other.[/b]

Heh, well... been there, done that. On the bright side, you've learned a lot, so that's good. You'll never forget it, ever.

I actually came very, very close to a situation like this with my home Gentoo box the other week because I had accidentally let a few non-stable packages infiltrate the system (in much the same way you described it).

Only thing that saved my bacon with Gentoo was realizing what happened then doing a massive 36 hour long rebuild of every single package on the system to get them back on the stable track. I was just lucky enough the system was still usable enough for compiles. Had it not been the case, I'd have to reinstall. Lesson learned!

I'm sorry you had to go through this suckiness... but thanks for sharing so that others will hopefully learn as well.

How big is the tar file of saved key files/dirs?

One idea would be to purchase extra disk quota - say, an extra 1 GB of disk space (assuming the tar file is less than 1 GB). Then use that space to create an empty ext3 disk image, and assign it to your Linode's profile.

Boot the broken Linode. Mount the blank disk image. Copy the tar file to it. Dismount disk image. Shut down the Linode.

Delete the broken Linode image, and recreate one with appropriate sizing, and reinstall Debian using the Distribution Wizard.

Then once the reinstalled Debian is running, mount the disk image with the tar file, and extract from it to some temporary directory on the Linode. Unmount the disk image.

That would be a cheaper way of getting back up quickly, rather than buying a couple of gigabytes of extra disk space.

The risk is, of course, that you didn't catch everything important with the tar cvpf <file>.tar <dirs> command with your current broken Linode.

Not much I can tell you other than you probably want to, at a minimum, tar up /etc, and any other directories where you have unique data (maybe your home directory? web directory?).

Otherwise, if you prefer to make 100% sure you didn't forget anything, then yeah, your idea for recovery would be better, though more expensive.

Hi Tronic,

I'm now very confident with having basically lots of machines at my disposal, the only limitation being one IP address and only up at any point in time - although I can have the others (such as the failed one) mounted so I can compare and copy across config files as I go. I don't need to do any tarring or anything more complex than working thru the Linode web pages (Config Wizard etc), and then mounting disks and occassionally changing fstab. That is what I was trying to say in the postscript.

I would suggest to anyone starting out with Linodes that they do some little experiments for starters - it will give them alot of confidence that basically you can't loose your machine. Say create two distributions, but both debian. They are now called configurations. Then for each you could change the configuration and have have an fstab such that another disk could be mounted at the directory /othermachine. Thus you could boot into either and whilst you are there check out the other. This is actually easier to do than talk about!

I did pay for an extra 1G a month that I don't need. Instead of this I could have simply reduced the size of the disk I have.

You do get a bit of a good feeling being able to do things easily that with real machines would be a PITA. I also mentioned in the postscript saving a disk. Of course when my machine has to be up all the time things will get a little more complex, and that's maybe what you were talking about with tarring etc.

I should have made it clearer that the post was already history, and that the problem had been fixed.


On debian you would mean the installed packages. Not big, not many. Whole machine was < 1Gig.


Did this and basically the other things you mentioned, without the tarring. Paid for the 1Gig, but didn't need to.

thanks for your support - Chris

I'm happy with Sarge

But keep in mind that Debian testing will get security updates last.

Both stable and unstable will get security updates before testing. Something I didn't fully understand when I initially chose testing (Sarge).

Mixing testing with unstable is not nearly as bad as mixing stable with unstable.

In the near future, the Debian testing might get support from security team according to a recent interview. If this happens, then I think testing will become the most popular. I believe unstable is currently the most popular in terms of sheer number of users.