Hi All,

I think my Linux (CentOS release 4.4 (Final) Kernel 2.6.19) died. A week ago I received this error:

/etc/cron.weekly/yum.cron:

rpmdb: page 8: illegal page type or format
rpmdb: PANIC: Invalid argument
error: db4 error(-30978) from dbcursor->c_get: DB_RUNRECOVERY: Fatal error, run database recovery
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from dbenv->close: DB_RUNRECOVERY: Fatal error, run database recovery

Then, in less than two hours I received this:

/etc/cron.daily/yum.cron:

rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30978)
error: cannot open Packages database in /var/lib/rpm
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 97, in main
    result, resultmsgs = do()
  File "/usr/share/yum-cli/cli.py", line 477, in doCommands
    return self.updatePkgs()
  File "/usr/share/yum-cli/cli.py", line 957, in updatePkgs
    self.doRpmDBSetup()
  File "__init__.py", line 187, in doRpmDBSetup
  File "__init__.py", line 78, in addDB
TypeError: rpmdb open failed
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30978) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30978)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 80, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 170, in getOptionsConfig
    self.doConfigSetup(fn=opts.conffile, root=root)
  File "__init__.py", line 82, in doConfigSetup
  File "config.py", line 273, in __init__
  File "config.py", line 385, in _getsysver
TypeError: rpmdb open failed

Since then I've been receiving this error so I logged in today to see what's going on. As you can suspect, I just made it worse. There were a lot of processes runnung so I figured it's time to restart. After restart my network doesn't work either. I checked the log and it appeard that all the errors start when the system enters run lvl 3:

Jan 28 14:13:25 localhost init: Entering runlevel: 3
Jan 28 14:13:26 localhost iptables: /etc/rc3.d/S08iptables: line 29:  1604 Done(1)                 lsmod 2>/dev/null
Jan 28 14:13:32 localhost named[1823]: zone lecws.com/IN: sending notifies (serial 2006122801)
Jan 28 14:13:26 localhost iptables: Flushing firewall rules:
Jan 28 14:13:26 localhost iptables:       1605 Segmentation fault      | grep -q ipchains
Jan 28 14:13:26 localhost iptables: /etc/rc3.d/S08iptables: line 34:  1606 Done                    /sbin/modprobe --version 2>&1
Jan 28 14:13:26 localhost iptables:       1607 Segmentation fault      | grep -q module-init-tools
Jan 28 14:13:26 localhost iptables:  succeeded
Jan 28 14:13:26 localhost iptables: [
Jan 28 14:13:26 localhost iptables:
Jan 28 14:13:27 localhost iptables: Setting chains to policy ACCEPT: raw nat mangle filter
Jan 28 14:13:27 localhost iptables:  failed
Jan 28 14:13:27 localhost iptables:
Jan 28 14:13:27 localhost iptables: Unloading iptables modules:
Jan 28 14:13:27 localhost iptables: Opening /proc/modules: No such file or directory
Jan 28 14:13:27 localhost iptables: /etc/rc3.d/S08iptables: line 47:  1646 Segmentation fault      grep -q "^${mod}" /proc/modules
Jan 28 14:13:27 localhost iptables: Opening /proc/modules: No such file or directory
Jan 28 14:13:27 localhost iptables: /etc/rc3.d/S08iptables: line 47:  1652 Segmentation fault      grep -q "^${mod}" /proc/modules
Jan 28 14:13:27 localhost iptables:  succeeded
Jan 28 14:13:27 localhost iptables:
Jan 28 14:13:27 localhost iptables: Applying iptables firewall rules:
Jan 28 14:13:27 localhost iptables:  succeeded
Jan 28 14:13:27 localhost iptables: [
Jan 28 14:13:27 localhost iptables:
Jan 28 14:13:27 localhost rc: Starting iptables:  succeeded
Jan 28 14:13:27 localhost network: Setting network parameters:
Jan 28 14:13:28 localhost sysctl: net.ipv4.ip_forward = 0
Jan 28 14:13:28 localhost sysctl: net.ipv4.conf.default.rp_filter = 1
Jan 28 14:13:32 localhost named: named startup succeeded
Jan 28 14:13:28 localhost sysctl: net.ipv4.conf.default.accept_source_route = 0
Jan 28 14:13:28 localhost sysctl: kernel.sysrq = 0
Jan 28 14:13:28 localhost sysctl: kernel.core_uses_pid = 1
Jan 28 14:13:28 localhost network: Setting network parameters:  succeeded
Jan 28 14:13:28 localhost network: [
Jan 28 14:13:28 localhost network:
Jan 28 14:13:32 localhost named: [
Jan 28 14:13:28 localhost network: Bringing up loopback interface:
Jan 28 14:13:28 localhost ifup: network-functions: line 39:  1686 Done                    basename $CONFIG
Jan 28 14:13:28 localhost ifup:       1688 Segmentation fault      | grep -q '[^g]-'
Jan 28 14:13:28 localhost ifup: network-functions: line 179:  1693 Done                    LC_ALL= LANG= ip -o link
Jan 28 14:13:28 localhost ifup:       1694 Segmentation fault      | grep -q $1
Jan 28 14:13:29 localhost ifup: network-functions: line 179:  1695 Done                    LC_ALL= LANG= ip -o link
Jan 28 14:13:29 localhost ifup:       1696 Segmentation fault      | grep -q $1
Jan 28 14:13:29 localhost ifup: ./ifup: line 240:  1697 Exit 71                 ethtool -i $DEVICE 2>/dev/null
Jan 28 14:13:29 localhost ifup:       1698 Segmentation fault      | grep -q "driver: bonding"
Jan 28 14:13:29 localhost ifup: ./ifup: line 258:  1699 Done                    iptables -L -n 2>/dev/null
Jan 28 14:13:29 localhost ifup:       1700 Segmentation fault      | LC_ALL=C grep -q RH-Lokkit-0-50-INPUT
Jan 28 14:13:29 localhost ifup: ./ifup: line 429:  1705 Broken pipe             LC_ALL=C ip addr ls ${REALDEVICE}
Jan 28 14:13:29 localhost ifup: Error adding address 127.0.0.1 for lo.
Jan 28 14:13:29 localhost ifup:       1706 Segmentation fault      | LC_ALL=C grep -q "${IPADDR}/${PREFIX}"
Jan 28 14:13:29 localhost ifup: RTNETLINK answers: File exists
Jan 28 14:13:29 localhost ifup: ./ifup: line 443:  1713 Exit 71                 ethtool -i $DEVICE 2>/dev/null
Jan 28 14:13:29 localhost ifup:       1714 Segmentation fault      | grep -q "driver: bonding"
Jan 28 14:13:29 localhost ifup: network-functions: line 39:  1721 Done                    basename $CONFIG
Jan 28 14:13:29 localhost ifup:       1724 Segmentation fault      | grep -q '[^g]-'
Jan 28 14:13:29 localhost ifup: /etc/sysconfig/network-scripts/ifup-post: line 85:  1737 Done                    iptables -L -n 2>/dev/null
Jan 28 14:13:29 localhost ifup:       1738 Segmentation fault      | grep -q RH-Lokkit-0-50-INPUT
Jan 28 14:13:29 localhost network: Bringing up loopback interface:  succeeded
Jan 28 14:13:29 localhost network: [
Jan 28 14:13:30 localhost network:
Jan 28 14:13:30 localhost rc: Starting network:  succeeded
...

Anybody has any ideas what could be the problem? Any help is appreciated.

Thanks,
LEC

By the way I did rename /lib/tls but it didn't help. I'm sill getting segfaults during bootup. Please someone help!

Thanks,
LEC

Wow, I guess this is a bigger problem than I thought. Hmmm, then I guess I have to reinstall...

Thanks,
LEC

The first set of error messages indicate that your package database is corrupt:

Fatal error, run database recovery

Your package manager (yum) should have an option to recreate the database from installed files. Find the option and run it.

The second set of error messages indicate there is something wrong with /bin/grep -- perhaps some auto-install process messed up with grep or a dependency. Try reinstalling /bin/grep (package: grep) with your package manager.

HTH,

Cliff

Thank you for your response. I looked up how to rebuild the rpm database but that didn't help. And unfortunately yum doesn't work without internet access so I cannot even remove grep.

Thanks,
LEC

Like this?

http://jimmetzger.net/sediments/2006/11 ... fixup.html

Yes, that's exactly what I found, on another site though. My network still doesn't start without grep and I can't run yum without a network. Catch 22...

Thanks,
LEC

This is what I get in /var/log/messages when I do a /etc/init.d/network restart:

Jan 31 20:30:37 localhost kernel: line_ioctl: tty0: ioctl TIOCLINUX called
Jan 31 20:30:37 localhost last message repeated 2 times
Jan 31 20:30:37 localhost ifdown: network-functions: line 39:  4661 Done                    basename $CONFIG
Jan 31 20:30:37 localhost ifdown:       4664 Segmentation fault      | grep -q '[^g]-'
Jan 31 20:30:37 localhost ifdown: network-functions: line 233:  4668 Done                    echo $1
Jan 31 20:30:37 localhost ifdown:       4669 Segmentation fault      | grep -q ':'
Jan 31 20:30:37 localhost ifdown: network-functions: line 233:  4670 Done                    LC_ALL=C ip -o link ls dev $1 2>/dev/null
Jan 31 20:30:37 localhost ifdown:       4671 Segmentation fault      | grep -q ",UP"
Jan 31 20:30:38 localhost ifdown: network-functions: line 233:  4677 Done                    echo $1
Jan 31 20:30:38 localhost ifdown:       4678 Segmentation fault      | grep -q ':'
Jan 31 20:30:38 localhost ifdown: network-functions: line 233:  4679 Done                    LC_ALL=C ip -o link ls dev $1 2>/dev/null
Jan 31 20:30:38 localhost ifdown:       4680 Segmentation fault      | grep -q ",UP"
Jan 31 20:30:38 localhost ifdown: network-functions: line 39:  4687 Done                    basename $CONFIG
Jan 31 20:30:38 localhost ifdown:       4689 Segmentation fault      | grep -q '[^g]-'
Jan 31 20:30:38 localhost ifdown: network-functions: line 293:  4693 Done                    LC_ALL=C ip route list match 0/0
Jan 31 20:30:38 localhost ifdown:       4694 Segmentation fault      | grep -q default
Jan 31 20:30:38 localhost ifdown: network-functions: line 293:  4695 Done                    LC_ALL=C ip route list match 0/0
Jan 31 20:30:38 localhost ifdown:       4696 Segmentation fault      | grep -q default
Jan 31 20:30:38 localhost network: Shutting down loopback interface:  succeeded
Jan 31 20:30:38 localhost kernel: line_ioctl: tty0: ioctl TIOCLINUX called
Jan 31 20:30:38 localhost sysctl: net.ipv4.ip_forward = 0
Jan 31 20:30:38 localhost sysctl: net.ipv4.conf.default.rp_filter = 1
Jan 31 20:30:38 localhost sysctl: net.ipv4.conf.default.accept_source_route = 0
Jan 31 20:30:38 localhost sysctl: kernel.sysrq = 0
Jan 31 20:30:38 localhost sysctl: kernel.core_uses_pid = 1
Jan 31 20:30:38 localhost network: Setting network parameters:  succeeded
Jan 31 20:30:38 localhost kernel: line_ioctl: tty0: ioctl TIOCLINUX called
Jan 31 20:30:38 localhost ifup: network-functions: line 39:  4733 Done                    basename $CONFIG
Jan 31 20:30:38 localhost ifup:       4736 Segmentation fault      | grep -q '[^g]-'
Jan 31 20:30:38 localhost ifup: network-functions: line 179:  4740 Done                    LC_ALL= LANG= ip -o link
Jan 31 20:30:38 localhost ifup:       4741 Segmentation fault      | grep -q $1
Jan 31 20:30:39 localhost ifup: network-functions: line 179:  4742 Done                    LC_ALL= LANG= ip -o link
Jan 31 20:30:39 localhost ifup:       4743 Segmentation fault      | grep -q $1
Jan 31 20:30:39 localhost ifup: Error adding address 127.0.0.1 for lo.
Jan 31 20:30:39 localhost ifup: ./ifup: line 240:  4744 Exit 71                 ethtool -i $DEVICE 2>/dev/null
Jan 31 20:30:39 localhost ifup:       4745 Segmentation fault      | grep -q "driver: bonding"
Jan 31 20:30:39 localhost ifup: ./ifup: line 258:  4746 Done                    iptables -L -n 2>/dev/null
Jan 31 20:30:39 localhost ifup:       4747 Segmentation fault      | LC_ALL=C grep -q RH-Lokkit-0-50-INPUT
Jan 31 20:30:39 localhost ifup: ./ifup: line 429:  4752 Done                    LC_ALL=C ip addr ls ${REALDEVICE}
Jan 31 20:30:39 localhost ifup:       4753 Segmentation fault      | LC_ALL=C grep -q "${IPADDR}/${PREFIX}"
Jan 31 20:30:39 localhost ifup: RTNETLINK answers: File exists
Jan 31 20:30:39 localhost ifup: ./ifup: line 443:  4760 Exit 71                 ethtool -i $DEVICE 2>/dev/null
Jan 31 20:30:39 localhost ifup:       4761 Segmentation fault      | grep -q "driver: bonding"
Jan 31 20:30:39 localhost ifup: network-functions: line 39:  4768 Done                    basename $CONFIG
Jan 31 20:30:39 localhost ifup:       4771 Segmentation fault      | grep -q '[^g]-'
Jan 31 20:30:39 localhost ifup: /etc/sysconfig/network-scripts/ifup-post: line 85:  4784 Done                    iptables -L -n 2>/dev/null
Jan 31 20:30:39 localhost ifup:       4785 Segmentation fault      | grep -q RH-Lokkit-0-50-INPUT
Jan 31 20:30:39 localhost network: Bringing up loopback interface:  succeeded

Ah, of course!

But there is still a way to (try to) rescue this.

Set up a Finnix profile in the LPM that includes your CentOS disk image(s) in the profile, and boot into that. Once booted, set up a chroot into your CentOS distro, enter it, and then run the yum/rpm db recovery process (above).

Then fix the rest of installation (i.e. broken grep), exit chroot and reboot into CentOS ... .


Cliff

Thank you for your response. Unfortunately, I'm not too Linux savy so I only got to the point where I have Finnix setup and I can mount the disk of my CentOS. Is there any instruction on how to setup chroot into my CentOS from Finnix? I searched the wiki page but it returns 0 results for the word finnix.

Thanks,
LEC

Assuming your CentOS disk is at /dev/ubdb and mounted at /mnt/ubdb (by Finnix):

mount /proc /mnt/ubdb/proc -t proc
chroot /mnt/ubdb /bin/bash

If successful, you will now have a root shell in your CentOS disk, along with network access from the Finnix boot. (Everything you do now is as if you were in a regular ssh environment in your CentOS.)

Do your repair job on the database, carry out any updates, and then make sure that your grep works properly.

echo hello > file.txt
grep hello file.txt
rm file.txt

If all is well your CentOS has been repaired and it is just a matter of issuing the reboot command via your LPM. (Make sure that CentOS, is the default profile, else an unscheduled reboot will put you back in Finnix.)

Good luck!

Cliff

Thank you Cliff. I did what you said but unfortunately it didn't work. First. Here's the full story.

When I chrooted to my old distro I was able to do "yum remove grep" which of course told me that there were 194 dependecies. So I googled around and found that I can remove the rpm entry and then I can install grep. So that's what I did. Unfortunately, this still didn't fix the segfaults. Since I also had a brand new install of CentOS (I figured I had to start over) I copied grep from the new distro to the old one. That fixed the segfault for grep. Cool! Reboot, same segfaults.

So finally, after many copy/reboot, I ended up copying the whole /bin dir to the old distro and that fixed the problem!
I still can't help but wonder what caused this problem. Can it be that the hard drive got damaged? Or the file system? Anybody has any ideas?