So in an apparent attempt at self-mutilation, I have decided to try my hand at migrating over a new CentOS 7 system and using encrypted volumes. I have everything copied over, have created the luks containers and so on, but when I try to boot I get dropped to the grub shell. Here are the configuration details.

I have the following device layout:
/dev/xvda - /boot - formatted as xfs
/dev/xvdb - swap - formatted as, well, swap, on top of luks
/dev/xvdc - / - formatted as xfs, on top of luks

/etc/cryptab (the UUID is from the unencrypted device)

crypt-xvdc              UUID=69371f88-53d0-4622-92f7-7fa8f8b31194               none                    luks
crypt-swap              /dev/xvdb                                                                               /dev/urandom    swap

/etc/fstab

/dev/mapper/crypt-xvdc /                       xfs     defaults,x-systemd.device-timeout=0 1 1
/dev/xvda /boot                   xfs     defaults        1 2
/dev/mapper/crypt-swap none                    swap    defaults,x-systemd.device-timeout=0 0 0
proc    /proc   proc    defaults

/etc/init/hvc0.conf:

# hvc0 - getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.
 
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
 
respawn
exec /sbin/getty -8 38400 hvc0

Grub file locations have been changed to account for mounting /boot directly under xvda.

root@hvc0:/media/xvda# ll
total 87084
-rw------- 1 root root  2841075 Aug  6 21:21 System.map-3.10.0-123.6.3.el7.x86_64
-rw------- 1 root root  2840084 Jun 30 12:17 System.map-3.10.0-123.el7.x86_64
drwxr-xr-x 3 root root       17 Sep  7 22:09 boot
-rw-r--r-- 1 root root   122063 Aug  6 21:21 config-3.10.0-123.6.3.el7.x86_64
-rw-r--r-- 1 root root   122059 Jun 30 12:17 config-3.10.0-123.el7.x86_64
lrwxrwxrwx 1 root root       10 Sep  7 22:09 grub -> boot/grub/
drwxr-xr-x 6 root root      104 Sep  7 17:07 grub2
-rw-r--r-- 1 root root 26468718 Sep  7 16:46 initramfs-0-rescue-99f4b8fcbd9d4075ba85e8fb70f2cb15.img
-rw------- 1 root root  9804482 Sep  7 17:07 initramfs-3.10.0-123.6.3.el7.x86_64.img
-rw------- 1 root root 10449065 Sep  7 18:02 initramfs-3.10.0-123.6.3.el7.x86_64kdump.img
-rw------- 1 root root 10301174 Sep  7 16:51 initramfs-3.10.0-123.el7.x86_64.img
-rw------- 1 root root 10447182 Sep  7 16:56 initramfs-3.10.0-123.el7.x86_64kdump.img
-rw-r--r-- 1 root root   589615 Sep  7 16:38 initrd-plymouth.img
-rw-r--r-- 1 root root   228612 Aug  6 21:23 symvers-3.10.0-123.6.3.el7.x86_64.gz
-rw-r--r-- 1 root root   228562 Jun 30 12:20 symvers-3.10.0-123.el7.x86_64.gz
-rwxr-xr-x 1 root root  4902656 Sep  7 16:47 vmlinuz-0-rescue-99f4b8fcbd9d4075ba85e8fb70f2cb15
-rwxr-xr-x 1 root root  4903968 Aug  6 21:21 vmlinuz-3.10.0-123.6.3.el7.x86_64
-rwxr-xr-x 1 root root  4902656 Jun 30 12:17 vmlinuz-3.10.0-123.el7.x86_64
root@hvc0:/media/xvda# ll boot/
total 0
drwxr-xr-x 2 root root 41 Sep  7 21:49 grub
root@hvc0:/media/xvda# ll boot/grub/
total 8
-rw-r--r-- 1 root root  351 Sep  8 02:46 menu.1st
-rw-r--r-- 1 root root 1350 Nov 15  2011 splash.xpm.gz

grub/menu.1st

timeout 5
title CentOS (3.10.0-123.6.3.el7.x86_64)
groot=(hd0)
kernel /boot/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=/dev/xvda
initrd /boot/initrd-plymouth.img

## ## Start Default Options ##
## default kernel options
## default kernel options for automagic boot options
kopt=root=/dev/mapper/crypt-xvdc cryptdevice=/dev/xvdc:crypt-xvdc console=hvc0 ro

logview log

Showing last 100 lines from current boot
-----------------------------------------
[3568101.500641] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[3568101.500686] TCP: Hash tables configured (established 16384 bind 16384)
[3568101.500729] TCP: reno registered
[3568101.500745] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[3568101.500762] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[3568101.500826] NET: Registered protocol family 1
[3568101.500887] Unpacking initramfs...
[3568101.502897] Freeing initrd memory: 1368k freed
[3568101.503237] platform rtc_cmos: registered platform RTC device (no PNP device found)
[3568101.503628] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x416
[3568101.503649] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x416
[3568101.503717] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[3568101.504121] futex hash table entries: 512 (order: 3, 32768 bytes)
[3568101.504148] Initialise system trusted keyring
[3568101.504240] audit: initializing netlink socket (disabled)
[3568101.504258] type=2000 audit(1410144607.447:1): initialized
[3568101.580683] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[3568101.581509] zbud: loaded
[3568101.581676] VFS: Disk quotas dquot_6.5.2
[3568101.581715] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[3568101.581892] msgmni has been set to 3985
[3568101.582008] Key type big_key registered
[3568101.583264] alg: No test for stdrng (krng)
[3568101.583281] NET: Registered protocol family 38
[3568101.583288] Key type asymmetric registered
[3568101.583292] Asymmetric key parser 'x509' registered
[3568101.583324] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[3568101.583371] io scheduler noop registered
[3568101.583375] io scheduler deadline registered (default)
[3568101.583403] io scheduler cfq registered
[3568101.583456] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[3568101.583471] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[3568101.583993] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[3568101.584467] Non-volatile memory driver v1.3
[3568101.584473] Linux agpgart interface v0.103
[3568101.584540] crash memory driver: version 1.1
[3568101.584555] rdac: device handler registered
[3568101.584600] hp_sw: device handler registered
[3568101.584604] emc: device handler registered
[3568101.584607] alua: device handler registered
[3568101.584635] libphy: Fixed MDIO Bus: probed
[3568101.584688] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[3568101.584695] ehci-pci: EHCI PCI platform driver
[3568101.584707] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[3568101.584710] ohci-pci: OHCI PCI platform driver
[3568101.584720] uhci_hcd: USB Universal Host Controller Interface driver
[3568101.584767] usbcore: registered new interface driver usbserial
[3568101.584774] usbcore: registered new interface driver usbserial_generic
[3568101.584782] usbserial: USB Serial support registered for generic
[3568101.584795] i8042: PNP: No PS/2 controller found. Probing ports directly.
[3568102.613816] i8042: No controller found
[3568102.613953] mousedev: PS/2 mouse device common for all mice
[3568102.674341] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[3568102.674439] rtc_cmos: probe of rtc_cmos failed with error -38
[3568102.674487] hidraw: raw HID events driver (C) Jiri Kosina
[3568102.674598] usbcore: registered new interface driver usbhid
[3568102.674602] usbhid: USB HID core driver
[3568102.674637] drop_monitor: Initializing network drop monitor service
[3568102.674730] TCP: cubic registered
[3568102.674736] Initializing XFRM netlink socket
[3568102.674861] NET: Registered protocol family 10
[3568102.675086] NET: Registered protocol family 17
[3568102.675303] Loading compiled-in X.509 certificates
[3568102.675339] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[3568102.675370] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[3568102.675898] Loaded X.509 cert 'CentOS Linux kernel signing key: 51f4683f502ac48a18cc459fa0796a580712887d'
[3568102.675942] registered taskstats version 1
[3568102.676502] Key type trusted registered
[3568102.676994] Key type encrypted registered
[3568102.677389] IMA: No TPM chip found, activating TPM-bypass!
[3568102.677422] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[3568102.677426] xenbus_probe_frontend: Device with no driver: device/vbd/51728
[3568102.677430] xenbus_probe_frontend: Device with no driver: device/vbd/51744
[3568102.677433] xenbus_probe_frontend: Device with no driver: device/vif/0
[3568102.677522] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[3568102.677643] md: Waiting for all devices to be available before autodetect
[3568102.677650] md: If you don't use raid, use raid=noautodetect
[3568102.677825] md: Autodetecting RAID arrays.
[3568102.677833] md: Scanned 0 and added 0 devices.
[3568102.677837] md: autorun ...
[3568102.677840] md: ... autorun DONE.
[3568102.677875] List of all partitions:
[3568102.677880] No filesystem could mount root, tried: 
[3568102.677887] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[3568102.677895] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.6.3.el7.x86_64 #1
[3568102.677901]  ffffffff817e0028 00000000a9ca04fe ffff88007bc01d60 ffffffff815e20bb
[3568102.677910]  ffff88007bc01de0 ffffffff815db579 ffffffff00000010 ffff88007bc01df0
[3568102.677920]  ffff88007bc01d90 00000000a9ca04fe 00000000a9ca04fe ffff88007bc01e00
[3568102.677928] Call Trace:
[3568102.677940]  [<ffffffff815e20bb>] dump_stack+0x19/0x1b
[3568102.677948]  [<ffffffff815db579>] panic+0xd8/0x1e7
[3568102.677957]  [<ffffffff81a0955d>] mount_block_root+0x2a1/0x2b0
[3568102.677965]  [<ffffffff81a095bf>] mount_root+0x53/0x56
[3568102.677971]  [<ffffffff81a096fe>] prepare_namespace+0x13c/0x174
[3568102.677978]  [<ffffffff81a091cb>] kernel_init_freeable+0x203/0x22a
[3568102.677984]  [<ffffffff81a0892b>] ? do_early_param+0x88/0x88
[3568102.677993]  [<ffffffff815c3960>] ? rest_init+0x80/0x80
[3568102.678000]  [<ffffffff815c396e>] kernel_init+0xe/0x180
[3568102.678008]  [<ffffffff815f26ec>] ret_from_fork+0x7c/0xb0
[3568102.678015]  [<ffffffff815c3960>] ? rest_init+0x80/0x80

So it begins to boot but cannot find the root parition. What strikes me as odd is that no filesystems could be found. I would normally think this is a problem with the encrypted partitions, but I can mount them under Finnix. Perhaps that's still the issue but I don't see it.

And here are the guide I have been using as a reference:
http://spin.atomicobject.com/2013/03/18/linux-encryption-cloud-luks-linode/
https://www.linode.com/docs/tools-reference/custom-kernels-distros/run-a-distributionsupplied-kernel-with-pvgrub/#centos-6-and-newer
https://www.linode.com/docs/migrate-to-linode/disk-images/migrating-a-server-to-your-linode

So if anyone can see an obvious mistake I made, I would appreciate the pointer.

The immediate problems I saw:

1) The grub config filename needs to be /boot/grub/menu.lst (as in list, not first)
2) The groot and kopt lines in your grub config will not work, which is likely why it's failing to boot. Try this instead:

timeout 5
title CentOS (3.10.0-123.6.3.el7.x86_64)
root (hd0)
kernel /boot/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=/dev/mapper/crypt-xvdc cryptdevice=/dev/xvdc:crypt-xvdc console=hvc0 ro
initrd /boot/initrd-plymouth.img

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

Bah. Old man eyes strike again!



Thank you very much. I got further. Now I see the normal grub menu and the system tries to boot, but I think I ended up at the same spot. The problem seems to be here:

My guess would be that your initrd doesn't have the necessary tools to deal with LUKS volumes. You may want to try using initramfs-3.10.0-123.6.3.el7.x86_64.img instead of initrd-plymouth.img to see if that helps.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

dwfreed, thanks to your help, I have made significant progress.

Since this is CentOS, I regenerated the initrd using dracu after chrooting in the Finnix environment. That allowed xvdc to be decrypted and mounted as the fs root. I'm not quite out of the woods yet, though. It seems to be having a hard time with the swap partition. It stops here for awhile before timing out:

(1 of 2) A start job is running for dev-mapper-crypt\x2dxvdb.device
(2 of 2) A start job is running for dev-disk-by\x2du...002ca63B.device

I tried re-encrypting and re-formatting swap using the same method as I used for xvda, but it did not change that error.

The system then proceeds to boot and presents me with a logon screen (yay!), but when I attempt to logon as root at the console, I get this:

Last login: Tue Sep  9 22:52:16 on hvc0
 -- root: no shell: Permission denied

/root exists, as does /bin/bash and the entries in /etc/passwd look correct. If I had to guess, I'd say this wasn't really a permissions issue but maybe something with a console/tty setting; however, considering I can connect to the console to see the boot process, perhaps that's not the case.

Update: An selinux relabel seems to have fixed the root logon problem, so the only remaining boot problem is with the attempt to decrypt and load xvdb!

Aaaand, stick a fork in me 'cause I'm done! Turns out I had the UUIDs for xvdb and xvdc transposed in /etc/crypttab. Now it boots fast and I can log in just fine. Thanks again for your help!