Hello and thanks for your wonderful service.

I wonder if you could provide some advice on how I can download and deliver a cloned linode disk image to a client who doesn't have linode yet. I would recommend they get linode but cannot force them to do so, and certainly not immediately. I cannot keep the image on my account for them forever either, so I need to save it offline.

Background: I purchased another linode to build some proof of concept projects for a client that include a configured LAMP stack and other packages. To make it easy for them, I'd like to deliver a bootable disk image in addition to a tarball with installation instructions. I'd like to explore something like docker in the future but that is too advanced for them for now.

- How can I download an image I create in the linode dashboard?
- How to restore it if the customer purchases a linode account?
- Can it be installed on the customer's server that is not on linode or running xen?
- The linode image is 11GB but df says only 3.8GB are used. Is there a way to shrink an image in the dashboard without losing data? (This would be very useful to know in general.)
- Is the procedure something like shut down linode, clone image, reboot, and then mount and somehow compress the image?
- How to download it. I might be able to stick a copy on a server on another continent, but realistically it has to be delivered on a USB stick, so I need to download it to my laptop which will take days at wireless speeds and probably will get halted many times before it finishes. Do I need a block-level download program?
This page (Copying a Disk Image Over SSH) doesn't seem realistic. https://library.linode.com/migration/ssh-copy
This page mentions things like partimage and ddrescue... not sure if any of these tools can copy over lossy connections. http://www.cyberciti.biz/open-source/linux-based-disk-cloning-imaging-software/

I've already presented to the customer that linode is great. Help me figure out the process that would help get them onboard. It is the kind of company that if they start to use linode they probably would start using it a lot.

Thank you!

Not sure why you would do proof of concept on a VPS (instead of a local Virtual Box, ESXi, Parallels, etc)....

In any case, assuming you're not a charity, you charge your client for storage, they can either pay you to maintain their dev work on YOUR VPS or they can setup their own VPS and you'll (for a small fee) migrate the dev work to their system.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

Um, I'm just asking for technical answer please? I am familiar with the theories you espouse.

However it's not about the money. This was the best solution at the time considering client resources and bureaucratic issues. It would now be desirable from a contractual standpoint to physically deliver a USB stick or DVD-ROM, especially if I wanted to provide a full deployable disk image. Setting up their own VPS is both logical and inexpensive, but may be disjoint. FWIW in Japan, delivery of physical media is always desirable, hence my question.

The way mentioned via the Linode Library is the way to do it. In the Linode Manager, you can edit your disk image, change it to like 4000MB and then download it via SSH/dd.

I suppose another way if you don't like to copy via SSH is create another disk image that is about 4GB, boot in rescue mode, create the dd and save it to the secondary disk image. Then boot back up in normal mode, mount the other filesystem that has your dd and then you can do with it what you want (transfer via ftp, or put it on a webserver.. whatever..). Just make sure you properly encrypt/protect the disk image if you prefer a way like this.

EDIT: If your connection is slow/unreliable, you can also use tar to compress it and split it up into multiple parts.

You mention the library article about transferring an image over ssh is not realistic, but you're asking how to transfer 'x' bits of disk image data so there's only so many ways to do it. The general notion of treating the disk image as a raw file is the common theme. This will transfer the full image size as opaque data. (I would suggest adding -C to the ssh command in the library article to try to get at least a little benefit from compressing the transfer)

Another option would be to do a filesystem level transfer while the Linode is running, or off of a mounted copy of a cloned image (so it's not in active use). That's not going to be a block level backup, but depending on what you've installed and what it takes to run on another system, may work as well and require less data to be transferred. Since it will essentially be a one way transfer, I'd suggest rsync.


Unlikely. It's not just a Xen issue but on Linode the kernel is supplied external to any images, so you probably don't have the necessary information to actually boot. There's also no partition information, although restoring the image locally to an existing partition rather than a disk can handle that.


You can resize disk images from the Linode Manager, which providing you've used the default ext3 filesystem will include resizing the underlying filesystem. Resizing does have to be done while the Linode is powered off. And yes, resizing down would minimize the data transferred when moving raw images.


You could do that too - you'd have to attach the cloned disk to your Linode before rebooting, and then treat the new disk device however you like. That's similar to what booting into the Finnix rescue mode is doing, just using your own Linode as the host environment. My guess is doing that simply to compress the disk image would be of modest benefit over just including compression in whatever transfer method you use.


Can't help much with transfer rates - moving the data is, well, moving the data and will need whatever time it needs based on your connection. I'd certainly prefer a wired connection for the first transfer if there was such an option. If you think you'll be interrupted a lot, I think Finnix also has rsync (or you could use the clone disk approach and ensure you have rsync on your Linode) and use that instead of a straight ssh. Those other tools, I believe, are intended to work with raw local devices, so probably aren't what you want.


Honestly, if I were you I'd consider starting afresh locally with something like VirtualBox. Install the same distribution you used on Linode to get to a starting point, and then just replicate the packages you installed on your Linode. That can all be done locally, and thus very efficiently.

Then, just transfer (again, I'd prefer rsync, but pretty much any file transfer tool works) the application-specific configuration and files you have from your Linode to your new local virtual machine.

At that point you'll have a local virtual machine with your demo system that can easily be packaged up (export the appliance through virtualbox or just provide the config/disk image) and that you'll know the customer can run most anywhere with their own installation of VirtualBox.

It's going to be more steps, but may in fact be less time consuming since you won't have to be fighting to get a large disk image transferred remotely and then turned into something that can be used on a local PC by your customer.

-- David

Thank you very much for very useful replies, problem solved.

To add a clarification, by "not realistic" I only meant that the connection would break several times before completion. Now I understand rsync can probably handle that.

Also I should note that what we call "proof of concept" is a net-accessible demo outside the firewall, not something for a local PC. Sorry to be unclear.

Based on all of the valuable information I received in this thread I think the solution for me now is:
- Delivery shall just be a tarball with installation instructions.
- Make a reduced size disk image and hang onto it in linode for a while (I learned you can't destroy data by accident with reduce size)
- Download image for peace of mind using rsync
- Recommend linode, and offer to provision it with the image from my linode.
- Meanwhile, I will consider using virtualbox -- locally, since it might speed up development somewhat. As for deploying virtualboxen the client might not like to deploy them on customer facing production servers, but I had already proposed training client on virtualbox or docker and will continue to follow that up and see if we can use it in the future. Could be a nice way to deliver a PoC environment to devs to try out locally in the absence of dedicated infrastructure.

Thanks again for your help!

Best regards,

Matt

You can download VirtualBox, which is a good virtualization program. An important note about setting up the virtual hard drive file: the size you set for the dynamically allocated hard drive is the maximum size, not the starting size. You should also definitely use that option, otherwise you'll take up too much space on your actual hard drive, especially if you create VMs to use as templates to copy. It will be more secure to do networking on a VM then on a public server.