Linode Forum

[caker]

Linode Backup Service (beta 2.0)

We've been working on and testing our newly revamped backup service, and are looking for additional beta testers to help shake out any remaining issues. There is no cost involved (it's free!) and Backup Service Beta is available in all facilities. If you'd like to participate in the backup beta, please open a support ticket.

If you're participating in the beta and discover an issue, please open a new forum thread in the Backup Beta category.

Service Overview
The Linode Backup Service is designed to be an easy to use, simple and reliable on-site backup solution for your Linode. It performs backups without causing any interruption of your running system, and is seamlessly integrated into the Linode Manager. There are four backup slots: Three of the slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. The fourth backup slot is a user-initiated snapshot and remains in place until another user-initiated snapshot is taken.

You can configure the time upon which the automatic backups are initiated from a list of 2 hour windows -- you'll want to perform any database dumps before this window. You can also configure which day of the week to consider for the weeklies.

You can restore a backup to any of the Linodes attached to your account, even if it does not have backups enabled. Currently only a full restore is possible. Cross-datacenter restores will be supported in the future.

Features and Limitations
The backup system must be able to mount your disk images on the host. If you've used fdisk on your images to create partitions, or created encrypted volumes, or LVM, or done anything other than use our deployment or disk image creation tools, we won't be able to back up the data. The backup system operates on files, not at the block level.

A failed backup will never rotate out a good one. If a backup fails on the day of a weekly backup, the next oldest backup will be used for that weekly slot.

Files that have been modified, but are the same length and without any metadata changes (like mtime) will not be considered "changed" during a subsequent incremental backup. Currently, only ext2/3 volumes can be backed up (this limitation may be removed in an upcoming release). ACLs and extended attributes are NOT tracked.

Sounds good - what do I need to do?
Simply request that we enable backups for your Linode by opening a support ticket. Once enabled, a new Backup subtab will appear under your Linode's dashboard.

-Chris

[jcw5002]

Awesome! If we were in the original backup beta program, will be automatically be enrolled in this one? Checking my dashboard... it would appear to be the case, but I wanted to verify.

Thanks!!

[caker]

jcw5002 wrote: Awesome! If we were in the original backup beta program, will be automatically be enrolled in this one? Checking my dashboard... it would appear to be the case, but I wanted to verify. Yes...

[wjstone]

are there any known issues that we should be aware of?

[caker]

wjstone wrote: are there any known issues that we should be aware of?
None that I didn't outline above.

-Chris

[jstn]

If there's room in the cool club, can I get in? :roll:

[caker]

jstn wrote: If there's room in the cool club, can I get in? :roll:
There's a LOT of capacity with our current deployments. Just submit the ticket above and you'll be set!

-Chris

[Guspaz]

I set it up on two of my linodes (both 540s). Here are the times:

Disk utilization:
Linode 1: 1.7GB
Linode 2: 13GB

Initial snapshot
Linode 1: 4m18s
Linode 2: 1h49m36s

First automatic snapshot (6h-8h)
Linode 1: 1m11s
Linode 2: 10m54s

I'm not sure why it's so slow on linode 2; sure, it has a lot more data than linode 1, but 11 minutes is slower than my own rsync-based backups over the internet (which cover the vast majority of the filesystem), and the initial snapshot is 20x slower than Linode 1 for only 8x the data.

Is this normal? Higher load on linode 2's host? I don't think it's necessarily a problem, but that initial snapshot very nearly went over the 2 hour window mark had it been an automatic.

[caker]

Guspaz, it's because it's doing far more awesome than you can possibly realize. Nothing to be concerned about.

-Chris

[db3l]

caker wrote: Guspaz, it's because it's doing far more awesome than you can possibly realize. Nothing to be concerned about.

-Chris
What sort of host impact can be expected by ramping up the backup beta again (or later in production)? One of my Linodes gets fairly serious I/O wait times (it itself does very little I/O) generally in the early AM hours ET. I manually execute some regular tasks in that time block each day so experience the differences. Strangely it's been way better recently until the last few nights.

Definitely not scientific but the performance degradation seemed inversely correlated to the interruption in the backup beta, and since I suspect many folks would prefer off hours backups (it's a Newark Linode) it's probably not unreasonable to think that some peer Linodes might be in the beta and getting backed up in that window. Of course, it could just be another Linode who started nightly processing, or something end-of-month related, etc... But it does make me wonder about the additional I/O the backup process must by necessity add to the host system.

Given that disk I/O is a constrained resource, what sort of additional I/O overhead from the backups running are anticipated on a given host? I'm assuming snapshots/backups are spread across time to avoid spikes, but is there a worst case analysis if all Linodes on a host are participating or if a large number choose the same time window?

-- David

Edit: Forgot to mention that my Linode itself is not in the backup program. I realize that running on top of a snapshot during a Linode's own backup might have different costs - I'm more interested in the potential backup impact on other Linodes on the same host.

[caker]

db3l, that is something that's very difficult to answer - it depends on many factors. But in the vast majority of cases it will have little to no noticeable impact.

We schedule backups on hosts to prefer only one is running at any given time, to avoid backup-storms as best as possible. During a backup an LVM snapshot is created, and during this time writes from within the Linode that's being backed up do get amplified into a few io ops, but whether this has any noticeable effect on the host overall depends on the activity of those nodes at the time, and what type of IO they're doing. Backup IO is no different than Linode IO, and our same fair-queuing and io-priority facilities apply.

Keep in mind that after the initial backup, subsequent backups are light weight - a simple filesystem scan and then reads of any files that have changed, and then a scrubbing of the cow file that the snapshot has generated (which itself is throttled against io load on the host).

If I found the correct account, one of your Linodes is indeed participating in the backup beta, however the window is set to 8:00 AM eastern. No users on that host have a backup window in the early AM.

-Chris

[db3l]

caker wrote: We schedule backups on hosts to prefer only one is running at any given time, to avoid backup-storms as best as possible. During a backup an LVM snapshot is created, and during this time writes from within the Linode that's being backed up do get amplified into a few io ops, but whether this has any noticeable effect on the host overall depends on the activity of those nodes at the time, and what type of IO they're doing. Backup IO is no different than Linode IO, and our same fair-queuing and io-priority facilities apply.
Knowing that you try to prefer one at a time is very helpful to know, thanks. So it's additional I/O that's not being generated by peer Linodes themselves, but does sound like it shouldn't cause any real impact if it's playing by the same rules as the Linodes themselves for I/O.

Quote: If I found the correct account, one of your Linodes is indeed participating in the backup beta, however the window is set to 8:00 AM eastern. No users on that host have a backup window in the early AM.
Oops, I should probably have mentioned - the Linode in question is on a business account since it uses separate billing, while I use a personal account (which yes, has a Linode on the beta) for forums.

The questionable Linode is on host newark174 and my morning processing is typically in the midnight-2am ET range. Tuesday morning (~1am) I got average I/O waits in the 30-50% range with peaks at 100% for a bit, so a database transfer that typically takes a few seconds probably took almost a minute (small absolute, huge relative change). I'd normally just put it up to host contention but couldn't help noticing the timing with the backup beta, given how good the I/O performance had been recently. Maybe others on the host are just getting their first full backups or something, but given your description, it sounds more likely that it's just coincidence. If it continues I'll worry about digging further, but that's definitely off topic for the backup beta.

Thanks.

-- David

[obs]

Here's my $0.02.

My 360 node took 6 mins 51 seconds to do the initial one and 1 minute 40 seconds to do a snapshot, so all good :)

One question though, the snapshots are the full backups or incremental? Looking at the time I'd say incremental, which raises the question what happens when the auto ones become newer than the snapshot?

[caker]

obs wrote: One question though, the snapshots are the full backups or incremental? Looking at the time I'd say incremental, which raises the question what happens when the auto ones become newer than the snapshot?
Don't think about it like that. We do all sorts of magic that handles rotating out unreferenced files, etc. A snapshot is a full snapshot. The autos rotate out.

-Chris

[obs]

Nice :) I like magic.

I'll probably try creating a new node sometime soon and restoring to it to see how it goes.

Out of curiosity how long do you plan on testing for? (Assuming nothing screws up).