Both of your accounts are on the same host server -- I've seen this on other hosts which were new and people were busy signing up and deploying an install into their accounts.

Deploying a new filesystem causes the host to perform a large copy, an fsck, and a resize -- all at a much lower priority than the Linodes run.

There are still some I/O issues in the 2.4 kernel that there have been many LKML (Linux Kernel Mailing List) posts regarding. Issues that affect large I/O queues starving smaller reads. Which would cause a block. Linux 2.4.22-pre* already includes some fixes, so once that becomes more stable I'll look into upgrading.

Hopefully once people get settled in, you won't be seeing this issue. I'd like to prove that theory, so next time you get a pause, note the time (and time zone) and let me know when it happened. I'll look in the logs to determine if a deployment was occuring at that time.

If we do continue to see it, then I'll migrate the deployment over to a dedicated machine to off-load that from the hosts.

Thanks for keeping me informed!
-Chris

Just out of intrest how would moving the deployment to a dedicated machine help.

You would still have to do the copy, resize and fsck on the host machine, so it would still have the same load.

Adam

The only thing that would need to be done on the host is trasnfer a file over the network (or a private network, but that's a topic for another discussion). Also, this would benefit I/O load conditions since the network would be slower than a full-throttle copy...

-Chris

I keep forgetting that they are not actual partitions but only files.

So I can see how doing things on a different machine would help, but the transfer of a gig would be a bit big. But if it was a throttled transfer or over a different network then it would give some sort of benifit.

The use of an internal 'admin' network would be a good idea, as it would allow for the images to be transfered, but also allow for internal mirrors, which you said you may look into would not hit the performance of the external network.

Adam

I've seen about 11MB/sec transferring using scp over the 100base-T network we've got. Disk benchmarks on the host see somewhere between 30-50MB/sec, so yes, deploying over the network would take about 3x as long (say, 8 minutes per gig?)...

My dream is to have an internal Gigabit network with shared storage devices. That way, I could dynamically distribute each Linode when it boots onto a different host server (in the same plan group) for total redundancy and load balancing.

That's Linode 2.0 in my opinion, as it requires a significant investment in gear -- but that's the holy grail for me

-Chris

That would be good and would mean that the newly booted linode would go on to a server which had the least load.

The only problem is how do you tell when to remove a linode from a machine.

It does seem un-nessecary though, I was imagine that most linodes are running 24/7 in less they where taken down for some reason.

So would there be any need to remove a linode and add it to another server. And like you said gig-e technology is very expensive. It would be easier to stick will a seperate 100 meg internal network that would allow for inital images to be transfered and for internal mirrors and other internal traffic.

People will not mind that a new image takes 30 mins or so to be set-up, but waiting that length of time for a server to be started is not what people want.

Adam

In the distributed model, the filesystems are never transferred around.

When you boot your Linode, the clustering software would determine the least-loaded Host to boot your machine off of. Your Linode would transparently access it's filesystem over the private GigE network.

The host wouldn't have any disk responsibilities at that point, so you're not waiting 20 minutes for your images to be copied.

-Chris

In that case it could work a but I still think that there would be problems on a busy network with loads if linodes.

Not taking into account the ability of the hard drives in the storage machines a gige network can theoritically put through 119.2 meg\second

Which is not a lot when you are transfering data for lots of linodes.

It seems that it would cause more problems than it would solve, since you dont oversell your machines and they appear to get sold out quickly would there be any benifits.

Adam

All good things to consider. This is just a dream at this point. I suppose there are pooling methods, large caches, and other methods to improve number of clients. I don't have personal experience with that type of configuration to know the number of clients it could handle. But, in general, I've observed very low aggregrate disk I/O (excluding deployments) so far...

The biggest benefit in the distributed model is 100% redundancy, which would help me sleep better at night.

Hopefully, some of the fixes in 2.4.22 will get rid of the starvation issues, otherwise it'll be time for an off-loaded deployment system.

Thanks,
-Chris

Yeah I can see where you are coming from for that part.

It would mean just loading the linode on another machine, as long as you had the capacity.

But you would have to ensure that you had an n+1 set-up so you always had at least one spare machine that you could move the linodes to, to ensure you had could move all the the linodes on the machine that died.

The other option is to back up the linodes to a remote machine which had an array of large capacity HDs as part of a backup routine and if a machine died, you could just move them required linodes to a standby machine. Which would take time.

So by using a distributed system you could have the linode up and running within minutes of the host machine going offline.

There are many ways to provide the redundency, the problem is having the hardware there to do it.

Adam

http://heroinewarrior.com/firehose.php3

You should look into an app I found called firehose - it basically stripes redundant ethernet connections allowing an internal network to transfer at unbelievable speeds. (it doesnt work with internet yet) Right now it's used by video rendering farms to transfer the video to multiple rendering machines at ultra high speed. I know you are using an offsite host but a dedicated connection and your own boxes may prove feasable at some point in the future. An app like firehose could open new possibilities on the command and control end. "Unlike RAID striping, FIREHOSE striping load balances the network devices so every ounce of bandwidth is utilized. Combine a 400Mbit firewire eth device with a 100Mbit eth device to get 500Mbits of power. Combine 10 100Mbit ethernet ports for a gigabit pipe. The number of devices which can be striped is limited only by imagination and budget."

[url][/url]

_________________
Be warned that typing \fBkillall \fIname\fP may not have the desired effect on non-Linux systems, especially when done by a privileged user.

Transfering data across multiple connections is something that has been around for years.

It looks like all it does it take all the eth connections and bind them to a single IP address, which allows for transfer across all of the connection. But you are still left with having to purchase the additional hardware.

Depenedent upon what type of motherboard the machines have in you could probably have up to a max lets say 5 cards. Which could give you either 5 * 100 or 5 *1000.

So either 500 meg or 5000 meg. The 5 * 100 cards would probably still be cheaper than a single gige network, at the current prices. But still rather slow. Going for mulitple gige network cards would provide a faster set of connections, but you are limited by the speed of the hard drives which are still the slowest part of the machine and since it would be impossible to cache the data to memory due to the size of the files, you would be limited to the max throughput of the machine, which will be anywhere near the capacity of 5 * 1000 meg connections. You start to max out when you try to go for maximum throughput on a single gige connection.

You also have to take into account the extra hardware cost, it is not only additional networks cards but additional, switches, UPS etc which adds to the cost and it starts to not become cost effective.

You start to wonder if it would be easier to mirror the hard drives and move them to a new machine if the machine dies, or swap the drives if one of the drives dies. It depends how far you are from the datacenter.

In the end it all down to what you want over the cost of what it will take to get it working, at a level that you require.

You have to remember that centralized file storage works by moving files to either a cache server if they are used a lot or transfered to the local machine and then echoed back to the central file storage when they are either saved or when the machine is released by the machine that requested it. This helps to cut down network traffic but since the images are been constantly written to, log files, database access etc. They will be constantly written back to the host.

Just a few things to think about.

Adam