Why is he a bad neighbor if he uses 400% of CPU on single node? Linode is not overselling, so the alloted and guaranteed CPU time is yours to use. Or am I missing something obvious here?

Also, DB on dedicated? Did you get one in the same datacenter with Linodes?

Linode does not guarantee 400% CPU, only a fair share of the host's resources. The fair share depends on what everyone else is doing, because the host probably has only 8-16 CPUs. Call it overselling if you want, but if you constantly use 400% CPU, they'll open a ticket and gently ask you to fix your server.


That would be worth looking into.


This shouldn't be a problem unless OP is getting tons of comments and trackbacks. Most blogs are read-heavy, so the amount of processing power devoted to comment filtering may be negligible.

The DB is running on another dedicated machine, same data center via a local IP address.

Dave

I think Azathoth misunderstood that. You're using a Linode that is dedicated to the database, not a "dedicated server" with another company, right?

Correct, a Linode running a dedicated mySQL server over a local ip address in the same data center

Yes, Azathoth aparently misunderstood, because it should have been "dedicated node" not "dedicated machine", and I jumped at it because I'm looking to have a hybrid setup just like that: app on virtualized nodes, and dedicated (servers) for the DB cluster.

As for CPU.... I was under the impression that with Xen you can limit how much CPU single node will consume so it's not like OpenVZ or similar where one hog can bog down everyone else on the host.

So if we can't use more than alloted, and it is alloted fairly (of X nodes on the host, each can use 1/X of the CPU time), where's the problem? Or am I under wrong impression re: Xen or Linode's Xen setup for nodes?

The issue is that nodes *can* burst higher than their minimum allotted fair share, all the way up to 400%, and users usually expect to be able to do so. It's rude to take advantage of that excessively and force your neighbors into getting less than 400%. It's about sharing and the tragedy of the commons. (Or at least that's what your hapless neighbors will think.)

Edit: Copy editing.

_________________
Matt Nordhoff (aka Peng on IRC)

Please bear with me, I'm trying to understand this 100%. So let's say there is single node on the host and it starts crunching numbers, ends up using 100% of computational power of the host (minus host's "administrative" overhead). Another node comes in doing the same thing, but each can now use 50% of the host's computational power. Consequently, X nodes can each use 1/X of the said power.

Now, each node has access to 4 cores, resulting with 400% max on the graphs. It is my understanding that if you had 4 processes crunching numbers and the guest/host kernel spreads them each to its own core, the graphs will show 400%, but your node in total is actually using only 1/X of the host's CPU time*, where X is the number of nodes on the host doing the same thing.

*) CPU time is the keyword here. You can use over 9000 cores if available, but on each you're assigned 1/X of the CPU time in the situation where X nodes are requesting max CPU.

So the graphs may show 400% because the guest kernel scheduler sees 4 processes active every time it looks, each on its own core, but is oblivious to the actual time slots assigned by the host kernel scheduler.

Am I being wrong?

The intricacies of Xen CPU scheduling are not something I know, but, uh, I'll pretend I do and keep talking anyway!

My understanding is that when a guest thinks it's using 400%, that means it's really using 400% of real CPUs.

I note that guests are aware of steal time, which is when the guest asks for CPU time but the host says no and schedules some other guest. (Look for an 'st' column in your favorite tool like vmstat.) I don't know how that fits into the guest's scheduling estimation and reporting, though.

I also note that the Linode manager graphs aren't generated by the guest, so they have access to what really happens, not whatever is going on in the mind of the guest's kernel.

Incidentally, Linode is believed to have more than 400% available on the hosts*, so a single node can't actually suck everything up on its own.

* At least I hope they're not paying for dual-capable CPUs and then putting only one in each host... (Or lying about what CPUs they use.)

Edit: Clarification, copy editing.

_________________
Matt Nordhoff (aka Peng on IRC)

True, but that still doesn't mean the 400% shown in graphs is not the sum of only available CPU time.

I seriously doubt that 400% on the graph means 400% of the underlaying hardware. Because then, on 8 core hosts (and I think I read somewhere Linode uses 8 cores), 10 nodes per host, each shouldn't be using more than 80% (800% being total, 1/x = 800 / 10 = 80%), or 40% with 20 nodes per host.

It would be great of someone from Linode could clarify this.

Like I said, Linode doesn't limit you to your fair share. If there isn't contention, you can burst all the way up to 400%. Edit: The fair share is a worst-case scenario.

_________________
Matt Nordhoff (aka Peng on IRC)

In which case, back to my original question. If fair sharing is enforced, how's anyone a bad neighbor? If you're using all of the CPU, and I don't need it, why should I complain. If I start needing it, I can rely on fair sharing to get what "belongs" to me.

If 20 of us content for the CPU on the same host (say we all run SETI@Home), it will be shared equally which is what we paid for (and everything above 1/X is "free" bonus). Unless Linode is overselling*, and we know they're not.

*) which would be, in this case, that the number of nodes per host is so great that individual nodes grind to a halt or have visibly poor performance, which I often saw with OpenVZ hosts, never here on Linode in 2 years that I'm here.

I think it's a question of equality vs. expectations. Yes, if multiple guests are all going full bore with CPU, they'll all at least get an equal share of the available cores that they share[*]). So in that sense, the CPU is being shared equally.

However, in practice, especially at the lower Linode sizes, there are lots of guests on a host, and it's pretty rare for them all to use maximum CPU simultaneously for long periods of time, and expectations for performance end up being established based on what actually happens. This is definitely different than a guarantee (e.g., on a Linode 512 you're only really guaranteed like 20% of a single core, assuming 20 guests per 4 cores[*]), but part of what makes Linode work so well is that you virtually always have larger burst levels available just due to the statistics of sharing.

The concept of a "bad neighbor" is soft, so I don't think it can be precisely defined. But I do think that there are expectations that most Linodes can burst quite high CPU-wise, so if a small number of guests on a host are burning all available CPU (even if shared equally), the experience will suffer in general. Taken to the extreme, if everyone tended to use 100% of the CPU they could get (even if shared) I suspect Linodes would be a whole lot less attractive.

In other words, over the long haul we actually all benefit by no one guest (or small number of guests) trying to take as much as possible, even if fairly distributed. Except of course for someone who would otherwise really use the maximum CPU for a long period of time in which case that runs slower over the long term by not taking as much. But that's where the concept of good or bad neighbor can come in. The up thread reference to tragedy of the commons is reasonable I think, if imperfect since CPU isn't a resource to deplete except over short time periods.

The nice part of all this is that it usually "just works" - e.g., do what you need and it all sort of evens out. But when you start talking about stuff like SETI (which will take whatever CPU it can get for absolutely as long as possible) that begins to break the statistical sharing that, in reality, we all benefit from.

-- David

[*] I believe that on a typical 8-core host, the available guests are all set to a maximum burst of 4 cores, so it's possible if two guests are actually on disjoint subsets of 4 cores they wouldn't interfere at all.

I think you're absolutely right. There have been many blog posts comparing the performance of a Linode with similarly priced Amazon EC2 instances and whatnot. Most of these benchmarks are CPU-bound. Linode always wins by a significant margin. Why? It's not like Linode can magically squeeze twice the performance out of the same hardware. The easiest answer is that Linode allows you to burst all the way up to 400%, unlike many other "cloud computing" services.

If one really wants a "non-oversold" service, one should try Amazon EC2. I don't think anyone will like their CPU performance, especially after having one's expectations spoiled by Linode. Overselling gets a bad name only because stupid hosts do it wrong. Some hosts overreact to this, and try to do away with overselling altogether. But both of these are extremes. Overselling is OK if the company is competent enough to manage it properly, and Linode has obviously mastered the art of CPU allocation over the last 8+ years. This gives us the best of both worlds. Everyone can use as much CPU as they need, as long as most people are courteous.


I would not want to share a server with 40 other people who all hammer the CPU, even though I'd still get my fair share. Fairness alone can't get us the fantastic performance that we've come to expect from a Linode. It would be so much better for everyone if the host's load was low enough that the enforcement mechanism didn't even need to kick in. Just because the police exists to enforce law and order doesn't mean that I wouldn't rather live in a neighborhood where the police is seldom needed in the first place.

But we're going way off topic here . . . . .

To put it simply:

Bad neighbour in the sense that if everyone were like that, all the Linodes on the machine would get a fixed percentage of CPU time instead of being able to burst up to all available processing when needed, like to handle a surge in traffic for example.

I think Linode should group all the bad neighbours together on a host (still keeping the node/server ratio of course) and let them see what equally sharing the CPU is like.

_________________
If all else fails, reboot...
PHP Tutorials and MySQL Tutorials