Linode Forum

[umdenken]

My site is down as I write this and can't yet connect to my console:

I discovered that my server is down and am connecting from a new location. And now, 5? 10? minutes have gone by and I'm still waiting for the security/confirmation email to arrive in my gmail account.

I've noticed this over the years I've had Linode servers: the email has never instantaneously arrived. Often, minutes go by. But right now, I really need fast access; it's critical. And as a sysadmin, I do know that if it was desired that the email arrive nearly immediately, it could be done.

[ericholtman]

umdenken wrote: And as a sysadmin, I do know that if it was desired that the email arrive nearly immediately, it could be done.
You know wrong.

How is linode supposed to control gmail? (and if you don't think gmail has ever had delivery problems, you are either naive or the luckiest person on planet earth).

[obs]

You can email support@linode.com and they can verify you to get you access.

[anderiv]

Once you receive the message, look at the "Received:" headers. You will be able to tell where the delay occurred. If the delay was within Linode's systems, then take it up with them. If the delay occurred after the mail left their servers, though, there's nothing they can do about it.

[db3l]

This may also be a good opportunity to review plans in place for outage handling. Given the current use of email to enable a new white listed host, there's always a risk of delay. Regardless of where the delay in the white listing email ends up being in this specific case, the nature of email means delays in future cases are also always possible. So taking steps and being prepared with alternate ways to deal with an outage is prudent.

I suppose one drastic solution is to disable the white listing, but just having fallbacks can avoid that need. For one thing, ensuring you have LISH console access via ssh ready to go (which assumes you know the right username and host to use, since you may not be able to refer to the manager remote access page) would be a good idea, as the white listing is just for the Linode Manager web login. So in a scenario such as this, at the least you could get to your Linode's console, check it out, and if necessary initiate a reboot just to get things running again. For myself, for example, I have local aliases for each of my Linodes (like xxx-host for Linode xxx) that map to the appropriate host, and then clauses in my ssh configuration file to automatically default to the right user (linode#####). So I can reach any console on a Linode xxx with just "ssh xxx-host".

You might also think of having at least one white listed host that itself is remotely accessible (could even be another Linode, which can even be advantageous compared to home locations due to the static address). That way, even on the road, in a pinch you can first access the white listed host and then proxy through to Linode (even a simple ssh forwarding would work).

-- David

[waldo]

ericholtman wrote: umdenken wrote: And as a sysadmin, I do know that if it was desired that the email arrive nearly immediately, it could be done.
You know wrong..

You know wrong.

I've had this same issue. It's not a gmail delivery issue, it's a slow cron job or email delivery issue from Linode. I can repeat the issue time and time again. It varies from 2 minutes to 15 minutes in my testing.

I've looked at the email headers. It's on Linode's side. Once the email actually gets sent I get it nearly instantly. Once it's sent...

I've already brought this up in the forums before and I'm not the only one who's seen it either, note db3l's reply in below mine here:
http://forum.linode.com/viewtopic.php?p=42881#42881

So I think the people who have seen it happen instantly have either been lucky with a very small queue or hit it when the cron job fires... I seriously hope something like this isn't being done via cron, but I've seen dumber setups.

I know one of the forums I admin, which is ran by Drupal, fires notification emails via cron, so it's not nearly as instant as the emails fired of by this phpbb based forum. If only the emails fired off by Linode's manager were fired off as quickly as the notification emails on this forum...

[umdenken]

anderiv wrote: Once you receive the message, look at the "Received:" headers. You will be able to tell where the delay occurred.

Not a bad idea. I'll do that.

[umdenken]

waldo wrote: ericholtman wrote: umdenken wrote: And as a sysadmin, I do know that if it was desired that the email arrive nearly immediately, it could be done.
You know wrong..

You know wrong.

I've had this same issue. It's not a gmail delivery issue, it's a slow cron job or email delivery issue from Linode. I can repeat the issue time and time again. It varies from 2 minutes to 15 minutes in my testing.


Exactly.

[ericholtman]

umdenken wrote:
Exactly.

So you still think linode sys admins can fix every single problem between their outgoing server and your incoming one?

Good luck!

[umdenken]

ericholtman wrote: umdenken wrote:
Exactly.

So you still think linode sys admins can fix every single problem between their outgoing server and your incoming one?

Good luck!

Are you doing ok? You seem to be picking a fight, and from your replies it sounds like you don't have the requisite knowledge to contribute productively to the discussion.

[ericholtman]

umdenken wrote:
Are you doing ok? You seem to be picking a fight, and from your replies it sounds like you don't have the requisite knowledge to contribute productively to the discussion.

I'm doing fine.

You're the one who wrote:

Quote: And as a sysadmin, I do know that if it was desired that the email arrive nearly immediately, it could be done.

Now, it is possible or even likely that linode could speed up the sending of the emails? Sure. In fact, in this case it seems to be the case. Which would mean, I was wrong.

It is possible for them to control every machine between them and you? Not hardly. It doesn't matter how much experience you have as a sysadmin. Once the remote machine has accepted your outgoing mail, you don't have control over it any more.

[umdenken]

ericholtman wrote: It is possible for them to control every machine between them and you? Not hardly. It doesn't matter how much experience you have as a sysadmin.

That's the thing, actually. This isn't really the question to ask. On paper, it sounds logical, I realize. But MTA (email server) configuration is it's own surprisingly complex field with its specific set of expertise required.

Getting mail to arrive w/in seconds at another well configured host is almost always a matter of getting a whole lot of seemingly unrelated configurations right: DNS (reverse dns entry, proper A record, SPF...), local mail queue, system hostname, the local mail server itself, the constructed email message headers, DNS lookup on the server, etc.

[ericholtman]

umdenken wrote:
Getting mail to arrive w/in seconds at another well configured host is almost always a matter of getting a whole lot of seemingly unrelated configurations right: DNS (reverse dns entry, proper A record, SPF...), local mail queue, system hostname, the local mail server itself, the constructed email message headers, DNS lookup on the server, etc.

I completely agree. And I've done all those things.

My point was, you don't have control over *any* of that on the receiving end, if gmail is your provider.

[sweh]

Here's the problem with looking at "received" headers.

If the delay shows up between 2 linode servers then you can be sure the problem is at the linode side of things.

If the delay shows up between 2 google servers then...

But what if the delay is between linode and google? Then you don't know who to blame. It could be linode being slow at sending. It could be google being overloaded and slow at receiving. Or they could be grey-listing. Or tar-pitting. Or deferring. Or some other anti-spam technique. You just can't tell from those headers. All you know is where the delay occured, but not why.

[pclissold]

In so far as it is possible, don't depend on the performance of systems outside your control for crisis handling. This is the way to go:

db3l wrote: This may also be a good opportunity to review plans in place for outage handling. Given the current use of email to enable a new white listed host, there's always a risk of delay. Regardless of where the delay in the white listing email ends up being in this specific case, the nature of email means delays in future cases are also always possible. So taking steps and being prepared with alternate ways to deal with an outage is prudent.

I suppose one drastic solution is to disable the white listing, but just having fallbacks can avoid that need. For one thing, ensuring you have LISH console access via ssh ready to go (which assumes you know the right username and host to use, since you may not be able to refer to the manager remote access page) would be a good idea, as the white listing is just for the Linode Manager web login. So in a scenario such as this, at the least you could get to your Linode's console, check it out, and if necessary initiate a reboot just to get things running again. For myself, for example, I have local aliases for each of my Linodes (like xxx-host for Linode xxx) that map to the appropriate host, and then clauses in my ssh configuration file to automatically default to the right user (linode#####). So I can reach any console on a Linode xxx with just "ssh xxx-host".

You might also think of having at least one white listed host that itself is remotely accessible (could even be another Linode, which can even be advantageous compared to home locations due to the static address). That way, even on the road, in a pinch you can first access the white listed host and then proxy through to Linode (even a simple ssh forwarding would work).

-- David