Hours Delay "Linode Events Notification"
Steps to reproduce:
1. If not already done, enable “Events Email Notification” at
2. Reboot any Linode server
3. Wait 15 minutes. The Linode’s automated email notification is not receive from. This is the challenge. We first notice this challenge four weeks ago. This challenge can be reproduce with all our Linodes.
4. The expected result is that it’s receive between a few seconds and 5 minutes. This was working for the last few years.
Two concerns with this challenge:
• Security risk. Because in the unlikely event that a linode is compromised, hours is a very big window for unauthorized actions and abuses. Then for us to take appropriate actions.
• False alarms and waste budget. Because the whole point of receiving automated event alerts is to be alerted in a timely fashion. When we receive an email hours letter sometime trigger false alarms because we did not rebooted the linode. We did hours ago but not now not 5 minutes ago. So at first is feels like an unauthorized event. But after using budget investigating, it turn out to be a false alarm from Linode.
Below is a copy of the last challenge with the Linode’s email notification we’re referring to. We did a manual reboot on January 28, 2018 around 2am PT. Then we received this email notification January 28 at 12:15pm PT. Roughly 6 hours delay.
> Linode Events Notification -
To:
Hello ! The following activity has recently occurred:
- ( ) Lassie initiated boot: Profile - Completed Sun, 28 Jan 2018 10:04:58 GMT
* * You can change your notification settings via https://manager.linode.com/profile .
Related documentation at
What we tried that did not work:
• Whitelisted domains:
linode.com
mail[0-9].linode.com
www[0-9].linode.com
• Wait for a few email notifications. In case it’s a temporary workload challenge with Linode mail servers.
• Review the configuration of our email servers. It seems appropriate. Including whitelisted linode email addresses. We do frequently receive automated email notifications from other suppliers. No delay with those.
• Whitelisted this domain webserver1.linode.com (webserver[0-9].linode.com)
Linode's reply "This is something that we are aware of and are working toward a solution presently. Just a little insight into the unevenness; sometimes we have to facilitate the mailing manually, which obviously does not happen with the same precision as if the Linodes were facilitating the sending."
10 Replies
For example, a reboot was initiated at 8:22 in the morning and the notification email came at 18:30.
My own status notifications via icinga2 come instantly.
I would be happy to contribute testing if needed
You should check if your mailserver's queue hasn't grown to silly sizes.
Thanks!
Terry
jackley
Linode Staff
@drzow:
Sorry to bump a month-old thread, but I was doing some work on my Linode today and saw the same thing. At first I thought it might be that the mail wasn't going out until local time (CDT, GMT-7) matched the GMT time the event happened at, but the times do not line up quite that perfectly. In any case, it's taking in the neighborhood of 7-8 hours for the notifications to come through. Wondering if I should open a support ticket, but since this was already being discussed (albeit last month), thought I'd start here.
Thanks!
Terry
Hey Terry -
Thanks for reporting this, and no worries on bumping an old thread. It's still an issue, and we're still working on a permanent fix. Until that's in place, job event notifications are going to be a bit inconsistent.
- Jim