Dallas: sporadic downtime, dropped packets..

I am on Dallas311. I monitor my server with pingdom.

Overall things look good but I get sporadic downtime alerts from pingdom. I scoured my logs and have not found anything amiss. I can not correlate the downtime with my cron jobs, database backups or anything else. There is no unusual activity on the CPU or disk IO at these times.

Pingdom shows traceroute reports for some of the downtime and there seems to be some packet loss. I am not a network expert in any way so I am not sure how to figure out what this is telling me.

I experienced problems on

  • 10/25 early AM

  • 10/27 early AM

  • 10/28 early AM

Here are two traceroute reports from 10/28:

 1  173.248.147.17 (173.248.147.17)  0.569 ms  1.015 ms  1.050 ms
 2  * * *
 3  gige-g2-11.core1.den1.he.net (216.66.79.41)  0.493 ms  0.565 ms  0.593 ms
 4  10gigabitethernet1-4.core1.lax2.he.net (72.52.92.38)  25.466 ms  25.715 ms  25.767 ms
 5  10gigabitethernet7-3.core1.sjc2.he.net (184.105.213.5)  35.956 ms  36.017 ms  36.067 ms
 6  eq-sanjose-10G.theplanet.com (206.223.116.148)  62.920 ms  63.546 ms  62.820 ms
 7  c2.fd.5746.static.theplanet.com (70.87.253.194)  62.876 ms  62.773 ms  62.820 ms
 8  te7-2.dsr02.dllstx3.theplanet.com (70.87.253.26)  63.000 ms te9-2.dsr02.dllstx3.theplanet.com (70.87.253.30)  62.898 ms 22.ff.5746.static.theplanet.com (70.87.255.34)  62.846 ms
 9  46.ff.5746.static.theplanet.com (70.87.255.70)  63.587 ms te1-3.dsr02.dllstx2.theplanet.com (70.87.253.122)  63.113 ms 6e.ff.5746.static.theplanet.com (70.87.255.110)  63.330 ms
10  56.ff.5746.static.theplanet.com (70.87.255.86)  63.110 ms  62.912 ms  63.168 ms
11 mysite.com (___.___.___.___)  64.635 ms  64.918 ms  66.134 ms
1  208.43.68.57-static.reverse.softlayer.com (208.43.68.57)  1.122 ms  1.152 ms  1.200 ms
 2  po101.cer01.sr01.wdc01.networklayer.com (208.43.118.133)  0.388 ms  0.466 ms  0.525 ms
 3  po2.bbr01.eq01.wdc02.networklayer.com (173.192.18.196)  0.866 ms  0.917 ms  0.959 ms
 4  eq-ashburn-10G.theplanet.com (206.223.115.248)  1.000 ms  1.190 ms  1.223 ms
 5  bd.fd.5746.static.theplanet.com (70.87.253.189)  33.305 ms  33.376 ms  33.424 ms
 6  1a.ff.5746.static.theplanet.com (70.87.255.26)  33.470 ms te3-5.dsr01.dllstx3.theplanet.com (70.87.253.86)  33.382 ms  33.422 ms
 7  46.ff.5746.static.theplanet.com (70.87.255.70)  33.760 ms * *
 8  52.ff.5746.static.theplanet.com (70.87.255.82)  33.572 ms  33.582 ms 56.ff.5746.static.theplanet.com (70.87.255.86)  33.290 ms
 9 mysite.com (__.__.__.__)  34.925 ms * *

Questions:

  • Is anybody else on Dallas311 or Dallas.** experiencing something like this at the same times?

  • Does anybody have any idea what could be going on by looking at the traceroute reports?

  • Linode staff: is there a known issue in the Dallas data center? I see it is slated for maintenance this weekend.

Thanks,

Oliver

12 Replies

I'm not on the same host but I do have a dallas box also monitored by pingdom it has 100% uptime for quite some time.

Try raising a support ticket.

I'm at dallas 1xx and no known downtime either:

http://www.pingdom.com/reports/88tkx6pr … me=jubilee">http://www.pingdom.com/reports/88tkx6pru16j/check_overview/?name=jubilee

Best bet is to contact support.

M.

I've definitely seen some early-morning network issues reaching a Dallas node of mine on host dallas91 (from my own monitoring machine in the northeast), including yes, this morning. But for any of them that occurred over enough of a time period for me to dig into it (this morning's wasn't long enough), it's always been within the network, and somewhat removed from the data center, and so far generally being limited to selected paths. Typically testing from another of my Linodes in Newark is fine, as from other test points on the west coast.

So I guess the best I can say is that it sounds like I'm seeing similar behavior to you, at least from one northeast test source, but so far I haven't seen evidence that it's close enough to the data center in terms of network topology to be under Linode's control.

– David

Perhaps this is the reason for the network maintenance downtime scheduled for Dallas on October 31st?

http://status.linode.com/

I opened a ticket and got a prompt reply from Linode.

There have been short-duration DoS attacks against another Linode customer in Dallas. The times match up exactly with the downtime that pingdom reported on my site.

The maintenance on the Dallas center on Oct 31 will upgrade the network and prevent future attacks of that sort.

There you have it.

Thanks for the help.

Oliver

@OLF:

I opened a ticket and got a prompt reply from Linode.

There have been short-duration DoS attacks against another Linode customer in Dallas. The times match up exactly with the downtime that pingdom reported on my site.

The maintenance on the Dallas center on Oct 31 will upgrade the network and prevent future attacks of that sort.

There you have it.

Thanks for the help.

Oliver

That's good to know. Thanks. I'm in Dallas too and didn't notice anything. My network traffic graph shows me pumping out pages all night long.

Just FYI the outages were short. In my case 5-10 minutes. I'm just bringing that up in case anybody is looking at hourly averages of data. The outages wouldn't show in these.

But if you are looking at the 5-minute average 24-hour graphs on your linode control center, then I would think you would see it there. ( I don't because my traffic is sporadic at that time).

Cheers-

O

I've had similar problems with my VPS in Dallas. My website was not under a particularly heavy load, and the sporadic-ness of the outage makes me believe it was a network issue. In my case though, on Wednesday night, it was particularly bad. Over a period of an hour or longer my website would intermittently go out, or take forever to respond. It was a little frustrating, so I'm hopeful that Linode will have a leg up on DDOS'ers with whatever they're doing this weekend.

I'm on dallas300 and just noticed I was down for a few minutes. The linode website went down as well so I'm sure somebody noticed that one.

Yep. I'm having sporadic issues with Dallas right now too. Pingdom keeps sending me e-mails: down… up… down… up…

http://status.linode.com/2010/10/schedu … -2010.html">http://status.linode.com/2010/10/scheduled-network-maintenance-in-dallas-october-24-2010.html

Hopefully that's the end of the problems for Dallas!

I actually had to turn pingdom SMS notifications off this past week because I was going to burn through all my SMS notification credits. :(

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct