Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Board index » Linux Community Forums » General Discussion
  Previous topic | Next topic 
Author Message
Xan
 Post subject: HE network problems?
PostPosted: Wed Jun 28, 2006 3:45 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Okay, I'll be the first... What's the story with Hurricane Electric today? It's been "browning out" for some time now.
Top
   
Reply with quote  
mikegrb
 Post subject:
PostPosted: Wed Jun 28, 2006 3:53 pm 
Offline
Linode Staff
User avatar

Joined: Fri Oct 17, 2003 12:38 am
Posts: 287
Location: Dr Wierd's Lab, South Jersey Shore
You wouldn't happen to be in the North East, would you?

We've had two other reports from people on Bell Sympatico in Ontario. With info provided by one, I was able to determine the problem was a Telia link in Chicago. I've contacted the HE NOC and they confirmed that the Telia link in Equinix Chicago was saturated and they are working to upgrade the link capacity.
Top
   
Reply with quote  
miallen
 Post subject:
PostPosted: Wed Jun 28, 2006 3:54 pm 
Offline
Senior Newbie

Joined: Thu Jan 08, 2004 3:53 pm
Posts: 13
Website: http://www.ioplex.com/~miallen/
Location: New York
My node is totally unresponsive right now. Actually it comes and goes. Is this what you're seeing? How do you know it's HE? What kind of diagnostic are you using?
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 3:54 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
No, I'm in Texas, actually. Here's a traceroute to he.net.

Code:
# traceroute he.net
traceroute to he.net (216.218.186.2), 30 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  0.690 ms  0.681 ms  0.886 ms
 2  10.33.128.1 (10.33.128.1)  7.265 ms  7.511 ms  7.544 ms
 3  gig2-1.austtxk-rtr2.austin.rr.com (66.68.0.53)  8.432 ms  8.755 ms  7.943 ms
 4  srp0-0.austtxrdc-rtr2.austin.rr.com (24.27.12.34)  8.657 ms  7.768 ms  7.789 ms
 5  pos10-3.austtxrdc-rtr4.texas.rr.com (66.68.1.106)  63.898 ms  65.673 ms  65.490 ms
 6  son0-1-3.hstqtxl3-rtr1.texas.rr.com (24.93.37.61)  65.750 ms  64.234 ms  63.387 ms
 7  pop1-hou-P0-2.atdn.net (66.185.133.153)  17.900 ms pop1-hou-P0-1.atdn.net (66.185.133.145)  19.072 ms pop1-hou-P0-0.atdn.net (66.185.133.157)  17.586 ms
 8  bb1-hou-P2-0.atdn.net (66.185.150.148)  64.928 ms  62.528 ms  76.737 ms
 9  bb1-atm-P7-0.atdn.net (66.185.152.184)  164.928 ms  276.444 ms  202.914 ms
10  pop2-atm-P0-0.atdn.net (66.185.147.209)  36.849 ms  37.650 ms  36.511 ms
11  Telia-atm.atdn.net (66.185.138.46)  87.784 ms  86.298 ms  83.688 ms
12  dls-bb1-link.telia.net (213.248.80.146)  104.869 ms  103.197 ms  105.358 ms
13  hurricane-113208-dls-bb1.c.telia.net (213.248.92.22)  106.121 ms  104.466 ms  103.746 ms
14  pos5-0.gsr12012.lax.he.net (66.160.184.5)  433.108 ms  379.999 ms  449.236 ms
15  * * *
16  * * *
17  * *                                                                                                 


Top
   
Reply with quote  
mikegrb
 Post subject:
PostPosted: Wed Jun 28, 2006 3:54 pm 
Offline
Linode Staff
User avatar

Joined: Fri Oct 17, 2003 12:38 am
Posts: 287
Location: Dr Wierd's Lab, South Jersey Shore
Just noticed that you have "Location: Austin" in the forums. Could you provide the output of a traceroute from you to your Linode? Even better, if you have a Linux box local to you, the output of "mtr --report $your_linodes_address". mtr is available in the mtr-tiny package for Debian and Ubuntu users.
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 3:55 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
miallen wrote:
My node is totally unresponsive right now. Actually it comes and goes. Is this what you're seeing? How do you know it's HE? What kind of diagnostic are you using?


That's exactly what I'm seeing, yes; it comes and goes, and when it comes it's really slow (~500ms pings). Tracing that route to he.net makes me believe it's a network congestion issue at the datacenter.
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 3:59 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Code:
$ mtr --report host18.linode.com
HOST: local                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.1                   0.0%    10    0.6   0.8   0.6   1.8   0.4
  2. 10.33.128.1                   0.0%    10    7.5  16.7   7.4  89.5  25.6
  3. gig2-1.austtxk-rtr2.austin.r  0.0%    10    7.9   8.2   7.6   9.9   0.7
  4. srp0-0.austtxrdc-rtr2.austin  0.0%    10    8.2   9.6   8.0  12.7   2.0
  5. pos10-3.austtxrdc-rtr4.texas  0.0%    10   65.1  64.7  63.4  66.4   1.0
  6. son0-0-0.hstqtxl3-rtr1.texas  0.0%    10  105.8  70.9  63.5 105.8  12.9
  7. pop1-hou-P0-1.atdn.net        0.0%    10   16.6  16.9  16.2  18.6   0.7
  8. bb1-hou-P2-0.atdn.net        10.0%    10   64.7  65.7  63.8  72.3   2.6
  9. bb1-atm-P7-0.atdn.net        20.0%    10   84.2  85.4  83.5  87.6   1.4
 10. pop2-atm-P0-3.atdn.net        0.0%    10   37.0  38.7  36.8  44.7   2.8
 11. Telia-atm.atdn.net            0.0%    10   85.0  87.3  83.5 100.0   5.1
 12. dls-bb1-link.telia.net        0.0%    10  103.4 103.8 103.0 105.6   1.0
 13. hurricane-113208-dls-bb1.c.t  0.0%    10  103.7 104.0 102.2 112.0   2.8
 14. pos5-0.gsr12012.lax.he.net   10.0%    10  389.7 393.7 377.5 413.4  12.1
 15. pos3-2.gsr12416.pao.he.net    0.0%    10  130.6 248.6 127.4 344.7  79.7
 16. pos5-0.gsr12416.fmt.he.net   10.0%    10  355.2 250.3 154.4 381.4  87.9
 17. pos8-0.gsr12012.fmt.he.net   10.0%    10  130.0 132.8 129.2 155.0   8.3
 18. host18.fremont.linode.com    10.0%    10  130.2 121.2  78.9 139.2  24.0
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 4:02 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Well now I'm getting consistent sub-50ms pings to my node. Maybe they've fixed it.

But don't let them think they got away without us noticing, Mike! :-)
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 4:03 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Whoops, nope, unresponsive again. I'll not jump to such strange conclusions in the future.
Top
   
Reply with quote  
mikegrb
 Post subject:
PostPosted: Wed Jun 28, 2006 4:03 pm 
Offline
Linode Staff
User avatar

Joined: Fri Oct 17, 2003 12:38 am
Posts: 287
Location: Dr Wierd's Lab, South Jersey Shore
Xan wrote:
No, I'm in Texas, actually. Here's a traceroute to he.net.

Code:
# traceroute he.net
traceroute to he.net (216.218.186.2), 30 hops max, 52 byte packets
<snip>
13  hurricane-113208-dls-bb1.c.telia.net (213.248.92.22)  106.121 ms  104.466 ms  103.746 ms
14  pos5-0.gsr12012.lax.he.net (66.160.184.5)  433.108 ms  379.999 ms  449.236 ms
                                                                                             


I had assumed hop 13 here was in Chicago based on the path of the traffic from the other customer in Ontario. It seems from your traceroute it is more likely that it is in Equinix, Los Angeles. At any rate, this is the same router that the Ontario customers were having issues with so it looks like your problems are related after all.
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Wed Jun 28, 2006 4:06 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Here's the perspective of another server I have in Dallas:

Code:
# mtr --report host18.linode.com
HOST: layer0                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 241.160.92.64.reverse.layere  0.0%    10    0.5   2.3   0.5  16.7   5.1
  2. 10.1.3.13                     0.0%    10    0.6   0.6   0.6   0.8   0.1
  3. 216.39.69.53                  0.0%    10    0.5   0.6   0.4   0.8   0.1
  4. bhr2-po-1.fortworthda1.savvi  0.0%    10    0.4   0.5   0.4   0.6   0.1
  5. dcr2-so-3-2-0.dallas.savvis.  0.0%    10    1.6   1.6   1.4   1.7   0.1
  6. dcr1-so-7-2-0.losangeles.sav  0.0%    10   34.1  34.1  33.9  34.9   0.3
  7. bpr3-so-7-0-0.losangelesequi  0.0%    10   34.2  34.2  34.0  34.3   0.1
  8. 208.174.196.54                0.0%    10   33.8  38.0  33.8  75.2  13.1
  9. hurricane-108839-las-bb1.c.t  0.0%    10  359.7 342.5 318.3 359.8  15.3
 10. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
Top
   
Reply with quote  
miallen
 Post subject:
PostPosted: Wed Jun 28, 2006 4:07 pm 
Offline
Senior Newbie

Joined: Thu Jan 08, 2004 3:53 pm
Posts: 13
Website: http://www.ioplex.com/~miallen/
Location: New York
Code:
# mtr --report 66.220.1.142
HOST                                    LOSS  RCVD SENT    BEST     AVG   WORST
???                                     100%     0   16    0.00    0.00    0.00
bogus.union.nj.panjde.comcast.net    0%    16   16   11.37   13.35   22.42
po10-ur02.union.nj.panjde.comcast.net     0%    16   16   11.38   13.11   15.44
po10-ur01.jerseycity.nj.panjde.comcast.net    0%    16   16   12.01   13.50   17.35
po10-ur02.jerseycity.nj.panjde.comcast.net    0%    16   16   11.84   13.76   15.93
po10-ur01.narlington.nj.panjde.comcast.net    0%    16   16   12.75   14.32   17.82
po70-ar01.verona.nj.panjde.comcast.net    0%    16   16   13.13   24.51  178.02
ar01.plainfield.nj.panjde.comcast.net     0%    16   16   13.60   15.59   23.77
GE-2-0-cr01.plainfield.nj.core.comcast.net    0%    16   16   13.46   14.83   17.13
12.118.149.9                              0%    16   16   14.38   16.95   23.59
tbr2-p014001.n54ny.ip.att.net             0%    16   16   15.51   17.22   19.74
12.123.0.93                               0%    16   16   14.96   16.29   18.34
nyk-b2-link.telia.net                     0%    16   16   14.29   17.00   30.07
nyk-bb2-pos1-1-0.telia.net                0%    16   16   14.57   15.81   17.08
chi-bb1-pos7-0-0-0.telia.net              0%    16   16   35.42   38.68   48.66
dls-bb1-pos6-0-0.telia.net                0%    16   16   56.44   61.10  108.59
hurricane-113208-dls-bb1.c.telia.net      0%    16   16   56.19   57.69   58.84
pos5-0.gsr12012.lax.he.net                0%    16   16   89.36   92.35  100.59
???                                     100%     0   16    0.00    0.00    0.00
Top
   
Reply with quote  
mikegrb
 Post subject:
PostPosted: Wed Jun 28, 2006 4:07 pm 
Offline
Linode Staff
User avatar

Joined: Fri Oct 17, 2003 12:38 am
Posts: 287
Location: Dr Wierd's Lab, South Jersey Shore
miallen wrote:
My node is totally unresponsive right now. Actually it comes and goes. Is this what you're seeing? How do you know it's HE? What kind of diagnostic are you using?


Traceroute and mtr output is the same as what one would expect with a saturated link. This can be caused by a link not having enough capacity to handle normal surge traffic at the busiest part of the day. Another frequent cause is an outage of one link causing saturation of another link that is still up due to the much greater than normal traffic. Upon finding this I contacted the HE NOC and they confirmed that the problem was indeed a saturated link and they were working with Telia to get it upgraded.
Top
   
Reply with quote  
raman
 Post subject: HE network problms
PostPosted: Wed Jun 28, 2006 4:11 pm 
Offline
Senior Newbie

Joined: Tue Nov 30, 2004 10:01 pm
Posts: 17
This is the trace from Coral Springs, Florida:

Code:
HOST: ?????                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. router.local                  0.0%    10    1.1   1.1   0.9   1.6   0.3
  2. 10.125.192.1                  0.0%    10    8.3  10.4   6.9  22.1   4.4
  3. hsrp1-cs.myacc.net            0.0%    10    7.9  10.0   7.5  15.5   2.4
  4. border5.3-5.advancecable-7.m  0.0%    10    9.5  13.1   9.5  29.4   6.7
  5. core2.pc1.bbnet1.mia003.pnap  0.0%    10   16.2  11.8   9.6  16.5   2.6
  6. 12.118.175.81                 0.0%    10    9.3  10.8   9.3  13.0   1.3
  7. tbr2-p013603.ormfl.ip.att.ne  0.0%    10   26.8  29.3  26.7  38.1   3.5
  8. tbr1-cl1474.attga.ip.att.net  0.0%    10   27.0  28.5  26.9  33.4   2.0
  9. ggr2-ge00.attga.ip.att.net    0.0%    10   26.4  44.9  25.8 181.8  48.8
 10. 192.205.33.42                 0.0%    10   28.7  30.7  25.3  55.8   9.4
 11. dls-bb1-link.telia.net        0.0%    10   54.4  53.9  47.3  91.1  13.3
 12. hurricane-113208-dls-bb1.c.t  0.0%    10   48.7  49.1  45.5  60.0   4.0
 13. pos5-0.gsr12012.lax.he.net   10.0%    10  366.4 349.0 319.9 366.4  15.3
 14. pos3-2.gsr12416.pao.he.net   10.0%    10  239.0 112.2  88.0 239.0  48.7
 15. pos5-0.gsr12416.fmt.he.net   10.0%    10  169.0 212.9 122.3 359.2  83.5
 16. pos10-0.gsr12012.fmt.he.net   0.0%    10   90.8  90.8  87.7 106.2   5.5
 17. host16.fremont.linode.com    10.0%    10   88.8  89.5  88.5  91.2   1.2


Excuse my ignorance but why do the high latencies show up on the second router after the Telia link?
Top
   
Reply with quote  
mikegrb
 Post subject: Re: HE network problms
PostPosted: Wed Jun 28, 2006 4:16 pm 
Offline
Linode Staff
User avatar

Joined: Fri Oct 17, 2003 12:38 am
Posts: 287
Location: Dr Wierd's Lab, South Jersey Shore
raman wrote:
This is the trace from Coral Springs, Florida:

Code:
HOST: ?????                      Loss%   Snt   Last   Avg  Best  Wrst StDev
<snip>
 12. hurricane-113208-dls-bb1.c.t  0.0%    10   48.7  49.1  45.5  60.0   4.0
 13. pos5-0.gsr12012.lax.he.net   10.0%    10  366.4 349.0 319.9  366.4  15.3
<snip>


Excuse my ignorance but why do the high latencies show up on the second router after the Telia link?


Number 12 is a Telia router. Number 13 is an HE router. The latency and packet loss starts at hop 13 and continues. This indicates the problem is the link between these two routers.
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic

Board index » Linux Community Forums » General Discussion


Who is online

Users browsing this forum: No registered users and 2 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group