| Linode Forum https://forum.linode.com/ |
|
| Linode suddenly stuck on SYN_RECV for most requests https://forum.linode.com/viewtopic.php?f=19&t=7159 |
Page 1 of 1 |
| Author: | yuchant [ Wed May 25, 2011 6:15 pm ] |
| Post subject: | Linode suddenly stuck on SYN_RECV for most requests |
Hi everyone, My server suddenly started to timeout on every local request yesterday. I'm pretty inexperienced in networking and would love to learn a process for debugging these connectivity issues. What confuses me is that yesterday, some people (my phone, me at home, friends at home) could consistently access the site and I see with netstat that a connection has been established. I disabled firewalls and set iptables to accept all connections to rule out any strange auto rules blacklisting our IP. I'm not sure if its relevant but a traceroute from the local network times out - traceroute from some machines outside find my server. My phone can access 69.164.201.172, but not anybody on this network. Judging by the server load charts, I'm guessing it's not my network locked out. I can SSH into this linode from my other linode, but will get the SYN_RECV timeout if I attempt to ssh into my linode from my local network. I've confirmed various settings are correct by comparing to the settings on my development server. The following files match my dev environment (except for their respective ip addresses): Code: /etc/hosts Apache is listening on port 80 and the setup looks exactly the same as my functioning server. Code: # server that doesn't work: Here's my attempt at understanding what is happening... Every time I load the page once, netstat -an | grep :80 reveals all connections in SYN_RECV state. Code: tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN So the SYN_RECV means the server is waiting for an ACK to be sent back from the client. How do I debug whether an ACK is being sent back? How do I debug where this communication is failing? Here's what a tcpdump looks like when I attempt to load the page once. What does this mean? That the client isn't getting the response? Or perhaps I'm swallowing the response somewhere in the server? How do I know to narrow down the culprit further? Code: tcpdump -i eth0 -n -tttt port 80 tcpdump for functional server Code: 00:00:00.000000 IP 71.56.137.10.57260 > 72.14.189.46.80: Flags [S], seq 34114118s [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0 Any suggestions, explanations, or comments would be hugely appreciated so that I can understand TCP a little more and hopefully be a little more useful next time I need to debug a problem like this. I've gotten some feedback on ServerFault, but the netmask that appears in ifconfig is the same on both servers. Quote: The one time I've seen this before it was a strange timing issue. The connections were getting stuck in the half-open state (what SYN_RECV means) and hanging. What ended up being the problem was two fold:
The server had an incorrect netmask (/16 instead of /24) There were two devices on the server subnet that issue proxy-ARP packets Can anyone suggest where or how I debug this next? Even reading material to point me in the right direction would help. My solution at the moment is to just ignore this problem and upgrade the dev linode, but I'd really like to be able to figure this out. Thank you so much! EDIT: Update - the problem has fixed itself... but I wish it hadn't, so I could feel less helpless facing this situation. I'd like to be able to get enough information out of my debugging attempts to get a reasonable idea why something is failing. It seems like the fact that it fixed itself without any further input from me mean it was outside my control in the first place. |
|
| Author: | otherbbs [ Thu May 26, 2011 5:17 pm ] |
| Post subject: | Re: Linode suddenly stuck on SYN_RECV for most requests |
You mentioned traceroute from the local network timed out. I would guess there was a routing problem between the source location (home) that was experiencing the timeouts and the destination (linode). Disabling iptables is a good start to rule out problems. Then you should use mtr (winmtr for windows) from both sides of the connection points. That will give you an idea whether it's a routing issue. I have seen cases where traffic was getting to the destination (server), but the replies were not getting back to the workstation because of peering issues, which usually resolve themselves within a reasonable amount of time. Once you have ruled out networking issues, you can start looking at the application layer. -- Travis |
|
| Author: | airthrow [ Mon Aug 29, 2011 1:17 am ] |
| Post subject: | Same here |
Hi everyone, My server is also experiencing a similar issue. I get many SYN_RECV, SYN_SENT, and CLOSE_WAIT states. But currently, I'm seeing many SYN_RECV. This appears to be due to normal traffic and not a denial of service. example: # netstat -an |grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 173.255.221.179:80 66.249.68.147:40942 SYN_RECV tcp 0 0 173.255.221.179:80 124.115.0.169:58194 SYN_RECV tcp 0 0 173.255.221.179:80 220.181.108.111:9299 SYN_RECV tcp 0 0 173.255.221.179:80 220.181.94.227:62780 SYN_RECV tcp 0 0 173.255.221.179:80 119.63.196.107:44705 SYN_RECV tcp 0 0 173.255.221.179:80 173.255.221.179:55539 SYN_RECV tcp 0 0 173.255.221.179:80 69.171.224.244:59395 SYN_RECV tcp 0 0 173.255.221.179:80 173.255.221.179:55540 SYN_RECV tcp 0 0 173.255.221.179:80 173.242.125.206:51632 SYN_RECV tcp 0 0 173.255.221.179:80 173.242.125.206:43500 SYN_RECV another example: $ netstat -an | grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 1 0 173.255.221.179:80 123.125.67.204:28346 CLOSE_WAIT Because the SYN_RECV's don't go away, I think it's triggering my MaxClients limit. Thus, no new visitors can see my sites. It seems mainly due to bot crawlers. If so, not sure if there's anything we can do about that. This hasn't happened with other providers. ** Restarting apache2 does get things going again -- if only for a short period of time. For now, I've increased my MaxClients hoping it's good enough to wait out the ** hoping the problem at least goes way -- but like yuchant -- knowing the root cause is better. Thanks! |
|
| Page 1 of 1 | All times are UTC-04:00 |
| Powered by phpBB® Forum Software © phpBB Group http://www.phpbb.com/ |
|