I'm using ssh -D to tunnel my laptop's traffic through my Linode in Newark. It's been working fine (and I can see the dancing turtle). However, if I try to visit a site whose DNS servers are unresponsive or flaky, the whole tunneled connection blocks until the DNS lookup succeeds or times out after 15 seconds.

I noticed this because FreeDNS (afraid.org) is having some kind of massive outage today, and I have some sites using the afraid.org dns servers. For example, here's what happens when I try to hit www.anydns.com in Chrome over a tunneled connection, with ssh -vvv:

debug1: Connection to port 8888 forwarding to socks port 0 requested.
debug2: fd 11 setting TCP_NODELAY
debug3: fd 11 is O_NONBLOCK
debug3: fd 11 is O_NONBLOCK
debug1: channel 3: new [dynamic-tcpip]
debug2: channel 3: pre_dynamic: have 0
debug2: channel 3: pre_dynamic: have 3
debug2: channel 3: decode socks5
debug2: channel 3: socks5 auth done
debug2: channel 3: pre_dynamic: need more
debug2: channel 3: pre_dynamic: have 0
debug2: channel 3: pre_dynamic: have 16
debug2: channel 3: decode socks5
debug2: channel 3: socks5 post auth
debug2: channel 3: dynamic request: socks5 host www.anydns.com port 80 command 1

Here the tunnel locked up for 15 seconds before the DNS lookup timed out, and then:

channel 3: open failed: administratively prohibited: open failed
debug2: channel 3: zombie
debug2: channel 3: garbage collecting
debug1: channel 3: free: direct-tcpip: listening port 8888 for www.anydns.com port 80, connect from ::1 port 58806, nchannels 4
debug3: channel 3: status: The following connections are open:
  #2 client-session (t4 r0 i0/0 o0/0 fd 8/9 cc -1)

debug3: channel 3: close_fds r 11 w 11 e -1

If I do this while I have a big HTTP download going in another window, the download rate drops to zero for 15 seconds, then resumes right back at full speed. So this affects active connections too.

I cranked up the sshd log level to VERBOSE, but all I get in /var/log/auth.log is

sshd[2664]: error: connect_to www.anydns.com: unknown host (Name or service not known)

Ideas?

_________________
Hartford makes bread and jeans, and paper art.

Interesting bug known: https://bugzilla.mindrot.org/show_bug.cgi?id=1357

(edit - fix to right bug number as opposed to something I got interested in)

Nice find, thanks. That thread does indeed describe the symptom in question.

The workaround recommended (turning off AAAA lookups to appease broken DNS servers that choke on those queries) won't help in this case, because that's not the problem here; the servers in question are under DDoS attack, and aren't responding reliably to any queries.

Anyway, the IPv6 RR brokenness seems beside the point. The real issue, not addressed in the bugzilla thread, is that the entire tunneled connection shouldn't block on DNS resolution in the first place. Right? It makes sense that any given forwarding request needs to block until its hostname is resolved remotely, but as far as I know there's no technical reason the whole tunnel (including active connections) should freeze. Is this really how it's intended to work? Aside from the serious connection instability from garden-variety public internet flakiness, it seems like an easy way to driveby DoS anyone using this type of tunnel.

I'm aware one could circumvent this by doing DNS resolution locally, rather than forwarding it over the tunnel. But sometimes you don't want to do that (e.g. captive-portal airport WiFi, or behind the Great Firewall, both of which have been known to do DNS packet inspection). It's not an ideal solution.

Somebody with more knowledge of ssh internals want to tell me if I'm barking up the right tree here?

_________________
Hartford makes bread and jeans, and paper art.

You're right, it's stupid that it blocks on DNS lookups.

Personally, I suffer through it instead of doing anything proactive like you are.

_________________
Matt Nordhoff (aka Peng on IRC)

In the meantime, perhaps a better solution might be something like sshuttle. I find that it performs much more smoothly than a simple ssh tunnel. You can even use it to tunnel programs that don't know how to use SOCKS proxies, such as flash videos. All you need is python on the remote server and a privileged user account on your laptop (because sshuttle plays some clever tricks with iptables).

Hah, sshuttle is written by my old boss, Avery, from when I worked at Net Integration Inc. (now owned by IBM).

Yeah, sshuttle is awesome (no synchronous DNS lookups or connect(), not to mention doing away with tcp-over-tcp). However, it doesn't support ipv6 at all.

Your old boss (who seems like a very smart guy) also wrote this rant, so I'm not holding my breath on him adding it. Wish I had the time and expertise to write something myself. Looks like it's back to the torpid turtle for now.

_________________
Hartford makes bread and jeans, and paper art.

Warning: Long rambling reminiscing ahead.

He is a really smart guy. He works at Google now

He was the founder of the company (well, one of two, IIRC), but he had no interest in running it. Instead, he basically made himself the lead developer, and ran the R&D division in Montreal (it was purposefully put in Montreal, when corporate was near Toronto, to insulate us from the corporate culture). I had more than a few long talks with him about cool tech ideas he had, it was fascinating. He had some ideas for what amounted to a P2P cloud-based CDN that were pretty intriguing, and this was years before cloud computing was a thing.

Other neat stuff he did was wvdial, which I believe became the standard way for Linux boxes to use dialup. He also wrote netselect, which you basically fed a whole lot of IP addresses or hostnames, and then it used a variety of metrics to figure out which was the closest/most performant, which was a really great way to find fast debian mirrors.

There was also a lot of neat stuff built into the product we were selling, a semi-embedded Linux distro called Nitix. The big emphasis was autonomic computing, and one of the features that impressed me the most, despite not being all that sophisticated, was how it dealt with IP allocations.

The idea was that server was for small companies, to be trivial to set up, and they could just plug it into their LAN and start using it to run their IT services (mail, web, routing, etc). The first thing it would do is try to get an IP address from a DHCP server. If nothing responded, it would listen to existing traffic for a bit to figure out what subnets were the most common, and then assign itself an unused IP in that subnet. If still no DHCP server responded, and there were clients making unanswered DHCP requests, it would just step up and start assigning IPs to machines. In an enterprise network, that's a really bad thing, but in the context we're talking about here, it produces a "works like magic" scenario that was delightful.

Sadly, the priorities of the company shifted later in its life. They added a redhat-based virtualization system based on chroot jails with some extra custom code (kind of like virtuozzo), which was neat in that it took advantage of the other services provided by Nitix like the reliability, automatic incremental backup, multi-system-partition resiliency, etc. I worked on that early on as QA lead (which is a fun way of saying the only QA programmer), and it was neat, but eventually the entire product emphasis shifted to that. And then, over time they came to focus strictly on hosting Lotus software.

As I was finishing up my last co-op work term there, the company was in the process of shutting down the Montreal R&D office, and then not long after, IBM bought the company and merged it into the Lotus division, where Nitix got renamed Lotus Foundations.