First of all, thanks everyone for your insights.
hawk7000 wrote:
Looks like the varying routes start already in the networklayer.com network, as the traffic sometimes seem to go out through telia.net and sometimes through gblx.net? (Based on the varying hosts at the same hopcount in the trace above.)
That's true - I noticed some of the replies came from different hosts, but I didn't see a pattern in there. I agree with you, networklayer.com is probably switching routes between telia.net and gblx.net - which makes it hard to tell what network (telia.net or gblx.net) makes SCP take this long - or whether it's the switching altogether that causes it.
It still puzzles me that sometimes SCP finishes within seconds, and sometimes only after minutes have passed - yet the routes change within a traceroute. I'd assume that at some point, the speed of SCP would pick up if the route changes, or at least that it remains consistently low if route flapping itself is the issue - but instead the speed is either consistently slow, or consistently fast during a transfer.
hoopycat wrote:
This smells a lot like inconvenient packet loss.
Try laying down a ping while doing the scp, or maybe even mtr. Whatever is causing the routing to change is probably also dropping packets for a few seconds.
That's a good idea - it's funny how I use traceroute and all, and then forget using one of the most basic tools. I guess I neglected that because I assumed that commercial connections would not possibly have packet loss, and that it'd be a problem constrained to poor ADSL lines.
Anyway, you were correct; I ran a ping during the scp, and it would have a packet loss of around 20%.
For a 25 second SCP, I got
Code:
--- node.in.europe ping statistics ---
26 packets transmitted, 20 received, 23% packet loss, time 25026ms
rtt min/avg/max/mdev = 127.911/129.334/131.692/0.947 ms
and for a 2 minute SCP I got
Code:
--- node.in.europe ping statistics ---
140 packets transmitted, 114 received, 18% packet loss, time 139113ms
rtt min/avg/max/mdev = 126.246/128.959/133.576/1.009 ms
None of the replies were delivered out of order. During the "fast" scp (8 seconds) I had no packet loss. I tries to run SCP over a different port (1235) in an attempt to see whether port 1234 would be throttled, but I get the same figures.
So, thanks to you I realized that the problem is (a pretty significant?) packet loss, even though I am not sure, and probably can't find out, whether it's congestion caused or caused by the route switching. With that, is there anything I can do? Is this something Linode (or the German hoster) have any influence over? As in, could (and would) Linode choose a different route, or is it out of their hands anyway (in which case I wouldn't bother asking), because it's no longer in their network?
Telia.net is in Stockholm, networklayer.com's whois information is proxied (Domains By Proxy), which strikes me a bit as odd, seeing hiding contact information is something I would only expect individuals to do. Still, who would have more influence over the route - Linode or the provider in Germany?