Hi,

I'm trying to set up an OpenVPN VPN, which will carry some (but not all) traffic from the clients to the internet via the OpenVPN server.

My OpenVPN server has a public IP on eth0, and is using tap0 to create a local network, 192.168.2.x. I have a client which connects from local IP 192.168.1.101 and gets VPN IP 192.168.2.3.

On the server, I ran:

iptables -A INPUT -i tap+ -j ACCEPT
iptables -A FORWARD -i tap+ -j ACCEPT

iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o eth0 -j MASQUERADE

On the client, the default remains to route via 192.168.1.1. In order to point it to 192.168.2.1 for HTTP, I ran

ip rule add fwmark 0x50 table 200
ip route add table 200 default via 192.168.2.1
iptables -t mangle -A OUTPUT -j MARK -p tcp --dport 80 --set-mark 80

Now, if I try accessing a website on the client (say, wget google.com), it just hangs there. On the server, I can see

$ sudo tcpdump -n -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
05:39:07.928358 IP 192.168.1.101.34941 > 74.125.67.100.80: S 4254520618:4254520618(0) win 5840 <mss 1334,sackOK,timestamp 558838 0,nop,wscale 5>
05:39:10.751921 IP 192.168.1.101.34941 > 74.125.67.100.80: S 4254520618:4254520618(0) win 5840 <mss 1334,sackOK,timestamp 559588 0,nop,wscale 5>

Where 74.125.67.100 is the IP it gets for google.com .

Why isn't the MASQUERADE working? More precisely, I see that the source showing up as 192.168.1.101 -- shouldn't there be something to indicate that it came from the VPN?

Pardon me if this sounds dumb to you, but have you actualy enabled IP forwarding? It's disabled by default on most distributions.

net.ipv4.ip_forward should be set to 1 with sysctl.

Sorry again if this is too obvious, because most of your post is actualy way over my little head :-)

Hope it helps...

Yes.

$ cat /proc/sys/net/ipv4/ip_forward
1

Your iptables line is looking for packets originating from 192.168.2.0/24 and going to eth0, while the source of the packet is actually 192.168.1.101. Changing the iptables line to match 192.168.1.0/24 should clear it up.

Hoopycat -- do you know why this is? 192.168.1.101 is the random local IP address that my client has; tomorrow, it could be anything else.

As far as the server is concerned, the only fixed address is 192.168.2.2 . How can I have the packets coming from the VPN come from that source?

EDIT: This also doesn't appear to be working:

# sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 169K packets, 8922K bytes)
 pkts bytes target     prot opt in     out     source               destination 
    0     0 ACCEPT     tcp  --  any    any     linode.mikeage.net   anywhere            tcp dpt:www

Chain POSTROUTING (policy ACCEPT 1109K packets, 70M bytes)
 pkts bytes target     prot opt in     out     source               destination 
   34  2040 MASQUERADE  all  --  any    eth0    192.168.1.0/24       anywhere   

Chain OUTPUT (policy ACCEPT 1109K packets, 70M bytes)
 pkts bytes target     prot opt in     out     source               destination 

Although no connection can be established on outgoing port 80 [maybe the problem is with the return?]

mike: have you tried setting up a simple vpn first? Something without all the extra things to specify which kind of traffic goes through?

Yep; it works just fine. In fact, I'm using it right now....

So the client can use the vpn, it all stops working only when you try to have only http traffic going through?

edit: I'm asking because I see you're using bridge mode (why?), while routing mode is usually recommended.

Anyway, you're using bridged mode, yet we see no mention of br0 in your config. Is forwarding on on the br0 interface?

The client can still use the VPN, but when I try and have traffic from the client routed to the internet via the VPN, that traffic never makes it out. tcpdump suggests that it is making it to the VPN gateway.

I have no fundamental interest in routing vs. bridging; I have no network on the server to speak of (it's just my linode). I did switch from dev tun to dev tap a while ago, since I found the /30 addressing to be quite confusing.

You can enable NAT on the VPN client (192.168.2.2). Then, the packets will appear to be coming from 192.168.2.2 when they hit your server. However, you'd then be NATting traffic twice, which is somewhat silly. (Remember, the packet *originated* at 192.168.1.101; 192.168.2.2 is merely an intermediate router.)



Hmmm... if you tcpdump eth0, are you seeing the traffic going out with the right addresses? I'm wondering if having that ACCEPT in PREROUTING is messing things up and causing it to skip the POSTROUTING... a skim through the man page doesn't indicate anything for sure, but a SNAT in POSTROUTING will cause it to ignore any further rules, so that'd be my next thing to try

True, but my concern is that in defining the rules, I can easily add 192.168.2.x to my iptables setup. I don't know how I can know ahead of time that the VPN will be passing traffic through from 192.168.1.101. Maybe tomorrow I'll connect to my VPN from a very generous public wifi hotspot, and I'd have a regular public IP?



Nope

# iptables -t nat -F
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 178K packets, 9359K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1151K packets, 73M bytes)
 pkts bytes target     prot opt in     out     source               destination
    1    60 MASQUERADE  all  --  any    eth0    192.168.1.0/24       anywhere

Chain OUTPUT (policy ACCEPT 1151K packets, 73M bytes)
 pkts bytes target     prot opt in     out     source               destination

And then I ran telnet google.com 80 from my client, while monitoring the server...

# tcpdump -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
17:37:44.541139 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840 <mss 1334,sackOK,timestamp 11338016 0,nop,wscale 5>
17:37:47.442837 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840 <mss 1334,sackOK,timestamp 11338766 0,nop,wscale 5>

Right now, you've got two machines involved at the endpoint: your workstation (192.168.1.101) and your VPN gateway (192.168.2.2). If you connect straight from your workstation (or laptop, or whatever), your traffic should be originating from 192.168.2.0/24 or whatever the OpenVPN stuff is assigning.



How about on the eth0 side? Do you see it as 192.168.1.101 or something else?

(For the record, I do use OpenVPN, but not NAT; for NAT-like taste with half the calories, I use ssh -D. I'm a wee bit lazy like that. However, in theory, an OpenVPN tunnel is Just Another Interface.)

192.168.2.2 is the address my laptop gets from the VPN. The VPN server (aka my linode) is 192.168.2.1.



Since you asked

$ sudo tcpdump -i eth0 -n | grep 74.125.45.100
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:47:18.039794 IP 209.123.162.201.50604 > 74.125.45.100.80: S 1187533034:1187533034(0) win 5840 <mss 1334,sackOK,timestamp 13374642 0,nop,wscale 6>
20:47:18.074996 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990835218 13374642,nop,wscale 6>
20:47:18.075020 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990835218 13374642,nop,wscale 6>
20:47:18.483612 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990835627 13374642,nop,wscale 6>
20:47:18.483636 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990835627 13374642,nop,wscale 6>
20:47:19.084976 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990836228 13374642,nop,wscale 6>
20:47:19.085019 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990836228 13374642,nop,wscale 6>
20:47:20.284925 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990837428 13374642,nop,wscale 6>
20:47:20.284994 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990837428 13374642,nop,wscale 6>
20:47:20.867654 IP 209.123.162.201.50604 > 74.125.45.100.80: S 1187533034:1187533034(0) win 5840 <mss 1334,sackOK,timestamp 13375392 0,nop,wscale 6>
20:47:20.891638 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990838035 13374642,nop,wscale 6>
20:47:20.891663 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990838035 13374642,nop,wscale 6>
20:47:21.884928 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990839028 13374642,nop,wscale 6>
20:47:21.884955 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 1430,sackOK,timestamp 1990839028 13374642,nop,wscale 6>

$ sudo iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 181K packets, 9535K bytes)
 pkts bytes target     prot opt in     out     source               destination 

Chain POSTROUTING (policy ACCEPT 1157K packets, 74M bytes)
 pkts bytes target     prot opt in     out     source               destination 
    3   180 MASQUERADE  all  --  any    eth0    192.168.1.0/24       anywhere   

Chain OUTPUT (policy ACCEPT 1157K packets, 74M bytes)
 pkts bytes target     prot opt in     out     source               destination 

It may not be obvious at first from the logs, but the traffic here was in bursts (retries?). I'm really leaning towards a problem with the return traffic, but I don't know what else needs to be set up to have it re-masquerade back. I'm also not sure if I should double NAT it... what do you think?

I've also used SOCKS5 proxies (or just an SSH tunnel to 3128): my goal here is to set up a PC to always use a transparent squid proxy (to prevent direct access to the internet).

Ok... I just tried something else. I disabled by entire firewall and tried to do just plain NAT from the VPN client to the internet.

$ sudo iptables -v -L
Chain INPUT (policy ACCEPT 18433 packets, 20M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 29 packets, 1740 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  eth0   tap+    anywhere             anywhere            state RELATED,ESTABLISHED
    6   360 ACCEPT     all  --  tap+   eth0    anywhere             anywhere
   26  1560 LOG        all  --  any    any     anywhere             anywhere            LOG level warning

Chain OUTPUT (policy ACCEPT 13868 packets, 2891K bytes)
 pkts bytes target     prot opt in     out     source               destination

$ sudo iptables -v -L -t nat
Chain PREROUTING (policy ACCEPT 612 packets, 30485 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 2541 packets, 162K bytes)
 pkts bytes target     prot opt in     out     source               destination
  646 41013 MASQUERADE  all  --  any    eth0    anywhere             anywhere

Chain OUTPUT (policy ACCEPT 3268 packets, 208K bytes)
 pkts bytes target     prot opt in     out     source               destination

I did set up logging on the forwarded packets that were not being accepted:

$ dmesg | tail
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19188 PROTO=TCP SPT=80 DPT=33708 WINDOW=5672 RES=0x00 ACK SYN URGP=0
device eth0 entered promiscuous mode
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19108 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19109 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19110 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19111 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19112 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19113 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0

[btw, I did switch my test machine from 192.168.1.101 to 192.168.1.100.]

If I'm reading this correctly, the packets returning from google's server are being sent to 192.168.1.100, but my VPS doesn't know how to get there (since it only recognizes the machine called 192.168.1.100 by the name of 192.168.2.2)

Maybe I do need double NAT?

Alas, you've officially exceeded my knowledge of iptables. Hopefully someone else will chime in with advice...