I've followed multiple tutorials on trying to get VPN working on my 10.04 server. I started with this:
http://www.linode.com/wiki/index.php/An ... PPSKServer

It looks like the IPSec is working but its failing on the L2TP side....not sure where to go with this.

Here's my auth.log:

happy pluto[15326]: packet from 1.2.3.4:500: received Vendor ID payload [RFC 3947] method set to=109
happy pluto[15326]: packet from 1.2.3.4:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02] meth=107, but already using method 109
happy pluto[15326]: packet from 1.2.3.4:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] meth=106, but already using method 109
happy pluto[15326]: packet from 1.2.3.4:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-00]
happy pluto[15326]: packet from 1.2.3.4:500: ignoring Vendor ID payload [FRAGMENTATION 80000000]
happy pluto[15326]: packet from 1.2.3.4:500: received Vendor ID payload [Dead Peer Detection]
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: responding to Main Mode from unknown peer 1.2.3.4
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: STATE_MAIN_R1: sent MR1, expecting MI2
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: NAT-Traversal: Result using RFC 3947 (NAT-Traversal): peer is NATed
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: STATE_MAIN_R2: sent MR2, expecting MI3
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: Main mode peer ID is ID_IPV4_ADDR: '192.168.0.128'
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: transition from state STATE_MAIN_R2 to state STATE_MAIN_R3
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: new NAT mapping for #3, was 1.2.3.4:500, now 1.2.3.4:4500
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: STATE_MAIN_R3: sent MR3, ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=aes_256 prf=oak                               ley_sha group=modp1024}
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: ignoring informational payload, type IPSEC_INITIAL_CONTACT msgid=00000000
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: received and ignored informational message
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #3: the peer proposed: XXX.XXX.131.54/32:17/1701 -> 192.168.0.128/32:17/0
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: responding to Quick Mode proposal {msgid:3a02acf5}
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4:     us: XXX.XXX.131.54<XXX.XXX.131.54>[+S=C]:17/1701
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4:   them: 1.2.3.4[192.168.0.128,+S=C]:17/0===192.168.0.128/32
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: keeping refhim=4294901761 during rekey
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: transition from state STATE_QUICK_R0 to state STATE_QUICK_R1
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: STATE_QUICK_R1: sent QR1, inbound IPsec SA installed, expecting QI2
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: transition from state STATE_QUICK_R1 to state STATE_QUICK_R2
happy pluto[15326]: "L2TP-PSK-NAT"[2] 1.2.3.4 #4: STATE_QUICK_R2: IPsec SA established transport mode {ESP=>0x0a36ebd1 <0x945700aa xfrm=AES_256-HM                               AC_SHA1 NATOA=none NATD=1.2.3.4:4500 DPD=none}

It gets that far every time and then stops. Then the client times out.

My xl2tpd output:

xl2tpd[15694]: IPsec SAref does not work with L2TP kernel mode yet, enabling forceuserspace=yes
xl2tpd[15694]: setsockopt recvref[22]: Protocol not available
xl2tpd[15694]: This binary does not support kernel L2TP.
xl2tpd[15694]: xl2tpd version xl2tpd-1.2.5 started on anger PID:15694
xl2tpd[15694]: Written by Mark Spencer, Copyright (C) 1998, Adtran, Inc.
xl2tpd[15694]: Forked by Scott Balmos and David Stipp, (C) 2001
xl2tpd[15694]: Inherited by Jeff McAdams, (C) 2002
xl2tpd[15694]: Forked again by Xelerance (www.xelerance.com) (C) 2006
xl2tpd[15694]: Listening on IP address 0.0.0.0, port 1701
xl2tpd[15694]: get_call: allocating new tunnel for host XXX.XXX.30.50, port 36421.
xl2tpd[15694]: handle_avps: handling avp's for tunnel 61924, call 0
xl2tpd[15694]: message_type_avp: message type 1 (Start-Control-Connection-Request)
xl2tpd[15694]: protocol_version_avp: peer is using version 1, revision 0.
xl2tpd[15694]: hostname_avp: peer reports hostname 'anonymous'
xl2tpd[15694]: framing_caps_avp: supported peer frames: async sync
xl2tpd[15694]: assigned_tunnel_avp: using peer's tunnel 10712
xl2tpd[15694]: receive_window_size_avp: peer wants RWS of 1.  Will use flow control.
xl2tpd[15694]: challenge_avp: challenge avp found
xl2tpd[15694]: get_call: allocating new tunnel for host XXX.XXX.30.50, port 36421.
xl2tpd[15694]: handle_avps: handling avp's for tunnel 10553, call 0
xl2tpd[15694]: message_type_avp: message type 1 (Start-Control-Connection-Request)
xl2tpd[15694]: protocol_version_avp: peer is using version 1, revision 0.
xl2tpd[15694]: hostname_avp: peer reports hostname 'anonymous'
xl2tpd[15694]: framing_caps_avp: supported peer frames: async sync
xl2tpd[15694]: assigned_tunnel_avp: using peer's tunnel 10712
xl2tpd[15694]: receive_window_size_avp: peer wants RWS of 1.  Will use flow control.
xl2tpd[15694]: challenge_avp: challenge avp found
xl2tpd[15694]: control_finish: Peer requested tunnel 10712 twice, ignoring second one.
xl2tpd[15694]: build_fdset: closing down tunnel 10553
xl2tpd[15694]: get_call: allocating new tunnel for host XXX.XXX.30.50, port 36421.
xl2tpd[15694]: handle_avps: handling avp's for tunnel 37720, call 19256
xl2tpd[15694]: message_type_avp: message type 1 (Start-Control-Connection-Request)
xl2tpd[15694]: protocol_version_avp: peer is using version 1, revision 0.
xl2tpd[15694]: hostname_avp: peer reports hostname 'anonymous'
xl2tpd[15694]: framing_caps_avp: supported peer frames: async sync
xl2tpd[15694]: assigned_tunnel_avp: using peer's tunnel 10712
xl2tpd[15694]: receive_window_size_avp: peer wants RWS of 1.  Will use flow control.
xl2tpd[15694]: challenge_avp: challenge avp found
xl2tpd[15694]: control_finish: Peer requested tunnel 10712 twice, ignoring second one.
xl2tpd[15694]: build_fdset: closing down tunnel 37720
xl2tpd[15694]: get_call: allocating new tunnel for host XXX.XXX.30.50, port 36421.
xl2tpd[15694]: handle_avps: handling avp's for tunnel 32441, call 7051
xl2tpd[15694]: message_type_avp: message type 1 (Start-Control-Connection-Request)
xl2tpd[15694]: protocol_version_avp: peer is using version 1, revision 0.
xl2tpd[15694]: hostname_avp: peer reports hostname 'anonymous'
xl2tpd[15694]: framing_caps_avp: supported peer frames: async sync
xl2tpd[15694]: assigned_tunnel_avp: using peer's tunnel 10712
xl2tpd[15694]: receive_window_size_avp: peer wants RWS of 1.  Will use flow control.
xl2tpd[15694]: challenge_avp: challenge avp found
xl2tpd[15694]: control_finish: Peer requested tunnel 10712 twice, ignoring second one.
xl2tpd[15694]: build_fdset: closing down tunnel 32441
xl2tpd[15694]: Maximum retries exceeded for tunnel 61924.  Closing.

I've also tried with on a box sitting at home running 10.04 and same issue. I have a feeling its a firewall issue, but I'm not an expert in iptables. I've disabled ufw and believe I wiped all rules from iptables.

I had a similar problem on CentOS 6, and found this solution: http://bugs.centos.org/view.php?id=5832

It may be unrelated to your problem, though.