Weird problem with Linode private network
I'm experiencing very strange behaviour and was wondering if anybody here can point me what's wrong with my thinking.
I have a couple of linodes. I want to deploy database cluster on those Linodes (postgres-xl in this instance). On each Linode I have public IP as well as private IP. I want to have my cluster talking via private IP.
I have built clean linux boxes (arch linux) on those Linodes. For testing purposes those systems have nothing on them, not even iptables.
I test if cluster will communicate on public interface first. So far so good - no errors, I see happy logs.
I move my cluster to use private network, and at this point I start seeing weird logs suggesting something is going on which I don't quite understand.
5:2544785216:2017-09-12 15:46:20.389 UTC -LOG: Started to run as GTM-Active.
LOCATION: main, main.c:740
1:2536441600:2017-09-12 15:46:25.219 UTC -LOG: could not receive data from client: Bad file descriptor
LOCATION: pq_recvbuf, pqcomm.c:524
2:2536441600:2017-09-12 15:46:25.219 UTC -FATAL: Expecting a startup message, but received �
LOCATION: GTM_ThreadMain, main.c:1098
3:2536441600:2017-09-12 15:46:25.219 UTC -LOG: could not send data to client: Bad file descriptor
LOCATION: internal_flush, pqcomm.c:846
I have done nothing apart from changing IP addresses (to use private addresses rather than public). I am using IP addresses assigned to my Linode via web management service.
I have spent a week trying to find out what's going on. Google is not very helpful, also I am not sure where to look for more information. I have compared tcpdump output when postgres-xl was configured on public interfaces with output when postgres-xl was configured on private interfaces. When configured on public interfaces I can see packets with RST,ACK flag and those packets are missing when configured on private interfaces. This is what made me thinking maybe there is some filtering done on infrastructure level within private network.
Is it possible that there is something going on private network that is preventing my cluster from being happy cluster?
2 Replies
did you setup the private IP addresses? If you used the network manager thing (it's newer, I'm not super familiar with it, never used it), I think that sets up your private IP, but if you're just using DHCP to get your IP, that'll only get your public IP, you need to separately bring up the private IP.
