View All Products
Compute
Storage
Databases
Networking
Developer Tools
Delivery
Security
Services
Industries
Pricing
Community
Engage With Us
Community ▶ Questions ▶ no longer able to use SSH key-pair login
Log in to Ask a Question

no longer able to use SSH key-pair login

email

Working from a Debian 9 laptop.

Considering making new authentication key-pair:

https://www.linode.com/docs/security/se … n-key-pair">https://www.linode.com/docs/security/securing-your-server#create-an-authentication-key-pair

This afternoon. Cannot SSH from the Comcast address. At the same time email is working.

Ran sudo fail2ban-client status. Check blocked IP addresses in SSHD and SSHD-DDOS jails. Comcast address not there.

sudo cat /var/log/secure | grep Comcast IP address

no hits today

sudo cat /var/log/maillog | grep Comcast IP address

no hits today

sudo cat /var/log/dovecot.log | grep Comcast IP address

looking fine

sudo cat /var/log/sogo/sogo.log | grep Comcast IP address

no hits today

CentOS 7 email server had been fine for months.

Issues:

June 29th had Postfix (iRedMail) block an IP address from a customer. I was on-site . Used LISH access to remove the iptables blocks on INPUT and FORWARD each time.

June 30th:

4 AM yum-cron updated the kernel.

6:40 AM Emergency Maintenance. Linode restarted the VM after host issue.

July 1st to 3rd. That customer’s IP address and my residential Comcast address have been periodically blocked by the firewall.

Added addresses to IP address whitelist for OSSEC and Fail2Ban. Most has fixed the problems.

OSSEC is host-only install. Have email alerts active at level 7.

Have LogWatch installed.

Saw such as:

sudo iptables -L INPUT -v -n | grep "IP address"

506 30304 DROP all -- * * IP address 0.0.0.0/0

sudo cat /var/ossec/logs/active-responses.log| grep IP address

Sun Jul 2 17:42:48 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete – IP address 1499014544.273703 604

Sun Jul 2 17:42:48 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete – IP address 1499014544.273703 604

Sun Jul 2 17:42:51 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete – Comcast IP address 1499014544.273703 604

No more hits since.

Have not seen what action caused the blocks in the first place days ago.

Searched:

/var/log/secure

/var/log/maillog

/var/log/messages

/var/ossec/logs/active-responses.log

Rebooted VM before posting. No change.

11 Replies

I believe that a reboot will not "fix" anything, if firewalld/iptables reload their rules on boot. So whatever is blocked will continue to be blocked after a reboot. You need to clear the rules, in case of firewalld you need to use the –permanent parameter.

Check the following files in CentOS: /etc/sysconfig/iptables-config & /etc/sysconfig/iptables6-config, especially the options:

IPTABLESSAVEON_STOP="no"

IPTABLESSAVEON_RESTART="no"

Based on the above messages, it seems like one or more scripts, wrongly parsed the logs and generated/run a broken iptables rule. My suggestion, is to disable ossec and fail2ban, then clear all the firewalld/iptables rules, then you can check your connectivity status.

LISH in via SSH.

Changed the two options in /etc/sysconfig/iptables-config and /etc/sysconfig/ip6tables-config.

From no to yes.

Nothing in iptables blocking the Comcast address. At same time cannot SSH. Keeps timing out.

Email works.

sudo vim /etc/ssh/sshd_config

Change to no to disable tunnelled clear text passwords

PasswordAuthentication no to yes

sudo systemctl restart sshd

[Tue Jul 04 10:28:47 :~ ] $ ssh Linode IPv4 address -l user

ssh: connect to host Linode IPv4 address 22: Connection timed out

[Tue Jul 04 10:31:26 :~ ] $ ssh Linode IPv4 address -l user

ssh: connect to host Linode IPv4 address 22: Connection timed out

[Tue Jul 04 10:34:33 :~ ] $ ssh Linode IPv6 -l user

Last login: Mon Jul 3 13:33:35 2017

And then blocked by firewall?

sudo iptables -L INPUT -v -n | grep "Comcast IPv4 address"

nothing

sudo ip6tables -L INPUT -v -n | grep "Comcast IPv4 address"

nothing

[Tue Jul 04 10:42:23 :~ ] $ cat /etc/hosts.deny

ALL:145.236.80.46

ALL:186.178.173.28

ALL:46.148.18.163

sudo netstat -tulpn

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 26072/sshd

tcp6 0 0 :::22 :::* LISTEN 26072/sshd

PING from Comcast. Guessing Comcast has an issue.

[Tue Jul 04 10:45:59 :~ ] $ ping -c 3 www.linode.com

PING www.linode.com(2600:3c00::22 (2600:3c00::22)) 56 data bytes

64 bytes from 2600:3c00::22 (2600:3c00::22): icmp_seq=1 ttl=52 time=78.8 ms

64 bytes from 2600:3c00::22 (2600:3c00::22): icmp_seq=3 ttl=52 time=473 ms

–- www.linode.com ping statistics ---

3 packets transmitted, 2 received, 33% packet loss, time 2030ms

rtt min/avg/max/mdev = 78.886/276.307/473.729/197.422 ms

[Tue Jul 04 10:46:05 :~ ] $ ping -4c 3 www.linode.com

PING www.linode.com (72.14.191.202) 56(84) bytes of data.

–- www.linode.com ping statistics ---

3 packets transmitted, 0 received, 100% packet loss, time 2029ms

[Tue Jul 04 10:46:20 :~ ] $ ping -c 3 www.google.com

PING www.google.com(qm-in-x93.1e100.net (2607:f8b0:400d:c02::93)) 56 data bytes

64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=1 ttl=42 time=52.6 ms

64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=2 ttl=42 time=47.1 ms

64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=3 ttl=42 time=44.7 ms

–- www.google.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2002ms

rtt min/avg/max/mdev = 44.714/48.193/52.683/3.340 ms

[Tue Jul 04 10:46:38 :~ ] $ ping -4c 3 www.google.com

PING www.google.com (209.85.201.106) 56(84) bytes of data.

–- www.google.com ping statistics ---

3 packets transmitted, 0 received, 100% packet loss, time 2024ms

Well, I suggested that you clear ALL firewalld/iptables rules before proceeding, it is entirely possible that you are using a net mask that blocks a range of addresses, thus a simple "grep" for your IP won't produce any results.

You can delete all rules in a chain (for example INPUT) like this:

iptables -F INPUT

Thanks IfThenElse,

Believe there is something also going on with the Comcast stack.

Anyway, LISH via SSH. Ran:

sudo iptables -F INPUT

sudo iptables -F FORWARD

sudo ip6tables -F INPUT

sudo ip6tables -F FORWARD

sudo iptables-save

sudo ip6tables-save

ssh Linode IPv4 address -l username

Port 22 timeout. That command usually works.

Called Comcast, may take an in-home visit to sort out.

Soon after last work the Linode started to block emails from the customer again.

Ran:

sudo iptables -L INPUT -v -n | grep "IPv4 customer address"

693 41580 DROP all – * * IPv4 customer address 0.0.0.0/0

sudo iptables -D INPUT -s IPv4 customer address -j DROP

sudo iptables -D FORWARD -s IPv4 customer address -j DROP

sudo ip6tables -D INPUT -s IPv4 customer address -j DROP

address not found

sudo ip6tables -D FORWARD -s IPv4 customer address -j DROP

address not found

sudo iptables-save

sudo firewall-cmd --zone=public --add-rich-rule 'rule family="ipv4" source address= IPv4 customer address accept'

See if that is enough.

Ran ssh-keygen -b 4096. ssh-copy-id to the Linode.

SSH to Linode's IPv6 address works. IPv4 address times out, coming from Comcast.

If this is a CentOS 7 system, check that firewalld isn't loading extra rules on top of iptables. Essentially, both iptables and firewalld are just front-ends for the kernel module netfilter. Look under /etc/firewalld/ or use the firewall-cmd command.

Finally, why not try traceroute between the two systems? maybe there is something wrong with IPv4 routing.

Should have time to look at those later.

Also last night ran:

sudo firewall-cmd –zone=iredmail --add-rich-rule 'rule family="ipv4" source address=40.132.185.99 accept'

Customer’s IPv4 address blocked again after 1:30 EST.

sudo cat /var/ossec/logs/active-responses.log | grep Customer’s IPv4 address

Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:06 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/host-deny.sh add - Customer’s IPv4 address 1499233267.56991 604

Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh add - Customer’s IPv4 address 1499233267.56991 604

Wed Jul 5 01:41:07 EDT 2017 Unable to run (iptables returning != 1): 2 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604

You do not seem to be going anywhere. The firewall block seems to be a symptom, but not the real problem. I've already mentioned above that you should disable OSSEC and fail2ban first, clear the firewall (and hosts.deny) and see how things go from there.

Maybe you should contact whoever installed and configured OSSEC.

Yesterday morning /etc/firewalld/zones/iredmail.xml to have

Last night ran:

sudo /var/ossec/bin/ossec-control stop

sudo iptables -F INPUT

sudo iptables -F FORWARD

sudo ip6tables -F INPUT

sudo ip6tables -F FORWARD

sudo iptables-save

sudo ip6tables-save

sudo /var/ossec/bin/ossec-control start

Customer’s emails continued to arrived overnight. First time in about a week.

Found the XML file unchanged.

Ran: sudo cat /var/ossec/logs/active-responses.log | grep Customer’s IPv4 address. Better than before.

Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:06 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604

Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/host-deny.sh add - Customer’s IPv4 address 1499233267.56991 604

Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh add - Customer’s IPv4 address 1499233267.56991 604

Wed Jul 5 01:41:07 EDT 2017 Unable to run (iptables returning != 1): 2 - /var/ossec/active-response/bin/firewall-drop.sh delete - 40.132.185.99 1499189996.357265 604

Wed Jul 5 10:55:30 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499233267.56991 604

Wed Jul 5 10:55:31 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499233267.56991 604

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Log in to Reply
Community Code of Conduct