Munin issue - active but not running
Everything has uinstalled as per the tutorials from munin, and I have installed the dependencies. I then copied our existing plugins over to the new server, and created the symlinks for them.
For some reason, the munin-node-configure –suggest still shows that many of the plugins (i.e. cpu) are not active, and attempting to add the symlink gives the error that it already exists.
The webpage that shows up when going to the new munin servers IP address is the default apache "It Works" page. Even though I have set the munin.conf file by uncommenting the following:
dbdir /var/lib/munin
htmldir /var/cache/munin/www
logdir /var/log/munin
rundir /var/run/munin
# Where to look for the HTML templates
#
tmpldir /etc/munin/templates
I have also updated the apache24.conf and apache.cong file to reflect the server. I essentially made the configuration identical to the existing working server.
The one thing that doesn't exit on the existing server is apache, but I do have the apache.conf in the /etc/munin directory.
In apache24.conf, I followed a websites recommendation to comment out "Require local" and add "Require all granted" and "Options FollowSymLinks SymLinksIfOwnerMatch" in order to make sure that it can be accessed by more than just the localhost.
This hasn't changed anything either.
apache24.conf****
Alias /munin /var/cache/munin/www
<directory var="" cache="" munin="" www=""># Require local
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch</directory>
ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<location munin-cgi="" munin-cgi-graph=""># Require local
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
<ifmodule mod_fcgid.c="">SetHandler fcgid-script</ifmodule>
<ifmodule !mod_fcgid.c="">SetHandler cgi-script</ifmodule></location>
I then tested to see if debugging would throw an error, and it did not. (This applies to all plugins that I debug)
root@munin:/etc/munin# munin-run -debug cpu
# Processing plugin configuration from /etc/munin/plugin-conf.d/munin-node
# Processing plugin configuration from /etc/munin/plugin-conf.d/snmp_communities
# Setting /rgid/ruid/ to /112/65534/
# Setting /egid/euid/ to /112 112/65534/
# Setting up environment
# About to run '/etc/munin/plugins/cpu'
user.value 286879
nice.value 29712
system.value 32167
idle.value 659700
iowait.value 555
irq.value 0
softirq.value 889
steal.value 0
guest.value 0
root@munin:/etc/munin#
I then tried running cron as munin and got the following error:
root@munin:/etc# su -s /bin/bash munin
munin@munin:/etc$ munin-cron
[FATAL ERROR] Lock already exists: /var/run/munin/munin-update.lock. Dying.
at /usr/share/perl5/Munin/Master/Update.pm line 128.
munin@munin:/etc$ grep munin-cron /etc/cron.*/*
/etc/cron.d/munin:*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
munin@munin:/etc$ cat /etc/cron.d/munin
#
# cron-jobs for munin
#
MAILTO=root
*/5 * * * * munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
14 10 * * * munin if [ -x /usr/share/munin/munin-limits ]; then /usr/share/munin/munin-limits --force --contact nagios --contact old-nagios; fi
When checking the /var/run/munin/ to see if had duplicate crons, I do not, and there are no files in /var/spool/cron/crontabs.
root@munin:/etc/munin# ls -l /var/run/munin/
total 12
-rw-r--r-- 1 munin munin 5 Sep 6 11:26 munin-limits.lock
-rw-r--r-- 1 root root 5 Sep 6 10:18 munin-node.pid
-rw-r--r-- 1 munin munin 4 Sep 6 11:25 munin-Servers-<server>.<domain>.lock
root@munin:/etc/munin#</domain></server>
root@munin:/var/spool/cron/crontabs# ls -l
total 0
Another forum suggested that something may be killing the process (i.e. a plugin running that shouldn't be), so ran thier suggestion. Oddly, it appears it tried (and succeeded?) at adding the symlinks for plugins such as cpu.
root@munin:/etc/munin# munin-node-configure --suggest --remove-also --shell
rm -f '/etc/munin/plugins/apache_accesses'
rm -f '/etc/munin/plugins/apache_processes'
rm -f '/etc/munin/plugins/apache_volume'
ln -s '/usr/share/munin/plugins/cpu' '/etc/munin/plugins/cpu'
ln -s '/usr/share/munin/plugins/df' '/etc/munin/plugins/df'
ln -s '/usr/share/munin/plugins/df_inode' '/etc/munin/plugins/df_inode'
ln -s '/usr/share/munin/plugins/entropy' '/etc/munin/plugins/entropy'
ln -s '/usr/share/munin/plugins/forks' '/etc/munin/plugins/forks'
ln -s '/usr/share/munin/plugins/fw_packets' '/etc/munin/plugins/fw_packets'
rm -f '/etc/munin/plugins/http_loadtime'
rm -f '/etc/munin/plugins/if_'
rm -f '/etc/munin/plugins/if_err_'
ln -s '/usr/share/munin/plugins/interrupts' '/etc/munin/plugins/interrupts'
ln -s '/usr/share/munin/plugins/irqstats' '/etc/munin/plugins/irqstats'
ln -s '/usr/share/munin/plugins/load' '/etc/munin/plugins/load'
ln -s '/usr/share/munin/plugins/memory' '/etc/munin/plugins/memory'
ln -s '/usr/share/munin/plugins/munin_stats' '/etc/munin/plugins/munin_stats'
rm -f '/etc/munin/plugins/mysql_'
ln -s '/usr/share/munin/plugins/open_files' '/etc/munin/plugins/open_files'
ln -s '/usr/share/munin/plugins/open_inodes' '/etc/munin/plugins/open_inodes'
ln -s '/usr/share/munin/plugins/proc_pri' '/etc/munin/plugins/proc_pri'
ln -s '/usr/share/munin/plugins/processes' '/etc/munin/plugins/processes'
ln -s '/usr/share/munin/plugins/swap' '/etc/munin/plugins/swap'
ln -s '/usr/share/munin/plugins/threads' '/etc/munin/plugins/threads'
ln -s '/usr/share/munin/plugins/uptime' '/etc/munin/plugins/uptime'
ln -s '/usr/share/munin/plugins/users' '/etc/munin/plugins/users'
ln -s '/usr/share/munin/plugins/vmstat' '/etc/munin/plugins/vmstat'
# The following plugins caused errors:
# hddtemp_smartctl:
# Junk printed to stderr
# ip_:
# Nothing printed to stdout
# No valid suggestions
# pgbouncer_connections:
# Junk printed to stderr
# pgbouncer_requests:
# Junk printed to stderr
# postfix_mailqueue:
# Non-zero exit during autoconf (255)
# postgres_autovacuum:
# Non-zero exit during autoconf (255)
# postgres_bgwriter:
# Non-zero exit during autoconf (255)
# postgres_cache_:
# Non-zero exit during autoconf (255)
# postgres_checkpoints:
# Non-zero exit during autoconf (255)
# postgres_connections_:
# Non-zero exit during autoconf (255)
# postgres_connections_db:
# Non-zero exit during autoconf (255)
# postgres_locks_:
# Non-zero exit during autoconf (255)
# postgres_oldest_prepared_xact_:
# Non-zero exit during autoconf (255)
# postgres_prepared_xacts_:
# Non-zero exit during autoconf (255)
# postgres_querylength_:
# Non-zero exit during autoconf (255)
# postgres_scans_:
# Non-zero exit during autoconf (255)
# postgres_size_:
# Non-zero exit during autoconf (255)
# postgres_transactions_:
# Non-zero exit during autoconf (255)
# postgres_tuples_:
# Non-zero exit during autoconf (255)
# postgres_users:
# Non-zero exit during autoconf (255)
# postgres_xlog:
# Non-zero exit during autoconf (255)
# proc:
# In family 'auto' but doesn't have 'autoconf' capability
# slony_lag_:
# Junk printed to stderr
# tomcat_:
# Non-zero exit during autoconf (127)
root@munin:/etc/munin#
However, checking –suggest, I still don't see cpu running (small amount of output added):
root@munin:/etc/munin# munin-node-configure --suggest
Plugin | Used | Suggestions
------ | ---- | -----------
acpi | no | no [cannot read /sys/class/thermal/thermal_zone*/temp]
amavis | no | no
apache_accesses | yes | no [Port 80: Can't connect to 127.0.0.1:80]
apache_processes | yes | no [Port 80: Can't connect to 127.0.0.1:80]
apache_volume | yes | no [Port 80: Can't connect to 127.0.0.1:80]
apc_envunit_ | no | no [no units to monitor]
bonding_err_ | no | no [No /proc/net/bonding]
courier_mta_mailqueue | no | no [spooldir not found]
courier_mta_mailstats | no | no [could not find executable]
courier_mta_mailvolume | no | no [could not find executable]
cps_ | no | no
cpu | no | yes
cpuspeed | no | no [missing /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state]
cupsys_pages | no | no [could not find logdir]
df | no | yes
df_inode | no | yes
diskstats | yes | yes
entropy | no | yes
There is no munin forum, and I can't think of what to do next. Need recommendations on next steps to get this running. Let me know if there is anything else you'd need to see.