Hey folks, I'm maintaining a server with a couple of virtual servers (Teamspeak and a BYOND server), and I'm looking to make a script or two that'll run every couple of minutes to check if the applications are still alive, and in the case of the Byond server, to check if it's hogging the CPU, and check if it's doing this more than just once (check three times maybe? details on that I'll figure out myself).

In simple terms what I'm asking is, how do I execute a command in a screen without actually attaching to the screen? I'll be using crontab, if this is of relevance.

Thanks

I don't really understand what you mean by running "in a screen." Typically, server processes (often called daemons) are run in the background, not attached to a tty. Among other methods, they can be interacted with via network connections (local or remote) or by using signals.

Rather than rolling your own scripts, you should consider running an existing piece of software like monit on your server. It can check whether services are responding as well as their memory and CPU usage and restart them if need be. It's very flexible in the conditions and tests you can set to trigger events. It can send notifications as well, alongside or instead of restarting services. systemd also has the built-in capability to restart services that have exited, but I don't know if it can handle the other cases.

Hey, sorry for the slow response.

Thanks for the tip to use monit, will definitely use it. But that doesn't really solve my main issue, which is how I want to, via a script or command, execute a command in a detached screen. Normally I'd have to attach to it (screen -r ScreenName), punch in the command and detach (CTRL-A-D). I wish to do this via a script I'm going to put in a user's crontab. The reason I'm using screens for this is how the process actually needs an active session in order to stay alive, else it'll just close itself. Like, I run "DreamDaemon codebase.dmb -trusted -logself -core -thread on 12345", which will start DreamDaemon with the world file codebase.dmb on trusted mode, logging itself, dumping the core in case of a crash, turn threaded mode on and run the server on port 12345. After I press enter on this I'm left with a blank line, and if I close the session or if I'm on a screen, kill the screen, the process will kill itself. So I need something where I can, in case of a crash or CPU overload loop, have the server detect either a crash or prolonged >95% CPU usage and restart/kill and restart the server accordingly, in a screen session. Further, the process has to be run by a user that isn't root, for security purposes. By my own rule btw, the program wouldn't mind if I started it as root. If this is an issue, I could always run it as root just to make it easier.

This is a pretty urgent issue, as I'm having the process hog 100% of the CPU quite often (around every 18 hours I'd say), while I'm sleeping: http://imgur.com/BuFM2t8 (that's in local time, I'm eight hours ahead of this. I fixed this around 10 PM my time, as seen by the drop). While I'm working on actually fixing the source of the problem, I'd like a temporary fix that'll keep the server avalible for the community for as much as I can. Besides, I need a server restart script anyways.

tl;dr: Thanks for monit, will use. Byond doesn't run in the background though for reasons of ??? (stupid devs? we may never know), so I need to connect it to a tty/screen session, or keep a session up forever myself. So I still need a script that will reboot the server in the screen that's already there in the case of a crash / forced termination.

Hope that gives some insight in what I'm trying to do

Edit: (moar) elaborated +tl;dr

Thanks, that clears things up considerably. I have to admit I don't use screen and am just going by the man page, so the syntax may be off on my suggestions. But this should at least get you started.

You can write a script to kill the misbehaving daemon. Processes attached to a screen session are just normal processes, and can be interacted with using normal commands like ps and kill.

#!/bin/sh

count = 0
# Signal 0 just tests if a process exists
while killall -s 0 DreamDaemon >/dev/null 2>&1 ; do
  killall DreamDaemon >/dev/null 2>&1
  count = $(( $count + 1 ))
# Exit with error if we have tried 10 times unsuccessfully
  if [ $count -gt 10 ] ; then exit 1 ; fi
  sleep 1
done

This will try 100 times to kill any process named "DreamDaemon" - if at any iteration of the loop there is no process named "DreamDaemon" running, it will exit successfully. If it reaches the 100th time, it gives up and exits with an error. This is to alert whoever is running the script (monit) that there was a problem.

If you want to kill the screen session also, that requires something more fancy. It might be better to leave it running, so you can attach later and see if there was any worthwhile output. To kill both, maybe something like this:

#!/bin/sh

# Get process ID of first program named "DreamDaemon"
daemonpid = $(killall -v -s 0 DreamDaemon 2>&1 | sed 's/^.*(//
s/).*$//' | head -n 1)

# Get process ID of parent process (screen)
parentpid = $(ps -p $daemonpid -o ppid --no-headers)

count = 0
while kill -s 0 $daemonpid $parentpid >/dev/null 2>&1 ; do
kill $daemonpid $parentpid >/dev/null 2>&1
  count = $(( $count + 1 ))
# Exit with error if we have tried 10 times unsuccessfully
  if [ $count -gt 10 ] ; then exit 1 ; fi
  sleep 1
done

A note of warning - this is vulnerable to a race condition. If the first kill attempt succeeds, but while the script is sleeping another process starts with either the daemon or parent PID, the script will kill that process (PIDs get reused). I wouldn't recommend putting this into production without some tweaking.

You can apparently also send commands to a running screen session using screen -X command, but that's an exercise I'll leave to you.

You would also need a way to start the daemon; looks like something like screen -d -m /path/to/DreamDaemon codebase.dmb -trusted -logself -core -thread on 12345 would work.

So when configuring monit, you would set the stop command to be your kill script, and the start command to something like what's in the paragraph above. monit can be configured to alert you if either command exits with an error status.

Edits: modified scripts to only try 10 times (if it hasn't responded by then, it likely never will) and added warning about second script.