Hi,
I haven't used it, but maybe PRM (Process Resource Monitor) would fit the situation here.
http://rfxnetworks.com/prm.php
Quote:
PRM monitors the process table on a given system and matches process id's with set resource limits in the config file or per-process based rules. Process id's that match or exceed the set limits are logged and killed; includes e-mail alerts, kernel logging routine and more...
Download the current release of PRM distributed under the GNU GENERAL PUBLIC LICENSE:
...
Usual disclaimer - I'm not associated or affiliated with them. I find just find their GPL products of interest and they accept donations iif you like their work.
All the best,
Chuck