I'm doing research into development of a computer vision algorithm to perform classification on a certain type of image. My prototype for the tester is written in Python, a file called "test.py". It runs through all of the images in a given directory, classifies them, prints data about them to the controlling terminal, and also stores the data in a CSV file. It does this in about 30 sec an image right now (need to speed it up later obviously).

When I only had 60 images I was running tests on, it completed successfully after about 25 minutes. I would usually run it in the background with:

nohup python test.py > test.out 2> test.err < /dev/null

I added more images to my directory so it is now 200+ images. Now the process keeps dying before completion! It just stops at a random point in the classification of an image. The process continues running at 1-2% CPU for a while, then usually it dies (sometimes it doesn't--I just noticed one that has been running for 16 hours at 1% CPU and 80% memory). I've tried it both backgrounded/nohuped and running it in the foreground. I'm starting to think this might be a problem with Linode.

Anybody encountered anything like this before? Any solutions?

What does dmesg say? Maybe it's being reaped by the kernel.

You can attach strace to a background process to see the syscalls it's executing.

You can similar things if you compile python with debug symbols and run it under gdb.

One suggestion: run the top command in an xterm window before starting. Either filter by user (your user name) or hit M (capital M) to sort by memory. Now run your program and watch the memory consumed with different numbers of images - 50, 100, 150, etc. and see what you find.

James

You called it. Python was just being a major memory hog, even though I del'd all of my variables and manually ran the garbage collector between loop iterations. I switched it to use multiprocessing and run each task as its own process. Works fine now. Thanks!

Excellent.

James