I'm using a web crawler that is now using close to the maximum CPU.

https://github.com/KrasnayaSecurity/Wik ... crawler.py

Earlier designs did not use so much CPU, and I think it may have to do with the extra features added to prevent recrawling the same webpages. Memory management is better because of redundant URLs being removed. It used to be projected that it could not run for more than a day because the memory would fill up with the URLs in the crawl list.

If there were multiple crawlers running or other important processes running, would the CPU time just go to them instead and maybe make the crawlers run slightly slower, or would it cause a problem?



EDIT

The memory seems like it's starting to plateau and the crawler seems to be slowing down as if it has been getting almost all the pages on that site.