i've recently been getting lots of disk io rate notifications in email. i looked at the 24 hour graph and found a cyclical pattern. i then looked at the 30d graph and found a scarier thing: the average io rate has been trending upward linearly the entire month.
i ran iotop -a -o -t -b | tee output.txt for a couple of hours, and the only thing out of the ordinary was kjournald. while i run a web server on this linode, there weren't any appreciable web requests to speak of.
even though kjournald was an outlier, it was only 60MB of disk io. i used vmstat simultaneously to try and line up the time of the disk io to the process that was causing it. the only thing i could find was that a spike in IO happened exactly 22 minutes apart. here are a few lines of vmstat output around the spike. the bo column always has upwards of 37000 blocks out, then another smaller line, and then yet smaller but noticeable couple of lines afterwards. you can also see the context switch column go wild.
are there tools i can use to determine the process that is running every 22 minutes in this way? is there a simple script i could write that monitors /proc in some way? i looked through user's crontabs and did not find anything in them. there are only a few users in the system (really just myself and some fake users for projects i am working on).
Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 37156 33912 49288 325860 0 0 0 2 97 56 0 0 100 0
0 0 37156 33912 49288 325868 0 0 0 9 99 58 0 0 100 0
1 0 37156 30392 49300 327688 0 0 28 35730 16746 13233 3 3 82 12
0 0 37156 32664 49292 326312 0 0 1 22774 10248 8391 2 2 85 11
0 0 37156 32904 49292 326324 0 0 0 8 106 59 0 0 100 0
0 0 37156 33036 49292 326336 0 0 0 448 147 59 0 0 100 0
0 0 37156 33036 49292 326348 0 0 0 3 109 57 0 0 100 0
FAQ
Search
Members
Register
Login [ Anonymous ]
