One of my linodes has high I/O usage and I don't know why.

blktrace doesn't work:

blktrace:

[chris@lin-monitor ~]$ sudo btrace /dev/xvda
statfs: No such file or directory
/sys/kernel/debug does not appear to be a valid path

[chris@lin-monitor ~]$ sudo mount -t debugfs none /sys/kernel/debug
mount: mount point /sys/kernel/debug does not exist

[chris@lin-monitor ~]$ zgrep DEBUG_FS /proc/config.gz 
# CONFIG_DEBUG_FS is not set

[chris@lin-monitor ~]$ uname -a
Linux lin-monitor.aptivate.org 3.2.1-linode40 #1 SMP Mon Jan 23 10:42:16 EST 2012 i686 i686 i386 GNU/Linux

Please could you enable DEBUG_FS in Linode kernels? Along with my previous request for the Xen watchdog?

http://cateee.net/lkddb/web-lkddb/XEN_WDT.html

(strangely this isn't even listed in /proc/config.gz, even though it's supposed to be in kernels > 2.6.39, which this is).

1) How high is "high"?

2) Have you tried seeing what is causing the IO using something like iotop? If you want strange things in your kernel, you need to run your own kernel; most of us don't want debug stuff in our production kernels.

3) It looks like Xen watchdog appears to require a newer version of Xen than Linode runs.

Hi Guspaz,

1) Average 1325 blocks/sec (5 MB/sec?)

2) Yes, I tried iotop, doesn't work either:

[chris@lin-monitor ~]$ sudo iotop
Traceback (most recent call last):
  File "/usr/bin/iotop", line 16, in ?
    main()
  File "/usr/lib/python2.4/site-packages/iotop/ui.py", line 567, in main
    main_loop()
  File "/usr/lib/python2.4/site-packages/iotop/ui.py", line 557, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python2.4/site-packages/iotop/ui.py", line 465, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib/python2.4/curses/wrapper.py", line 44, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python2.4/site-packages/iotop/ui.py", line 457, in run_iotop_window
    process_list = ProcessList(taskstats_connection, options)
  File "/usr/lib/python2.4/site-packages/iotop/data.py", line 375, in __init__
    self.update_process_counts()
  File "/usr/lib/python2.4/site-packages/iotop/data.py", line 431, in update_process_counts
    stats = self.taskstats_connection.get_single_task_stats(thread)
  File "/usr/lib/python2.4/site-packages/iotop/data.py", line 158, in get_single_task_stats
    reply = self.connection.recv()
  File "/usr/lib/python2.4/site-packages/iotop/netlink.py", line 229, in recv
    raise err
OSError: Netlink error: Invalid argument (22)

I thought this might also be due to missing kernel options, but I couldn't work out from the source code or the almost nonexistent iotop documentation which ones they might be, so I figured it might be a bug in iotop and didn't mention it above.

3) You don't have to use it You don't have to mount /sys/kernel/debug. If people really object to even having the DEBUG_FS option enabled in the kernel, perhaps Linode could provide a version with it enabled? But although it says DEBUG, it's mainly used for blktrace as far as I can tell, and this is really useful in production kernels. RHEL5 enables it in config-2.6.18-308.1.1.el5, and if that's not conservative enough for you, you might want to run a custom kernel yourself

Cheers, Chris.

Could you run a custom kernel with it enabled?

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)