I'm running postgres, tomcat (with 2 webapss), mysql and apache.

I've had another linode with the same 128mb setup (and corresponding cpu entitlement) and same apps installed and running. Everything was a lot faster on that one. Even an ssh session 'lags'. This is all the time not just at high load times. And I'm on a super duper fast connection here.

Are the servers getting overloaded or is there something wrong with my setup?

Cheers
Willie

You've hit the IO limiter:

io_count=8002877 io_rate=499 io_tokens=-193 token_refill=512 token_max=400000

due to your high swap usage (most likely):

Filename Type Size Used Priority
/dev/ubd/1 partition 263160 130320 -1

-Chris

So what do I need to do to figure this problem out?

I need more ram ie move up ? What's the best way to benchmark and figure this thing out?



What are 'good' values for those and what do they mean?


My CATALINA_OPTS are

"-Xint -Xrs -Xmx48m -Xverify:none -XX:+UseParallelGC"

IE only 48m ram. But how does the jre (1.5) manage that. IE, will it hit 48mb of ram then start using swap or is it locked at using a total of 48mb ram + swap.

So if I were to increase it to say 96mb it could just overflow the rest of the ram I specify and use swap cause there isn't enough on the system.

What's the best way to figure out if its swapping or using ram?

I'm used to having 1-2gb of ram so this memory limited env is hard for me. I'm sure it can easily be done

Cheers
Willie

Ah... been doing some reading on this forum. I kinda understand a bit more but some advice would be great.

This is what my cat /proc/io_status is now (i've been watching io_tokens come out of negative value)
io_count=12475222 io_rate=107 io_tokens=1823 token_refill=512 token_max=400000

From another post I found
io_count = total amount of tokens used since boot
io_rate = tokens being used per time unit (per second???)
io_tokens = number of tokens available to be used
token_refill = number of tokens that io_tokens will increase by per second.

My io_tokens keeps going negative. So anyway - untill I figure this thing out I've stopped my services to make sure I aint killing anyone else (although I suppose that's what the limiter stops by itself)

hehe. Just while I was writing this I ps aux 'd and found a rouge java process running ant. I tried to compile a webapp last night and it tanked. I thought it would be killed when I logged out but obviously it wasn't.

Even on my machine the jdk couldn't compile the same webapp with 768mb ram - it kept getting out of memory exceptions. So I finally compiled it using jikes instead (jikes literally did it in 30 seconds where jdk javac took 2 minutes then crashed with an out of memory exception)

Then I uploaded the war. But all this time it seems that process has been chomping away. But anyway - since I killed it it hasn't had a positive effect on my io_tokens. So I've still shut tomcat down till I figure out how to run it without killing the machine.

Currently my swap is 256mb for a 128mb linode. Perhaps I could resize my partitions and increase swap to 512mb? Or should I increase my jdk memory allocation for the tomcat process or should I just bite the bullet and fork out some more money to caker for some more memory

Cheers
Willie

Since I shut down tomcat my io_tokens has skyrocketed:

io_count=12793985 io_rate=0 io_tokens=71668 token_refill=512 token_max=400000

The console still lags a bit though..... Hmmmm....

Maybe I could use Jetty instead of Tomcat? I understand its pretty small on its memory usage. Though I'd rather not go through the pain of learning *another* app server inside out. Tomcat is *so* easy.

I increased my CATALINA_OPTS java heap to -Xmx64m.

Now:

io_tokens is sitting at around 395,000 - 400,000 (usually stays close to 400000)
io_rate is sitting at around 15 to 5000. (usually stays close to 15-30)

How can such a minor change make so much difference? I mean, I only added a miniscule 16mb to the java heap and its now rocking. Such a minor change couldnt *possibly* have such a huge effect could it?

Cheers
Willie

If the minor change makes the difference between having to swap and not having to swap, then it certainly could have a major impact on I/O performance.

You might try the 1.4 series JVM instead of 1.5 too. I've tried 1.5 on linode with jboss and it will get to a point where it seems to max out CPU usage and hang. I disabled class sharing or whatever it's called and that helped slightly but it still wasn't stable and used more memory than 1.4.