You should really spin up a second linode and do comparative testing... It just costs a few cents...

I have a second linode already, but it's hard to simulate a similar load (especially for that tokens table, as a token somewhat corresponds to (specific interactions) with a logged in user). Jmeter would allow be to send a lot of requests to Apache, but the tokens table would be untouched unless I configure jmeter to login 20 users at the same time.

I guess as a side question if someone knows a tool that may be more suitable than jmeter to simulate load with authenticated users, it would help (I would need jmeter to be able to alternatively select POST values for user/pwd from a list for example, which doesn't seem possible now, at least to the extent of my knowledge).

You could always switch production traffic over to a database on the second linode for a day and see if the same problems occur, and then switch back the following night if they do.

My two linodes are 70ms apart, I feel that would hurt performance.. but I can give it a try I guess (when traffic goes down on the site, which is expected in a couple weeks maybe).

You can spin up a new linode in the same datacenter as the existing one for minimal cost...

Since the problem is below the client request layer how about just tweaking the top level PHP code to imagine that it's been receiving the requests. E.g., just write a top level loop (with any values/delays you want) and the execute the request handling path code just as it would be with a real web request.

You could also write some code to actually behave like a client, but that seems like it would be more work with very little diagnostic benefit.

Ideally you'd be able to generate a few delayed cases over a very short period (like no more than minutes). Having a controlled way to replicate the issue without depending on user traffic would be a big win when it comes to narrowing this down further.

Then you can re-run the same process under different scenarios and on different machines.

-- David