Just a wild idea... are you using IPv6? I mean, is your goal to make that website IPv6 accessible? If not, make sure that your web server is not listening on your IPv6 addresses. Its possible that IPv6 is trying to auto-configure or something when that traffic hits your Linode. That can cause a perceived disruption to IPv4 traffic, because a lot of (most?) browsers these days are configured to prefer IPv6 by default. So if an IPv6 link is coming up when you are browsing, you might be seeing that.
Just an idea.
If there aren't any AAAA records, it doesn't matter if the web server is listening on IPv6. If there are AAAA records, working IPv6 won't cause delays. (Unless, like, you have a relatively poor IPv6 route, but that's not what I meant.)