
Looks like it's not a speed issue with loading resources (which any plugin for caching/compressing will not help), but instead the clients are waiting 3+ second just to get a response to the request.
This sounds like either:
1. Apache/Nginx is handing on requests
2. PHP is running very slowly holding up the request
To rule out 2, if you copy all of the source of the HTML (go to View Source on the homepage, copy all the code, and create a file in your Document Root called test.html (or simular). Go to the webpage but with /test.html instead, and see how quick the page runs, if it's instant, then its probably PHP, if its still slow then it seems to be your Web Server. Can you then post you Web Server Configuration.
Regards,
Matt