I have about 5 websites running all with different TTFB but all made with Wordpress. I setup a new 1gb ram ubuntu 14.04LTS thinking a new clean install would fix it - but did not, unfortunately.
Once the homepage is loaded, the subpages load quick enough. My guess the server has a hard time loading all the Wordpress PHP, Ajax scripts and what not which causes the slow loads. Loading purely a http://www.domain.com/img.jpg from the server for example loads instantly so it's definately not a DNS problem.
TTFB averages 5 to 15 seconds, 30seconds or even refuses to load. Turning off plugins or changing themes helps a little bit but doesn't take care of the problem.
I use the W3 total cache plugin. I tried to switch on/off some of those options without too much difference in result.

Are there tests I can run that can find the cause of this? Or is there some memory or Apache setting that I can fine tune that will speed things up? I feel like I've researched all I could on this subject without any luck

My Setup:
PHP 5.5.9-1ubuntu4.9 (cli)
Apache/2.4.7
mysql Ver 14.14 Distrib 5.5.43,
I run ubuntu 14.04LTS of 1gb ram.
2800 MHz Intel(R) Celeron(R) M processor 1.50GHz

$ egrep --color 'Mem|Cache|Swap' /proc/meminfo
MemTotal: 1011464 kB
MemFree: 25740 kB
MemAvailable: 502984 kB
Cached: 433468 kB
SwapCached: 4868 kB
SwapTotal: 262140 kB
SwapFree: 210624 kB

Apache setup:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       6
    MaxSpareServers      12
    MaxClients          15
    MaxRequestsPerChild   3000
</IfModule>
KeepAlive Off
MaxKeepAliveRequests 100

Do you have APC or some other php cache installed? IIRC, the ubuntu package is called php-apc

Yes I have. one machine I've done the following:
APC is no longer needed in Ubuntu 14.04. It doesn't support PHP 5.5 which comes with its own built-in Opcache cache instead. To ensure that it is enabled, edit /etc/php5/apache2/php.ini and change:

;opcache.enable=0
 to 
opcache.enable=1

$ php5enmod opcache
$ service apache2 restart