Linode Forum

[JFlame]

I'm setting up my new Linode, trying to get a site that's on shared hosting over to it. Just today, while trying to import the database (14M worth of SQL DDL and DML) I got an out of memory error and the script failed. :(

I ran a ps aux and this is what I have:

Code:  
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2   2728  1140 ?        Ss   Nov06   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Nov06   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Nov06   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Nov06   0:23 [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S    Nov06   0:00 [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    Nov06   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S    Nov06   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    Nov06   0:00 [kworker/1:0]
root         9  0.0  0.0      0     0 ?        S    Nov06   0:00 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    Nov06   0:00 [migration/2]
root        11  0.0  0.0      0     0 ?        S    Nov06   0:00 [kworker/2:0]
root        12  0.0  0.0      0     0 ?        S    Nov06   0:00 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    Nov06   0:00 [migration/3]
root        14  0.0  0.0      0     0 ?        S    Nov06   0:00 [kworker/3:0]
root        15  0.0  0.0      0     0 ?        S    Nov06   0:00 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S<   Nov06   0:00 [khelper]
root        17  0.0  0.0      0     0 ?        S    Nov06   0:00 [kworker/u:1]
root        21  0.0  0.0      0     0 ?        S    Nov06   0:00 [xenwatch]
root        22  0.0  0.0      0     0 ?        S    Nov06   0:00 [xenbus]
root       148  0.0  0.0      0     0 ?        S    Nov06   0:05 [sync_supers]
root       150  0.0  0.0      0     0 ?        S    Nov06   0:00 [bdi-default]
root       152  0.0  0.0      0     0 ?        S<   Nov06   0:00 [kblockd]
root       162  0.0  0.0      0     0 ?        S<   Nov06   0:00 [md]
root       246  0.0  0.0      0     0 ?        S<   Nov06   0:00 [rpciod]
root       248  0.0  0.0      0     0 ?        S    Nov06   0:36 [kworker/0:1]
root       279  0.0  0.0      0     0 ?        S    Nov06   0:17 [kswapd0]
root       280  0.0  0.0      0     0 ?        SN   Nov06   0:00 [ksmd]
root       281  0.0  0.0      0     0 ?        S    Nov06   0:00 [fsnotify_mark]
root       285  0.0  0.0      0     0 ?        S    Nov06   0:00 [ecryptfs-kthrea]
root       287  0.0  0.0      0     0 ?        S<   Nov06   0:00 [nfsiod]
root       290  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsIO]
root       291  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsCommit]
root       292  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsCommit]
root       293  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsCommit]
root       294  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsCommit]
root       295  0.0  0.0      0     0 ?        S    Nov06   0:00 [jfsSync]
root       296  0.0  0.0      0     0 ?        S<   Nov06   0:00 [xfs_mru_cache]
root       297  0.0  0.0      0     0 ?        S<   Nov06   0:00 [xfslogd]
root       298  0.0  0.0      0     0 ?        S<   Nov06   0:00 [xfsdatad]
root       299  0.0  0.0      0     0 ?        S<   Nov06   0:00 [xfsconvertd]
root       300  0.0  0.0      0     0 ?        S<   Nov06   0:00 [glock_workqueue]
root       301  0.0  0.0      0     0 ?        S<   Nov06   0:00 [delete_workqueu]
root       302  0.0  0.0      0     0 ?        S<   Nov06   0:00 [gfs_recovery]
root       303  0.0  0.0      0     0 ?        S<   Nov06   0:00 [crypto]
root       865  0.0  0.0      0     0 ?        S    Nov06   0:00 [khvcd]
root       977  0.0  0.0      0     0 ?        S<   Nov06   0:00 [kpsmoused]
root       978  0.0  0.0      0     0 ?        S    Nov06   0:54 [kworker/1:1]
root       981  0.0  0.0      0     0 ?        S    Nov06   0:47 [kworker/2:1]
root      1008  0.0  0.0      0     0 ?        S    Nov06   0:02 [kjournald]
root      1012  0.0  0.0      0     0 ?        S    Nov06   0:50 [kworker/3:1]
root      1033  0.0  0.1   2368   544 ?        S    Nov06   0:00 upstart-udev-bridge --daemon
root      1035  0.0  0.0   2236   256 ?        S<s  Nov06   0:00 udevd --daemon
root      1180  0.0  0.0   2232   220 ?        S<   Nov06   0:00 udevd --daemon
root      1181  0.0  0.0   2232   220 ?        S<   Nov06   0:00 udevd --daemon
syslog    1911  0.0  0.1  29460   944 ?        Sl   Nov06   0:12 rsyslogd -c4
root      1928  0.0  0.1   2428   684 ?        Ss   Nov06   0:03 cron
root      2032  0.0  0.0   2288   452 ?        Ss   Nov06   0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclien
root      2065  0.0  0.1   3600   700 ?        Ss   Nov06   0:00 /usr/sbin/ntpd
ntpd      2066  0.0  0.1   3496   684 ?        S    Nov06   0:05 /usr/sbin/ntpd
root      8553  0.0  0.3   5600  1712 ?        Ss   Nov06   0:10 /usr/sbin/sshd -D
root     11839  0.0  0.0   1840   468 hvc0     Ss+  Nov07   0:00 /sbin/getty -8 38400 hvc0
mysql    12855  0.0  2.1 115564 11000 ?        Ssl  Nov07   7:01 /usr/sbin/mysqld
root     14091  0.0  0.1   4088   748 ?        Ss   Nov07   0:00 /usr/sbin/vsftpd
106      17669  0.0  0.0   6700   496 ?        Ss   Nov08   0:00 /usr/sbin/exim4 -bd -q30m
root     24751  0.0  0.4   8516  2424 ?        Ss   11:19   0:00 sshd: root@pts/0
root     24809  0.0  0.3   4720  1840 pts/0    Ss   11:19   0:00 -bash
root     24929  0.0  1.5  39056  7920 ?        Ss   11:37   0:00 /usr/sbin/apache2 -k start
www-data 24932  0.0  0.8  39144  4252 ?        S    11:37   0:00 /usr/sbin/apache2 -k start
www-data 24933  0.0  0.7  39128  4036 ?        S    11:37   0:00 /usr/sbin/apache2 -k start
root     24954  0.0  0.0      0     0 ?        S    12:04   0:00 [flush-202:0]
www-data 24967  0.0  0.7  39128  4048 ?        S    12:13   0:00 /usr/sbin/apache2 -k start
www-data 24968  0.0  0.7  39128  4048 ?        S    12:13   0:00 /usr/sbin/apache2 -k start
www-data 24977  0.0  0.7  39056  3952 ?        S    12:13   0:00 /usr/sbin/apache2 -k start
root     25050  0.0  0.1   3600   916 ?        Ss   12:19   0:00 ntpd 2066 0.0 0.1 3496 684 ? S Nov06 0:05 /usr/sbin/ntpd
ntpd     25051  0.0  0.1   3496   920 ?        S    12:19   0:00 ntpd 2066 0.0 0.1 3496 684 ? S Nov06 0:05 /usr/sbin/ntpd
root     25069  0.0  0.2   2764  1052 pts/0    R+   12:19   0:00 ps aux


Top shows this:

Code:
top - 12:22:12 up 19 days, 13:35,  2 users,  load average: 0.00, 0.01, 0.05
Tasks:  76 total,   1 running,  75 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    509084k total,   485356k used,    23728k free,    30152k buffers
Swap:   262140k total,    14152k used,   247988k free,   407212k cached


I'm confused, it doesn't look like the processes add up to the amount of memory that is being used. I'm fairly new to this, and need some suggestions.

This is an Ubuntu 10.04 Linode.

[JFlame]

I did some reading on memory reporting in Linux. So now I understand the meaning of "cached" in Top.

This command was helpful:

Code:
# free -m
             total       used       free     shared    buffers     cached
Mem:           497        450         46          0         29        357
-/+ buffers/cache:         63        433
Swap:          255          9        246


Plenty of free memory (433M).

So I'm not sure why MySql would hiccup on such a small script. My research indicates MySql doesn't have an upper bound memory setting.

Any ideas?
[/code]

[hoopycat]

You mention a "script"... are you importing a file of SQL commands (the output from mysqldump or similar) directly into MySQL (mysql [blah] < something.sql), or are you using some other thing? PHP does have its own memory limits, which an awkward import script could run into.

I do agree that MySQL shouldn't be having a problem here, nor does it look like the system itself has had a recent memory crunch.

[JFlame]

The script is a dump of both table structures and data from a Simplemachines Forum (SMF). I've dumped and reloaded entire databases from scripts like this multiple times on both CentOS (current host) and Windows (Local dev) without issues in the past.

I am running the file via mysql command line on the Linux box itself. the exact error is this:

Code:

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '<br />
<b>Fatal error</b>:  Out of memory (allocated 34603008) (tried to allocat' at line 1

[JFlame]

I can't believe this. The export script aborted and that out of memory error is the last line that was generated by the PHP extract.

The problem is with the extract.

Sorry to waste anybody's time.

[hybinet]

Haha, interesting. So you ran out of memory while you were producing the DB dump with a PHP script, but didn't realize it until you tried to reload the dump? I hope you didn't lose any data.

[JFlame]

hybinet wrote: Haha, interesting. So you ran out of memory while you were producing the DB dump with a PHP script, but didn't realize it until you tried to reload the dump? I hope you didn't lose any data.

Yup. The SMF db dump process didn't report any errors, and even gave me the file (partial).

I didn't lose anything, since I'm still "kicking the tires" on the Linode. I was trying to set up a test instance.

Another reason why I want away from "tools" that encapsulate the job that really needs to be done. I am much happier at the command line with full control.

[glg]

JFlame wrote: Another reason why I want away from "tools" that encapsulate the job that really needs to be done. I am much happier at the command line with full control.

You might check if your shared host has a backup on their panel, it likely will dump out the database. I came from one where the backup utility gave me a .tar.gz file. In it, it had the filesystem plus a mysqldump of the database.

[JFlame]

glg wrote: JFlame wrote: Another reason why I want away from "tools" that encapsulate the job that really needs to be done. I am much happier at the command line with full control.

You might check if your shared host has a backup on their panel, it likely will dump out the database. I came from one where the backup utility gave me a .tar.gz file. In it, it had the filesystem plus a mysqldump of the database.

Yes, I found they do have that capability and I dumped it out via that method and loaded it just fine.

The full backup utility is annoying though because it is limited to backing up the root directory, which has all my websites and exceeds the backup limit for the host. :(

Soon I'll be out of there though. The test instance of the site runs awesome on the Linode. I've never seen such good response times, even for an empty site on the shared host.

[glg]

JFlame wrote: The full backup utility is annoying though because it is limited to backing up the root directory, which has all my websites and exceeds the backup limit for the host. :(

wow, ingenious design they've got :? I think you'll be a lot happier here.