Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Board index » Linux Community Forums » Web Servers and Web App Development
  Previous topic | Next topic 
Author Message
1st
 Post subject: Big file?
PostPosted: Sun Jul 29, 2007 3:22 am 
Offline
Newbie

Joined: Mon Jul 16, 2007 5:12 pm
Posts: 2
Hi there, I quite recently bought a linode, and was wondering how to get this script working...

Here is the PHP code
Code:
<?
set_time_limit(0);
mysql_connect("localhost","root","[myrootpassword]");
mysql_select_db("db1");
$words=file("words.txt");

foreach ($words as $word) {
$word = mysql_escape_string(rtrim($word));
$query = mysql_query("INSERT INTO `des` (word) VALUES ('{$word}')");
}
echo "worked.";
?>


For every line in there, it's suppose to add it into the database. It works for small files, however, the file I am working with is around 2gb
So I updated the memory allocation in php.ini yet, when I run this I get a blank page... any suggestions?

Thank in advance,
a new user
Top
   
Reply with quote  
Xan
 Post subject:
PostPosted: Sun Jul 29, 2007 12:48 pm 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
It could take quite some time to insert a 2GB file line by line. Are you sure it's not actually working, until your Web server times out? Try running it without going through the Web server.

As an aside, I think one of the biggest problems with PHP is that it encourages quoting rather than binding variables when passing commands to SQL. One mistake and you've got an injection attack. http://sqlrelay.sourceforge.net/sqlrelay/programming/phppeardb.html#bindvars
Top
   
Reply with quote  
1st
 Post subject:
PostPosted: Sun Jul 29, 2007 3:10 pm 
Offline
Newbie

Joined: Mon Jul 16, 2007 5:12 pm
Posts: 2
Ah, I wouldn't worry about injection attacks, the page is basically isolated from the rest of the content, with no input executed, except that which is escaped from the inserting method. With few even knowing how to access the front page.

Well, I set it up so it wouldn't time out, and it doesn't even load. Just generates a blank page, with no queries being executed. Ideas?

By the way, when I try this on another server, it inserts the queries... but, it stops at 50000, for quota reasons.
Top
   
Reply with quote  
kenny
 Post subject: Re: Big file?
PostPosted: Tue Aug 07, 2007 4:58 pm 
Offline
Senior Member

Joined: Sat Jun 28, 2003 12:02 am
Posts: 66
Website: http://kenny.aust.in
1st wrote:
Hi there, I quite recently bought a linode, and was wondering how to get this script working...

Here is the PHP code
Code:
<?
$words=file("words.txt");
?>

It works for small files, however, the file I am working with is around 2gb


file reads an entire file into an array.. so you are trying to read a 2gb file into memory. Try fgets and insert one line at a time.

Code:

<?
set_time_limit(0);
mysql_connect("localhost","root","[myrootpassword]");
mysql_select_db("db1");

$handle = fopen("words.txt", "r");
while (!feof($handle)) {
    $word = fgets($handle);
    $word = mysql_escape_string(rtrim($word));
    $query = mysql_query("INSERT INTO `des` (word) VALUES ('{$word}')");
}
echo "worked.";
?>


kenny
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic

Board index » Linux Community Forums » Web Servers and Web App Development


Who is online

Users browsing this forum: No registered users and 3 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group