« previous next »
Pages: [1]   Go Down

Author Topic: PHP, Curl and XML Parser  (Read 664 times)

Offline Shrapnel

  • Level 9
  • *
  • Posts: 46
  • Reputation: +0/-0
    • View Profile
PHP, Curl and XML Parser
« on: September 05, 2010, 08:28:19 PM »
I'm trying to use the xml data provided by the GMs of a game I'm currently playing, but I'm having difficulty.  I'd appreciate any help.  Here's my code:
Code: [Select]
<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml");

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$output = curl_exec($ch);

curl_close($ch);    


$p = xml_parser_create();
xml_parse_into_struct($p, $output, $vals, $index);
xml_parser_free($p);
echo "Index array\n";
print_r($index);
echo "\nVals array\n";
print_r($vals);


?>
It doesn't work though.  Instead I get the following error:
Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 40 bytes) in /home/cased/public_html/IllyriadTownData.php on line 19

Can anyone help?
« Last Edit: September 05, 2010, 08:31:56 PM by Shrapnel »
Logged
"Never compromise. Not even in the face of Armageddon" -Rorschach, Watchmen (2009)

Offline Nox

  • Level 35
  • **
  • Posts: 768
  • Reputation: +12/-2
    • View Profile
Re: PHP, Curl and XML Parser
« Reply #1 on: September 06, 2010, 02:56:19 AM »
Em, well... my Opera froze upon openning that file so... I assume it's just too large to be processed with that memory size...

Line 19 is xml_parse_into_struct? (please highlight it upon posting since we don't see line numbers)

Currently I see following ways:
a) split the file and process it by parts, however given xml is hierarchy it might be more difficult
b) write your own parser, probably using some metadata and free everything as soon as possible, write (temp)results to file
c) parse it offline (unless it's mandatory to be parsed online), either by some program or php script with modified memory size/max execution time limits
Logged
Meet us at an IRC irc.freenode.net #bbg as well
https://vimeo.com/36579366 (a must-watch) | Join BOINC - no longer a hype, but you can help never the less

Offline Chris

  • Game Owner
  • Level 35
  • *
  • Posts: 2,217
  • Reputation: +28/-1
    • View Profile
Re: PHP, Curl and XML Parser
« Reply #2 on: September 06, 2010, 04:37:47 AM »
The xml file has 10MB. Typical PHP memory execution limit is 8MB (for files and script and variables), on free hosts might be less.

You need to chop the file into parts (assuming you can not increase memory limit). It would be no problem with normal fopen, but with xml... The answer might be "event-based parser", I have no clue what it is but that's what they recommend in PHP manual comments :D
Logged

Offline aquilax

  • Level 5
  • *
  • Posts: 20
  • Reputation: +1/-0
    • View Profile
    • Internet games
Re: PHP, Curl and XML Parser
« Reply #3 on: September 06, 2010, 08:46:03 AM »
Use the php's XMLReader http://www.php.net/manual/en/class.xmlreader.php It's a little tricky to work with states but it handles large files quite well.
Logged

Offline Shrapnel

  • Level 9
  • *
  • Posts: 46
  • Reputation: +0/-0
    • View Profile
Re: PHP, Curl and XML Parser
« Reply #4 on: September 08, 2010, 09:53:50 AM »
I revised my script based on the suggestions here (thanks guys).  It's been a pain in the butt, but I finally got it to print out town names and player names, however it's not 100% correct.  It's assigning towns to the wrong players.  Here's the new code:



Code: [Select]


<?php
     
  
$ch = curl_init("http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml");
$fp = fopen("towndata.xml", "w");

curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);

curl_exec($ch);
curl_close($ch);
fclose($fp);

  $reader = new XMLReader();
  $reader->open("towndata.xml");
  echo("<HTML><BODY>");
  
  
 
   while ($reader->read()) {
     
   FindElement($reader,"mapx");
   $mapx = $reader->value;
   FindElement($reader,"mapy");
   $mapy = $reader->value;
   FindElement($reader,"terraintype");
   $terraintype = $reader->value;
   FindElement($reader,"playername");
   $playername = $reader->value;
   FindElement($reader,"playerrace");
   $playerrace = $reader->value;
   FindElement($reader,"alliancename");
   $alliancename = $reader->value;
   FindElement($reader,"allianceticker");
   $allianceticker = $reader->value;
   FindElement($reader,"alliancetaxrate");
   $alliancetaxrate = $reader->value;
   
   FindElement($reader,"townname");
   $townname = $reader->value;
   
   FindElement($reader,"population");
   $population = $reader->value;
   FindElement($reader,"iscapitalcity");
   $iscapitalcity = $reader->value;
   FindElement($reader,"isalliancecapitalcity");
   $isalliancecapitalcity = $reader->value;
      
   if($allianceticker == "FDU")
   {
    echo "Town Name:  ", $townname, "<BR>";
    echo "Player Name:  ", $playername, "<BR>";
    
   }
   
   }
   
   echo("</BODY></HTML>");
   
        
function FindElement(&$readerobject, $name)
{
    while($readerobject->name != $name)
 {
  if(!$readerobject->read()) break;
 }
 if($readerobject->name == $name) 
 {$readerobject->read();
  return 1;
 }
 else return 0;
}   
?>




FindElement is a function I wrote to skip to the appropriate tag that contains the information I'm seeking.  I'm not familiar enough with XMLReader and had a heck of a time finding any examples or tutorials on how to use it.  The problem may be there.
Logged
"Never compromise. Not even in the face of Armageddon" -Rorschach, Watchmen (2009)

Offline Shrapnel

  • Level 9
  • *
  • Posts: 46
  • Reputation: +0/-0
    • View Profile
Re: PHP, Curl and XML Parser
« Reply #5 on: September 09, 2010, 07:39:14 AM »
I finally found the problem.  I think the script worked just fine but it made assumptions about the xml file that turned out to not be true.  I basically had to rewrite it, but it's working so far.
Logged
"Never compromise. Not even in the face of Armageddon" -Rorschach, Watchmen (2009)
Pages: [1]   Go Up
« previous next »
 
SimplePortal 2.3.3 © 2008-2010, SimplePortal