Skip to content Skip to sidebar Skip to footer

GAE Python: Parsing Compressed XML Exceeds Memory

I am trying to fetch and parse an XML file into a databse. The XML is compressed in GZIP. The GZIP file is ~8MB. When I run the code locally the memory on pythonw.exe builds up to

Solution 1:

What do you expect? First, you read a string into the memory, then - unzip it into the memory, then - construct a DOM tree, still in the memory.

Here are some improvements:

  1. del every buffer variable the moment you don't need it.
  2. Get rid of DOM XML parser, use event-driven LXML to save memory.

Post a Comment for "GAE Python: Parsing Compressed XML Exceeds Memory"