ZapImport memory handling

[BobAtTorrey]

In getting things set up recently I've been updating the EPG while I'm awake and watching and I wonder if there is a problem, or at least gross inefficiency, in the way ZapImport is working.

I've configured it to use SQLite and to re-use the zapdata.xml fetched by GBPVR's EPG update. In one run the ZapImport log file says "Total loaded" records was 8654. The zapdata.xml file was about about 20 MB, and the final zapimport.db3 was less than 7 MB.

YET, I watched zapimport's memory usage rise to about 325 MB, took 9 minutes 54 seconds to complete (with nothing else going on in the machine), and for most of that time took in the range of 80-99% of the CPU.

Is this as it should be?

[rowle1jt]

Mine does the same thing, so I would venture a guess it is normal. I set mine for 2am and let it go. My guess, is thats why sub programmed GBPVR to schedule it.

After all, 8000+ records is a fair amount of data to sort through in an XML file, I would think.

[BobAtTorrey]

Well, the job finishes without error, so I guess it's 'normal' in the sense of working the way it's been designed. But some of the numbers don't seem reasonable in terms of what you'd think would need to be done to accomplish the task.

Example: In a subsequent run I saw it go to about 360 MB. For simplicity assume that the size of the software is a negligible part of that. So 360 MB for about 9000 records, even if it were all in memory at the same time, means it's using 40,000 bytes per record. Doesn't sound reasonable, which is why something seems amiss in the implementation.

[UncleJohnsBand]

It handles all the parsing and readin in RAM.....that is why you see the memory increase........