EPG Update

[jcole998]

I was looking through the NRecord.log file and noticed that it's taking about 20 of the 26-minute update time just to delete old listings. Is this normal? Is there no faster way to delete rows from the database? My update consists of a 14-day look ahead period. I'm just concerned that for that length of time, the update is exposed to any manner of system glitch that could corrupt the database. (and I'm curious)

Thanks for the help.

[sub]

It usually wouldn't take that long. This section of the logs is actually saving the listings to the database too though, not just deleting.

You might have to look how the CPU, memory and I/O usage looks during an EPG update, so you can tell which is proving a bottleneck.

[sub]

To try emulating your setup, I added a schedules direct lineup with 668 channels (yours had 563). An EPG update here is taking about 4 minutes from beginning to end.

[jcole998]

Thank you, sub. I will dig further and post any insights. Maybe there is some aspect of the database I can tweak.

[sub]

I'd start by looking at the disk/cpu/memory usage during an EPG update. It's likely one of these is the bottleneck.

You could always delete the channels you know you never watch. I'm guessing at lot of those 563 channels are never used.

Otherwise you could reduce the number of days of listings.

[jcole998]

OK...

Launched the Resource Monitor from the Task Manager during the EPG Update. During the "slow" part of the update with NextPVR showing "Saving EPG Listings" CPU time is less than 1% but there is tremendous disk activity on a file named c:\users\npvr\npvr.db3-journal. Watching the folder within which this file resides, the file is continually popping in and out of view like it's being deleted and created hundreds of times.

Could this be the cause of the the lengthy update? Can I control the manner with which the database manipulates this file?

[mvallevand]

I've reported that deleting channel by channel has always been a bottleneck fo me and it looks like the same for you too

2016-02-09 04:19:21.245 [DEBUG][10] Deleting old listings from Channel OID: 7472 (Smithsonian Cha)
2016-02-09 04:19:23.179 [DEBUG][10] Deleting old listings from Channel OID: 7473 (TLC HD)
2016-02-09 04:19:25.427 [DEBUG][10] Deleting old listings from Channel OID: 7474 (Lifetime HD)
2016-02-09 04:19:27.645 [DEBUG][10] Deleting old listings from Channel OID: 7475 (LMN HD)

2 seconds a channel adds up when you have so many channels. I tried emptying the epg table first and it doesn't help.

Martin

[Graham]

... there is tremendous disk activity on a file named c:\users\npvr\npvr.db3-journal.

The journal file is an integral part of the SQLite database process used by NextPVR and is outside of NextPVR control. I doubt that there is much that can be done with regard to the journal ... https://www.sqlite.org/tempfiles.html

[sub]

I've reported that deleting channel by channel has always been a bottleneck fo me and it looks like the same for you too

2016-02-09 04:19:21.245 [DEBUG][10] Deleting old listings from Channel OID: 7472 (Smithsonian Cha)
2016-02-09 04:19:23.179 [DEBUG][10] Deleting old listings from Channel OID: 7473 (TLC HD)
2016-02-09 04:19:25.427 [DEBUG][10] Deleting old listings from Channel OID: 7474 (Lifetime HD)
2016-02-09 04:19:27.645 [DEBUG][10] Deleting old listings from Channel OID: 7475 (LMN HD)

2 seconds a channel adds up when you have so many channels. I tried emptying the epg table first and it doesn't help.

Martin

It's not the deleting that's taking the time there. That happens very quickly. It's the saving the listings which takes the time. It's happening between each of those log messages, but you don't see it in your logs.

A snippet from my logs, which show more info, should show you what I mean:

2016-02-12 06:14:33.087 [DEBUG][10] Deleting old listings from Channel OID: 7184 (WCBS)
2016-02-12 06:14:33.087 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.128 [DEBUG][10] - saved 255 epg events. updated 0. skipped 0
2016-02-12 06:14:33.128 [DEBUG][10] Deleting old listings from Channel OID: 7185 (WPXN)
2016-02-12 06:14:33.128 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.176 [DEBUG][10] - saved 227 epg events. updated 0. skipped 0
2016-02-12 06:14:33.176 [DEBUG][10] Deleting old listings from Channel OID: 7186 (WNBC)
2016-02-12 06:14:33.176 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.216 [DEBUG][10] - saved 253 epg events. updated 0. skipped 0
2016-02-12 06:14:33.216 [DEBUG][10] Deleting old listings from Channel OID: 7187 (WNYW)
2016-02-12 06:14:33.216 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.263 [DEBUG][10] - saved 289 epg events. updated 0. skipped 0
2016-02-12 06:14:33.263 [DEBUG][10] Deleting old listings from Channel OID: 7188 (WXTV)
2016-02-12 06:14:33.263 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.300 [DEBUG][10] - saved 208 epg events. updated 0. skipped 0
2016-02-12 06:14:33.300 [DEBUG][10] Deleting old listings from Channel OID: 7189 (WABC)
2016-02-12 06:14:33.300 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.352 [DEBUG][10] - saved 248 epg events. updated 0. skipped 0
2016-02-12 06:14:33.352 [DEBUG][10] Deleting old listings from Channel OID: 7190 (HSN)
2016-02-12 06:14:33.353 [DEBUG][10] - about to save epg events
2016-02-12 06:14:33.405 [DEBUG][10] - saved 221 epg events. updated 0. skipped 0

[Graham]

I doubt that there is much that can be done with regard to the journal

But, maybe ... just maybe ... you could try this ... https://sqlite.org/lang_vacuum.html ... immediately before an update.