Complete EPG Listings Missing for One Channel for UK OTA

[latenighter]

Hi Martin,

Many thanks for the offer, though if the data isn't in the OTA feed I'm not sure how the HDHR tuners would get the guide data. I've actually been looking at the XMLTV output from EPG Collector to see what is present OTA - I guess I could create complete NPVR guide data for Channel 5 by extracting the Channel 5 HD data from the EPG Collector XMLTV output, changing the ts_id in this data and then load it to NPVR as Channel 5 listings.

As I partly expected, this morning's scheduled EPG update did not add any more events to Channel 5 in the NPVR guide. My EPG Collector analysis shows the latest Channel 5 event in the mux is still the "Shoplifters..." item that finishes at 00:25 UK time on 09-NOV-2022 and that there are 265 Channel 5 events in the mux data but the NPVR logs indicate it updated 368 listings for Channel 5. So it still leaves the logging issue out there that I raised as point 2) in my "Breaking news..." post (#3).

I'm also starting to think that if dvbtee and EPG Collector use the same DVB extraction/processing libraries, as may NPVR, and that the data is in the mux they could all be failing to process the later data at the same point. (And maybe the HDHR uses a different DVB library which means it may successfully get the full listings so I'll give this a try.) This brings us back to the possible invalid character scenario. Although you didn't like the idea, I do have the skills to use a hex editor to look at the TScaptures provided I can get a raw format/layout definition to refer to.

Cheers,

[mvallevand]

My utility downloads the data from SiliconDust not from OTA and it uses Gracenote online data as a source like Schedules Direct.

You could use EPG collector to create the XMLTV file and map the channels that way, but if it doesn't have the data why bother?

For you hex editor feel free to read https://www.etsi.org/deliver/etsi_en/300...11101p.pdf and look at descriptor tables 0x4d and 0x4e. NextPVR and dvbtree will use custom parsing based on that spec, which I assume EPG Collector will use too.

Without a full mux I can't comment further, but I don't feel there is a bug worth investigating here. Don't go by the counts.

Martin

[latenighter]

Hi Martin,

Many thanks for the information, expecially the DVB reference.  I think I'm making progress and, if I'm right, I'll be raising a Feature Request to enhance the OTA EPG Update logic in NPVR.  I'll continue to check things out and validate my reasoning over the next few days before I raise a request.

Some context:
This morning's NPVR scheduled EPG Update added the missing Channel 5 EPG entries from early on 09-NOV-2022 to EOD 12-NOV-2022 but it removed the Channel 5 EPG entries for the whole of 08-NOV-2022 (I hadn't noticed this "big gap in the middle" behaviour before but haven't been looking at the EPG updates in such detail before either).  I used EPG Collector, looking at the Channel 5 mux only, to confirm this gap and then defined a new EPG Collector parameter set to collect and accumulate EPG data across all available muxes which EPG Collector is able to do.  This EPG Collector run extracted a complete set of Channel 5 EPG entries from today until EOD 12-NOV-2022 with no gaps.  I then ran another EPG Update in NPVR and the 08-NOV-2022 gap in the Channel 5 listings was still present.

I tried this approach because using a hex editor and the information from the DVB reference you kindly provided I could see that there were EPG listings in the Channel 5 mux for Channel 5 HD and Channel 5 +1 even though those channels are actually broadcast in different muxes (in reality each of these 3 channels are in different muxes).  This was consistent with the DVB reference which indicated that EPG listings in a mux can be present for channels not broadcast in the mux (top of page 26 in section 5.2.4 Event Information Table, transport_stream_id: This is a 16-bit field which serves as a label for identification of the TS, about which the EIT informs, from any other multiplex within the delivery system).

In summary, for UK Freeview at least, it looks like the mux in which a channel is broadcast may not contain that channel's full EPG listing (from the logs, NPVR logic seems to rely on the channel's mux containing the full data) but it looks like all available muxes together will contain the full EPG listing for the channel.  Why this should be the case is open to conjecture.  It could be that some muxes have limited bandwidth for EPG data to make sure that the actual video streams are not impacted.  This, in turn, could be dictated by the total bandwidth of a mux.  I see the bandwidth for the UK Freeview muxes in my region range from 24.1Mb/s to 40.2Mb/s.

(FYI, it looks like EPG Collector has functionality to dump a full mux capture and that it can do it for longer than the evaluation limit of TScapture - I successfully tried 5 minutes at one time to parallel the NPVR EPG limit.  This could be an alternative for Windows installations when more than 60 seconds of capture is required.)

Cheers,

[mvallevand]

For me it would be worth a cup of copy a month on Schedules Direct just to avoid the problem. The daily updates are fast and data is better too.

I have less than one month experience with DVB-T and there are no signals here in Canada. I do know that during scans even when one frequency is select the Linux drivers scan other frequencies, not sure about the Feel free to provide fullmux on a different frequency.

Martin

[latenighter]

Hi Martin,

Many thanks for your reply.

The OTA EPG is generally pretty good on UK Freeview and compares very well to the SD updates received by my Emby instance. I understand the OTA EPG is nowhere near as good in some regions like North America. As my analysis has shown/is showing the UK Freeview OTA data for any channel is complete when viewed across the muxes.

I've performed two separate full mux captures using EPG Collector, each for 5 minutes. One for the mux used to broadcast Channel 5 (frequency 522MHz, bandwidth 24.1Mb/s) and one for the mux used to broadcast Channel 5 HD (frequency 474MHz, bandwidth 40.2Mb/s). I had to guess which of the other muxes had the missing data or was more likely to have a full EPG listing but I got it right first time. The links are:

522MHZ - https://drive.google.com/file/d/1fjfxfic...sp=sharing
474MHz - https://drive.google.com/file/d/1tUR4hqc...sp=sharing

My own analysis on these captures show the 522MHz mux does not contain the Channel 5 (ts_id 8500) EPG data for 08-NOV-2022 but the 474MHz mux contains the full listing for Channel 5 (in fact from 04-NOV-2022 to EOD 12-NOV-2022 inclusive). Sub may need to confirm but at present I believe the NPVR EPG Update process only looks at the mux a channel is broadcast on for its EPG listings. A recent change allowed an alternative tuner to be used dynamically if the primary tuner was not available for the EPG Update process (eg. it is recording a different mux than the one it wants to inspect for EPG listings) but it was still only looking at the mux for the channel to get that channel's EPG.

Cheers,

[mvallevand]

The data for Channel 5 HD is not the same as for Channel 5 they have different SIDs I did see a few sid 8500 shows on 522 but not the missing block and nothing that indicates NextPVR would benefit from scanning this frequency

NextPVR for Linux can't decode more than 1 minute of mux so I cannot comment on 5 minute scan.

Martin

[sub]

Sub may need to confirm but at present I believe the NPVR EPG Update process only looks at the mux a channel is broadcast on for its EPG listings. 

That is incorrect.

Here in NZ, we get our 'Freeview' DVB-T service from 4 frequencies. I have only a single one of the frequencies enabled in the DVB EPG settings, and it gets listings for the channels on all frequencies with the single minute spent on one frequency.

[latenighter]

Hi Martin,

I think I may not have explained myself correctly. Your comment reads as if you got the opposite of what I intended to say. The 522MHz mux, which is used to broadcast Channel 5, does not contain the full Channel 5 EPG (though it does seem to contain the full Channel 5 HD EPG). The 474MHz mux contains full EPG for both Channel 5 and Channel 5 HD and it is used to broadcast Channel 5 HD. So, ideally, I'd see a complete solution with the OTA EPG data as it is currently looking as including scanning the 474MHz mux as part of building the Channel 5 EPG, etc.. It seems this is possible with existing settings - please see my reply to Sub immediately below.

Hi Sub,

My apologies, it looks like I've missed a trick with the NPVR settings, how they interact with my tuner/mux set-up (see below for more detail) and the standards for OTA EPG listings. Previously my logs have greatly implied that the system was only getting the EPG listings for a channel from the mux it was broadcast on. Prompted by your comment I explored the settings related to EPG Updates and discovered a section that enabled GUI/browser control of the frequencies scanned and the length of scan for each frequency - previously I had gone into config.xml to change the length of scan as I didn't realise the GUI function had been added. After some trial-and error I seem to have achieved a new EPG OTA set-up in NPVR that gives me complete EPG listings despite individual muxes having incomplete EPG listings, even for the channels that have video streams in the mux. With my latest set-up my interpretation of the logging of the EPG scan process still seems to imply that only the first mux containg the channel is scanned for the channel's EPG listings but the data in the NPVR guide seems to be built up from an accumulation of EPG listings across the muxes.

(You may not have spotted in the earlier posts on this thread that I had employed a 1-to-1 tuner/mux mapping to facilitate use of NPVR as a tuner for Emby Live TV via m3u tuner definitions without impacting NPVR's ability to record whenever/whatever it wishes. I don't use Emby's recording capabilities because of NPVR's much superior recording definition flexibility and only use Emby's library and Live TV functionality. For the new OTA EPG set up to work I had to allow the tuner I had previously dedicated to the 474MHz mux to scan through all muxes so it had a complete channel list from across all muxes. I then moved that tuner to the bottom of the tuner priority list so that it would never get picked for recordings/Live TV except for the 474MHz mux unique to it.)

Cheers,

[mvallevand]

I think I understood what you were saying, and maybe 474 has more data that Linux can't read in one minute.

Martin

[latenighter]

I think I understood what you were saying, and maybe 474 has more data that Linux can't read in one minute.

Martin

Hi Martin,

It may not be perfect for your needs but I see EPG Collector can be run under /mono or /wine in Linux against a .ts file.  (It can't use BDA tuners directly but I think that is a general limitation in Linux.)  This may give you extra options for analysing the EPG data in .ts files longer than 1 minute but I expect you probably won't be able to use it to capture .ts files under Linux.

Having satisfied myself that the missing EPG data in the 522 mux was in the 474 mux with the hex editor approach, I felt I could rely on the output from EPG Collector for analysis.  My general approach became one of creating a .ts file (for repeatable results across different times of capture) and then feeding this into EPG Collector.  I then opened the XMLTV output from EPG Collector in EXCEL and let EXCEL translate the raw XML.  I could then filter/etc. the output to my heart's content to check what was in it.

Hi Martin and Sub,

From my side I think we've resolved this challenge and I must thank you for the information and pointers you've provided.  And I've got to learn a bit more about NPVR and some extra tools like EPG Collector I now have at my disposal to help me self-resolve any future issues I may encounter.  Of course, these issues could be of my own making....

In case anyone else looks at this thread I feel I do need to give the detail about the latest tuner/mux mapping I use so they don't take themselves down a blind alley if they wanted to explore a similar set-up.  There are currently 6 muxes in the UK DVB service and I have 6 available tuners (1x 2-tuner PCI card that is DVB-T only and 1x 4-tuner HDHR box that is both DVB-T and DVB-T2 capable on all tuners).  I use the BDA method to access the HDHR tuners from NPVR so that NPVR's ability to re-use an active mux kicks in.  At present only 2 of the muxes transmit DVB-T2 though I believe the distant plans are that the UK DVB service will consolidate down to 4 muxes, all using DVB-T2 so that there is only a small loss of total bandwidth across the muxes (hence my reason to acquire the HDHR box to help future-proof my set-up).  For 5 of the tuners I have restricted them to a single unique mux each.  For the final tuner, in priority order, I have allowed it to access all channels across all muxes - this makes the 6th (DVB-T2) mux unique to it.  Additionally I set the <ReversePriorityForLiveTV> setting in config.xml to false so that any NPVR recording requests, and Live TV requests from within NPVR or from external clients that access NPVR such as Emby, will go to the appropriate tuner with only one mux defined or the 6th tuner which is unique to the final mux.  Previously all tuners only had one mux defined but I had to change the tuner/mux set up so that the NPVR OTA EPG Update wasn't restricted to getting EPG listings for a channel from the mux it was actually broadcast on - some muxes did not have complete EPG listings.  Changing things so that one tuner had access to all muxes, and all channels were defined to that tuner, resolved this.

Why do I do this?  It's all based on historical experience and that I could do so once I bought the HDHR box.  The first relevant historical experience, in an all muxes on all tuners set-up, was that non-overlapping Scheduled Recording entries were always set up with the highest priority capture source (notwithstanding that NPVR has logic to swap to an alternative tuner if it finds the planned tuner is busy on a different mux when recording time comes).  Consider the situation where all muxes are defined to all tuners.  NPVR is recording channel 1 from mux Z using tuner A (which is defined in the Scheduled Recording entry).  A Live TV session is then started against channel 2 from mux Y using tuner B.  The recording of channel 1 then stops, freeing up tuner A.  A short time later, while the Live TV session is still going on, NPVR starts recording channel 3 from mux Y.  Further experience shows this will be using tuner A because this is what is defined in the Scheduled Recording entry and tuner A is free.  So we now have two tuners active for mux Y.  Continuing this accumulation method with further recording and Live TV requests it is possible to end up with only 3 muxes active across all 6 tuners with no room for further simultaneous recording, or live TV sessions, from a 4th mux.

My current approach also allows NPVR automated scheduling to cater more easily with contemporaneous overlapping recordings on multiple channels on different muxes as it now has very little choice about which tuner to allocate.  Consider the situation where a number of closely sequential programs on channel 1 from mux Z are eligible to be recorded because of Recurring Recording entries.  Overlapping in time there is another set of closely sequential programs on channel 2 from mux Y also eligible to be recorded. When I had just a 2-tuner set up, I frequently came across situations when some of these programs weren't automatically scheduled because the automatic scheduling for the programs on channel 1 set up some of the Scheduled Recording entries for tuner A and some for tuner B when it would be possible to allocate just tuner A (not sure of the reason why - some possibilites that came to my mind included pre-/post-padding and relative time separation of the programs on the different channels versus NPVR processing order).  This left no sufficient gap on either tuner A or tuner B to fit in one or more of the programs on channel 2.  I was able to manually intervene by deleting all the Scheduled Recording entries for the affected programs in the time range and then manually add Scheduled Recordings in such a way as to guarantee the tuner allocated.  If I got the manual intervention right then I was able to achieve all the required programs on channel 1 being allocated to tuner A and all the required programs on channel 2 being allocated to tuner B.  In this way all required Scheduled Recordings were successfully set up manually that automated scheduling hadn't fully succeeded with.  I'm anticipating allowing the 6th tuner access to all muxes may re-introduce a possibility of it being used in a Scheduled Recording entry for a channel not included in its unique mux.  However the unique mux for the 6th tuner has been chosen as one that will only ever have (infrequent) manual recording requests set up for it so I'll soon know if this has happened and take action appropriately.  (As this is early days of this new arrangement I'm getting an idea of ithe likelihood of this situation by running some SQLite to see if any channels not in the mux unique to the 6th tuner have Scheduled Recording entries set up against it.  So far this hasn't occurred.)

As I've said, historical experience has framed how my current set-up looks.  It could well be that some of the features I'd come across over the last few years no longer exist and some or all of this non-standard approach is no longer necessary.  However, as this is where I find myself currently and I haven't come across any impacts from this approach other than the recent OTA EPG challenge, I'll leave things as they stand for the time being.

Cheers,