Complete EPG Listings Missing for One Channel for UK OTA

[latenighter]

(As advised by mvallevand creating a new thread as this issue was originally thought related to https://forums.nextpvr.com/showthread.ph...117&page=2 but may not be.)

Symptoms are that the EPG listings for Channel 5 in the UK are completely missing from the NPVR TV Guide.  Whenever an EPG update is run, scheduled or ad-hoc, only "now and next" EPG entries are created.

Only the one channel seems to be affected for OTA EPG updates.  Counterpart channels Channel 5 HD and Channel 5 +1 have complete listings for the full 7 day period the EPG update is for as is the case for all other channels on all muxes.

The attached logs were obtained with the <ExtendedLogging> option set to true and were taken ~20 seconds after the relevant frequency scan in the EPG update had completed.  With my rudimentary knowledge the log seems to indicate that the system thinks it stored 340 events for "Channel 5" but only a "now and next" seemed to be stored.  This is shown in the attached screenshots (system restarted at 19:07 on 31-OCT-2022 with the EPG Update requested immediately afterwards) the "A" .jpg shows the "now and next" for Channel 5 and the "B" screenshot for the following day shows no listings for Channel 5.  Other devices at home (eg. TV) show a full schedule for Channel 5 so the data would seem to be in the mux stream which the "340 events" entry in the log would support.

[latenighter]

mvallevand Wrote:
Sub typically needs you to create a 1 minute full mux capture from the frequency using his tscapture program https://tscapture.com/ to see what might be going on with EPG udpates. The captured file must be load to external sites like Google Drive or One Drive.

Martin

------------------------------------------------------------------------------------------------------------

A full mux capture has been taken and loaded to https://drive.google.com/file/d/1JegUkYE...sp=sharing. I assume it doesn't need to be contemporaneous with an EPG update. On the surface, from the logs, the problem doesn't seem to be one of extracting the EPG data for the channel from the mux but adding it to the database. If it is thought a copy of my database is needed I can add that to the thread.

Before posting here I did try some other actions such as re-scanning the channels within the system and then running the EPG update. I've re-booted the system too and looked in the database (eg. CHANNEL and EPG_EVENT tables) for non-printing characters that may have impacted the process. I also discovered that using <ExtendedLogging> and taking the logs at the end of the EPG update saw the logging of the early part of the EPG Update, where the relevant mux was processed, get dropped from the log files because of the verbosity of the Scheduling process at the end. However <ExtendedLogging> didn't seem to affect the verbosity of the mux extraction.

Cheers,

[latenighter]

(Originally posted the day after the above post.)

Breaking news, the Channel 5 listings have been captured by this morning's scheduled EPG update - well 6 of the 7 days requested anyway.  This is after more than a week of it not happening for this channel.  All other channels still have full listings (I have a read-only SQLite3 project that contains SQL I can run to check on gaps in the EPG).  To my mind that gives the following options:

1) There was some sort of temporary Channel ID mismatch between what was being transmitted and the NPVR database.
2) The listings were not in the mux.  This does not jive with the "...340 listings..." in the log but this may highlight a logging issue.  As it's a LG smart TV there is a possibility the TV used the internet to get listings it couldn't get OTA.
3) In combination with 1), if there are multiple channel identifiers in the stream maybe NPVR uses one to identify the listings and another to add to the database.  If this is the case then maybe the logic can be updated to be consistent and maybe try to use any/all of the identifiers at each stage.

To aid any investigation thought to be worthwhile I have taken a second full mux capture this morning, after the scheduled EPG update, that has been stored in https://drive.google.com/file/d/1bJYBFq7...sp=sharing.

Cheers,

[latenighter]

mvallevand Wrote:
The week long issue you saw is typical with EPG data issues. There can be a character NextPVR doesn't decode properly from the title, subtitle or description text and when the show or shows are no longer in the data, the EPG loads properly.

The only solution is to grab that full mux as soon as you see the problem is happening so sub can investigate.

Martin

-----------------------------------------------------------------------------------------------------

Hi Martin,

Many thanks for your rapid response. I'm familiar with the problem of non-printing characters of various persuasions affecting the EPG update and scheduling recordings but this one seemed different. As I said, this occurred for more for than a week so any original blocking event should have dropped from the EPG data in the mux some days before. And if it was an issue with a daily program then it may still not have allowed the updates to recommence even now.

I'll check tomorrow morning's EPG update and see if any events get added beyond the current point when they finish for the channel (first program that finishes after 00:01 UK time on 9th November). I'll report back and, as necessary, take a new full mux capture.

As you say, it probably needs an analysis of the before and after mux extracts to determine what the block really was.

Cheers,

[latenighter]

Hi Martin/Kotoka,

Please accept my apologies if I have incorrectly identified that this thread and my issue could be related. I did check out the forum extensively for issues similar to mine as I know duplicating items is as much as a problem for the forums as is linking items that are not related. I did consider the following points in this item as displaying similarities with what I am experiencing:

1) One or a small number of channels are not having EPG events visible in the NPVR guide;
2) Only "now and next" events actually seem to be present in the guide for the affected channels;
3) The problem persisted for much more than a week. I had discounted the non-printing character in EPG event title/description/etc. issues seen before as the persistence of missing EPG entries seemed to go beyond the 7-day period I use for my OTA EPG updates. Also the fact that the Channel 5 HD and Channel 5 +1 listings were always complete, when they would have the same events and it would be reasonable to assume the events have been formatted/etc. the same way, made it seem less likely it was a non-printing character issue.

If you think it appropriate to have a separate thread for my issue please feel free to move to a more appropriate thread or let me know to create a new thread.

That having been said this morning's scheduled EPG update did not add any new events to the guide for the affected channel. The last event shown for Channel 5 is still "Shoplifters & Scammers..." that ends at 00:25 UK time on 09-NOV-2022. This is hinting that it may be a non-printing character issue. However the Channel 5 HD and Channel 5 +1 listings are complete as before. In any event I have taken a new full mux capture this morning and it is saved at https://drive.google.com/file/d/1HodvBMA...sp=sharing to help investigations. If it's any help, according to the Channel 5 HD listings, the next event is supposed to be "Cause of Death..." running from 00:25 to 01:15 UK time on 09-NOV-2022.

I'm happy to try and look at this myself for everyone's benefit though can't guarantee the results. For me to do so, would anyone be able to provide a link to a reference layout/etc. used by NPVR for a DVB-T mux that I could use to look at the captured streams in a hex editor? Also if there is a way to get, or find, more detailed logging of what's happening when the "...saving nnn listings for channel..." debug entry appears in the current logs and around the EPG capture process itself just before and after that would help too.

Cheers,

[mvallevand]

You can't just look at it because it is binary not hex. I use a linux tool called dvbtee for my first triage and certainly ts_id 16515 is junk with dvbtee, but that might not be a concern. Testing how sub does it is pretty much up to him and I am not sure this is related. I also see it ending with "Shoplifters & Scammers: At War with the Law" in NextPVR and it dvbtee but I don't see any data past that, in the file you gave.

Therefore I don't consider it a NextPVR issue, if you can capture a longer file with Linux maybe it will be there? Right now NextPVR is showing what is available.

Ultimately if you want longer data you might want to consider Schedules Direct, like we use in North America. It is faster and quite reliable

Martin

[latenighter]

mvallevand Wrote:
The issues could be related but opening a new thread is "free" so it is always worth the effort. Most problems are not unique, so it would be silly trying to maintain a blog like thread for common problems.

You can't just look at it because it is binary not hex. I use a linux tool called dvbtee for my first triage and certainly ts_id 16515 is junk with dvbtee, but that might not be a concern. Testing how sub does it is pretty much up to him and I am not sure this is related. I also see it ending with "Shoplifters & Scammers: At War with the Law" in NextPVR and it dvbtee but I don't see any data past that, in the file you gave. If you can capture a longer file with Linux maybe it will be there? Right now I consider it a limit of the data in the file.

Ultimately if you want longer data you might want to consider Schedules Direct, like we use in North America. It is faster and quite reliable

Martin

---------------------------------------------------------------------------------------------------------------------------------

Hi Martin,

On your recommendation I have created this new thread. I'll post in the original thread that I have done this.

I believe the ts_id for Channel 5 is 8500 as shown in the NPVR Settings/Channels display so the issues with ts_id 16515 may be a red herring.

As I use Windows I don't think I can use dvbtee but prompted by your comment my searches have thrown up EPG Collector as maybe a way to view the EPG data in a DVB-T stream. I'll give it a go with an extended capture as you seem to think the EPG data in the mux capture for Channel 5 ends at the last current entry in the guide. As a point of information, for the last couple of years I have set my mux EPG scanning period to 150 seconds rather than the 60 seconds default because of small EPG gaps but had never encountered a complete channel missing before. So it may be possible that the 1 minute limit on the evaluation TScapture tool is insufficient to capture all the data. It may also point to the solution being to increase the scanning period further.

I actually have a joint NPVR/Emby set up, making use of the M3U tuner definition capabilities to facilitate use of NPVR for Live TV in Emby, but always use NPVR for recordings due to its superior recording flexibility. I can get around the "Emby requests could impact NPVR scheduling" challenge because I have defined my tuners and muxes in NPVR on a one-to-one basis. And, of course, Emby has a built-in Schedules Direct subscription so if it really came to it, I could schedule any Channel 5 recordings through Emby. But I'd like to get NPVR working fully again.

Cheers,

[mvallevand]

I deleted a couple of posts on the other thread. As I said I don't think the issue is NextPVR

The channel isn't missing, data just ends earlier then you'd like, at least in the one minute capture. We can also get a longer full mux from the HDHR if you want to use it, maybe 150 seconds isn't long enough.

Martin

[latenighter]

Hi Martin,

I've been able to perform an EPG Collector capture against the mux that used the EPG Collector extraction default of 300 seconds. This confirms your findings that at present the EPG entries for ts_id 8500 end with the "Shoplifters..." event that finishes early on 09-NOV-2022. I'd also performed a NPVR EPG update using 300 seconds per frequency and the NPVR guide didn't show any extra entries either. As my one-to-one mux/tuner set up would tolerate extended EPG update times I'll leave the per frequency setting at 300 seconds rather than the 150. I'm guessing EPG Collector has its default set to 300 seconds for a reason....

I'll keep an eye on things to see if any more entries get added for Channel 5 before we get to 09-NOV-2022. If nothing else gets added by then I'll be back to the situation where the only entries present for Channel 5 will be the "now and next" ones and I'll be in the same position as when I first noticed the issue that contributed to my thinking that it was likely to be the same as Kotoka's issue.

While this intermittent Channel 5 issue persists I may have to explore easier ways of working around it. Fortunately none of my Recurring Recordings are specific for Channel 5 and the HD and +1 channels still have complete listings that I can use to highlight creating manual recordings against Channel 5.

Cheers,

[mvallevand]

Good to know. I do have a utility that will download your HDHR guide data as XMLTV for 24 hours if it stays short and you need that channel.

Martin