IPTV User-Agent issue: 404 and NullReferenceException

[mellman]

Hi,

I'm having an IPTV problem which I think I have narrowed down to how NextPVR handles the User-Agent header for XC/M3U requests and live M3U8 streams.

This IPTV account still works correctly in TiviMate using Xtream Codes, but recently stopped working properly in NextPVR.

The original symptom in the NextPVR web player was:

Streaming Failed (transcoder exited)

After troubleshooting it, I think I have found a reproducible User-Agent issue.

SETUP

- NextPVR on Windows 10
- IPTV provider configured in NextPVR
- Same account/server works in TiviMate using "Xtream Codes parameters"
- Provider credentials are active and valid
- Account allows 2 connections
- Server/username/password omitted from this post

1. EXISTING NEXTPVR CHANNELS STOPPED WORKING

When trying to play a channel in the web player, NextPVR could not start the IPTV stream.

The eventual FFmpeg 404 was from NextPVR's own local /live URL, for example:

http://127.0.0.1:8866/live?channeloid=...

The important part of nrecord.log was earlier in the chain:

IPTV StartStream
Checking m3u8 stream (...)
Unable to StartStream(): One or more errors occurred. (A task was canceled.)
StartStream(...) allocated handle: 0
Handle: 0
Failed to start stream. Handle 0.

So the FFmpeg/transcoder error appears to be a consequence of the IPTV source failing to start, rather than an encoding problem.

2. "UPDATE CHANNELS" ALSO FAILED

Trying to update the existing IPTV source gave an empty "0 of 0 selected" channel list.

nrecord.log showed NextPVR attempting the provider M3U URL and receiving:

Downloading m3u https://<server>/get.php?....

Failed to download channel list:
System.Net.Http.HttpRequestException:
Response status code does not indicate success: 404 (Not Found).

at NShared.Recorders.IPTVScanner.WorkerThread()

However, the service is still active and working in TiviMate.

3. XC IMPORT ALSO RETURNS ZERO CHANNELS

I then tried adding a new IPTV device using "Import Using XC", entering the same server, username and password used by TiviMate.

NextPVR briefly showed "Logging in", then returned 0 channels.

The log showed the XC login failing with HTTP 404 in WebClient.DownloadString(), inside IPTVScanner.XCWorkerThread().

The same Xtream API endpoints work successfully in Chrome:

player_api.php?username=USERNAME&password=PASSWORD

player_api.php?username=USERNAME&password=PASSWORD&action=get_live_categories

player_api.php?username=USERNAME&password=PASSWORD&action=get_live_streams

All return valid data.

4. I CAN REPRODUCE THE XC 404 USING SYSTEM.NET.WEBCLIENT

This was the key test.

In Windows PowerShell I used the same System.Net.WebClient class:

$pw = Read-Host "Enter IPTV password"
$url = "https://<server>/player_api.php?username=USERNAME&password=$([uri]::EscapeDataString($pw))"

$wc = New-Object System.Net.WebClient
$wc.DownloadString($url)

Result:

Exception calling "DownloadString" with "1" argument(s):
"The remote server returned an error: (404) Not Found."

I then repeated the exact same request but added a User-Agent:

$wc = New-Object System.Net.WebClient
$wc.Headers["User-Agent"] = "Mozilla/5.0"
$wc.DownloadString($url)

This immediately succeeds and returns valid Xtream JSON containing user_info and server_info.

So:

System.Net.WebClient without User-Agent = HTTP 404
Same request with User-Agent: Mozilla/5.0 = success

This seems to match what happens in NextPVR's XCWorkerThread().

5. SAME THING HAPPENS WITH THE M3U DOWNLOAD

I downloaded the current provider M3U myself using:

$wc = New-Object System.Net.WebClient
$wc.Headers["User-Agent"] = "Mozilla/5.0"
$wc.DownloadFile($url, "C:\Users\Public\NPVR-data\current.m3u")

This succeeds.

The file is valid, starts with #EXTM3U, and contains the current channel URLs.

I imported this local M3U into a new NextPVR IPTV device successfully.

So the playlist itself is fine once it has been downloaded with a User-Agent.

6. THE ACTUAL LIVE STREAM ALSO REQUIRES A USER-AGENT

I then extracted one current M3U8 stream URL from the new playlist and tested that independently with System.Net.WebClient.

Without a User-Agent:

NO USER-AGENT: FAILED
The remote server returned an error: (404) Not Found.

With:

$wc.Headers["User-Agent"] = "Mozilla/5.0"

the exact same stream URL succeeds.

So the User-Agent requirement is not limited to the XC API or get.php download. It also applies to the actual live M3U8 stream.

7. LOCAL M3U WITH #EXTVLCOPT USER-AGENT

I created a one-channel M3U to test this in isolation:

#EXTM3U
#EXTINF:-1 tvg-name="LBW: HBO UA TEST",LBW: HBO UA TEST
#EXTVLCOPT:http-user-agent=Mozilla/5.0
http://<stream-server>/live/USERNAME/PASSWORD/STREAMID.m3u8

NextPVR imported this successfully.

When I tried to play it, NextPVR got as far as attempting the IPTV stream:

StartStream(...:LBW: HBO UA TEST)
IPTV StartStream
IPTV StartStream@2
IPTV StartStream@3
IPTV StartStream@3.1
IPTV StartStream@3.2
IPTV StartStream@3.3
IPTV StartStream@3.4
M3U8 (...)
Checking m3u8 stream (...)

It remained at "Starting Device. Please Wait..." for several seconds and then:

Unable to StartStream(): One or more errors occurred. (A task was canceled.)
StartStream(...:LBW: HBO UA TEST, ...) allocated handle: 0
Handle: 0
Failed to start stream. Handle 0.

The web player then showed "Streaming Failed (transcoder exited)".

8. TEST WITH |user-agent=Mozilla/5.0

I then created another one-channel test using the pipe/header syntax:

http://<stream-server>/live/USERNAME/PASSWORD/STREAMID.m3u8|user-agent=Mozilla/5.0

This produced a completely different failure.

The web player showed:

Error starting transcode: error: Internal Server Error

The new nrecord.log shows that this time NextPVR does not even get as far as IPTV StartStream.

Immediately after channel.transcode.initiate for the test channel:

InitiateTranscode
InitiateTranscode@2..
CaptureSource.LoadAll()

System.NullReferenceException: Object reference not set to an instance of an object.
  at NShared.Extend.RequestHandlers.TranscoderHLS.InitiateTranscode(String& reason)
  at NShared.Extend.ChannelRequestHandler.InitiateTranscode(Dictionary`2 parameters)
  at NShared.Extend.ChannelRequestHandler.HandleRequest(String method, Dictionary`2 parameters)
  at NShared.Extend.PluginFactory.HandleRequest(ServiceResponseStream serviceResponseStream, Dictionary`2 parameters)
  at NPVR.Controllers.ServicesController.HandleServiceRequest(String body, String contentType)

So the |user-agent syntax appears to trigger a NullReferenceException inside TranscoderHLS.InitiateTranscode() in my setup.

SUMMARY

As far as I can tell:

1. The IPTV provider now requires an acceptable User-Agent.

2. Requests without one appear to deliberately receive HTTP 404.

3. I can reproduce this independently using System.Net.WebClient:
  - no User-Agent = 404
  - Mozilla/5.0 = success

4. This applies to:
  - player_api.php / XC login
  - get.php / M3U download
  - the actual live M3U8 stream

5. This explains why TiviMate and Chrome work while NextPVR's XC/M3U requests fail.

6. Downloading the M3U manually with a User-Agent allows NextPVR to import the current channel list.

7. Adding #EXTVLCOPT:http-user-agent=Mozilla/5.0 does not allow the live stream to start.

8. Using:
  |user-agent=Mozilla/5.0

  causes a NullReferenceException inside:
  NShared.Extend.RequestHandlers.TranscoderHLS.InitiateTranscode()

Would it be possible for NextPVR to send a normal User-Agent for:

- XC API requests
- M3U downloads
- M3U8 stream requests

or alternatively provide a configurable User-Agent for these requests?

Also, is the NullReferenceException when using the |user-agent syntax something that can be fixed?

I'm happy to provide the full nrecord.log (attached) and any other logs required, or test a patched build/DLL if useful.

Thanks.

[mvallevand]

The user-agent can be added to the m3u and xc login request when you append |user=agent= ...

A manual m3u file with appended |user-agent=... does work to override the user agent here including 302 redirect typical with these type of services. However for m3ui it is case sensitive as "|User-Agent"=

IPTV stream playback should default to user agent "VLC/2.2.6 LibVLC/2.2.6" or "VLC/3.3.0 LibVLC/3.3.0" for m3u8 streans

The zipped logs almost always help.

Martin

[mellman]

Thanks Martin.

Your suggestion to append the User-Agent to the XC server address worked.

I changed the XC server entry to:

https://<server>|user-agent=Mozilla/5.0

and NextPVR was then able to retrieve the full XC channel list successfully.

I rebuilt the IPTV device from scratch using that method, imported the full channel lineup and updated the EPG. The EPG update completed successfully with over 317,000 listings inserted.

The IPTV stream itself is also now connecting successfully, so the original User-Agent/404 problem appears to be solved.

However, I now seem to have a separate database locking issue which is killing browser live playback.

I restarted the NextPVR Recording Service and reproduced this from a clean start.

For example, when starting LBW: HBO the log shows the IPTV stream starting successfully:

StartStream(57331:LBW: HBO)

content-type: video/mp2t

HTTP input source starting

StartStream [LIVE&...\live-LBW HBO-9708-2.ts] allocated handle: 2

StartStream(57331:LBW: HBO, ...) allocated handle: 1966082

Renew lease ok (0x1E0002)

FFmpeg also successfully identifies the incoming stream as H.264 video and AAC audio, so the IPTV connection itself appears to be working correctly.

At approximately the same time, the web player requests:

channel.show.artwork

event_id: 105943083

StreamArtwork()

About 30 seconds later I get:

Unexpected error updating air date:
Microsoft.Data.Sqlite.SqliteException (0x80004005):
SQLite Error 5: 'database is locked'.

at NShared.EPGManager.UpdateListing(EPGEvent e)

Then shortly afterwards:

Unexpected error in AddTempFile():
Microsoft.Data.Sqlite.SqliteException (0x80004005):
SQLite Error 5: 'database is locked'.

at NShared.TempFileManager.UpdateTempFile(Int32 handle)

Immediately followed by:

Live stream expired without 15 second renewal. (handle:1E0002)

StopStream...

Renew lease failed (0x1E0002)

At that point playback freezes/stops.

I have now reproduced essentially the same sequence twice after restarting the Recording Service.

On the first attempt:

- stream started successfully
- Renew lease ok
- channel.show.artwork started
- about 30 seconds later AddTempFile() got "database is locked"
- live stream expired because the lease could not renew
- artwork processing subsequently completed after about 33 seconds

I then tried the same channel again. It did eventually start playing and ran for roughly 20 seconds, but then froze. The new logs show the same database-lock/lease-expiry sequence again.

On that second attempt channel.show.artwork was requested at 12:33:07, the stream itself was fully started at 12:33:14, then:

12:33:37
Unexpected error updating air date:
SQLite Error 5: 'database is locked'.

12:33:44
Unexpected error in AddTempFile():
SQLite Error 5: 'database is locked'.

12:33:44
Live stream expired without 15 second renewal. (handle:1E0002)

The stream is then stopped.

There are some H.264 decode errors logged by FFmpeg at shutdown, but these occur after NextPVR has already stopped the live stream following the failed lease renewal, so I don't think they are the original cause.

There seems to be a strong correlation with the channel.show.artwork / EPG metadata operation holding or contending for the SQLite database long enough that TempFileManager cannot renew the live stream lease.

I don't know whether the artwork/EPG operation is definitely the process holding the lock, but the timing is very consistent.

Is this a known issue, or is there anything I can change/test to prevent the artwork/EPG database operation from blocking the live-stream lease renewal?

I have a fresh zipped log set from immediately after the latest reproduction if useful.

Thanks.

[Harry Jack]

Guide me

[mvallevand]

I rebuilt the IPTV device from scratch using that method, imported the full channel lineup and updated the EPG. The EPG update completed successfully with over 317,000 listings inserted.

I can't tell from the logs, but I expect the full channel list is huge and should be pruned from the start.  Did you try the test after restarting NextPVR to flush all the writes to the database.

Another possibility is you browser isn't requesting video because you haven't unblocked it.   A quick test of that might be to go the Settings->Access screen and tick Unauthenticated Access.  Then from the command line

Code:

"c:\Program Files\NextPVR\other\ffplay.exe" http://127.0.0.1:8866/live?channeloid=57331

Martin

[mellman]

Thanks Martin.

Yes, the previous browser test was done after restarting the NextPVR Recording Service. I had allowed the import/EPG work to finish, restarted NextPVR, waited for it to come back up, and then tested the stream. The browser playback still eventually failed on that setup.

I also tried your ffplay test.

I enabled Unauthenticated Access temporarily and ran:

"C:\Program Files\NextPVR\Other\ffplay.exe" "http://127.0.0.1:8866/live?channeloid=57331"

That worked successfully. The channel started playing in ffplay and I left it running for over five minutes without it freezing or stopping.

That seems significant because browser playback of the same source had been stopping after a much shorter period, whereas the direct /live stream through ffplay remained stable.

I have also taken your point about the size of the IPTV lineup.

The provider offers 13,819 streams in total and I had originally imported all of them. I have now gone back through the available groups and removed everything I would realistically never use. This NextPVR installation is primarily used for recording rather than browsing/watching IPTV, so there was no real reason for me to retain thousands of event, regional, 24/7, replay, international and duplicate streams.

I eventually reduced the wanted lineup to approximately 581 channels.

There was an additional complication in that deleting/recreating the IPTV device did not initially remove the old channels from the channel database, so I was still seeing thousands of old channels in Channels/Guide even though they were not part of the new selection. I have now cleaned those out as well and the Guide is populated only with the channels I actually want.

The difference in EPG size is substantial:

Original full import:
13,819 channels available
317,337 EPG listings inserted

Pruned setup:
approximately 581 wanted channels
24,184 EPG listings inserted

So the EPG/database workload is now dramatically smaller.

I have also disabled Unauthenticated Access again after completing the ffplay test.

My main use of NextPVR is scheduled recording rather than watching live TV in the browser. Am I right in thinking that the browser/transcoder/temporary live-stream path we have been troubleshooting is separate from the normal recording path?

I will now test browser playback again with the cleaned-up database as well, but I thought the successful 5+ minute ffplay test and the much smaller channel/EPG setup were worth reporting first.

Thanks again for the help.

[mvallevand]

You don't remove channels with a rescan, channels need to be manually deleted on Settings->Channels, I expect that was you confusion.

Since Live TV is recording that paths are quite similar. The logs seems to indicated that recording had started but I was thinking maybe you had just forgotten to unblock the browser, nothing browsers don't play video by default.

Martin