Random skipping in recordings

[mvallevand]

The logs show it getting multiple resolutions 20% is too much to buffer perhaps you are using a VPN or something which limits your downloads

Your questions are really about ffmpeg, even streamlink uses ffmpeg

[ballfam]

I'm running a smart DNS, which should have removed itself from the equation by the time the stream starts. Just to see if it is related to the network, I'll switch back to a VPN to see what happens. There are some distinct advantages......if I use a VPN, the stream runs directly to a local VPN server, which then tunnels it to my end of the link; but the VPN server itself does implement buffer caching, so the packets would be cached right next to the source of the stream until the other end of the tunnel can take them, rather than trying to push them all the way from Australia to the U.S. I think it even has check-summing and re-send ability if packets are lost. BTW: smart DNS vs VPN is the one thing that has changed recently, since I have started seeing these dropouts.

Thanks for your help Martin, I have enough information to try a few things; I'll post if I manage to solve the issue.

[ballfam]

So I think I solved it, but you won't believe how.

First I checked the networking, and it seems to be all good, there should be no reason to drop packets. Then I ran an iostat while doing 2 recordings and I noticed that areq (average request time) quite frequently goes up past 500ms, and the write sizes are really small. Either ffmpeg is flushing really small blocks, or the kernel driver for this disk/fs is configured to make many small writes, it's just a standard ext4 drive, so nothing strange there. With this, and the fact that there are other things writing to that disk at the same time (backups, torrent downloads, etc...), the request time goes up too far, and it looks like ffmpeg decides to drop some of the frames to keep up. The disk has the ability to write 10 times this amount of data at least, if the correct blocking factor is used.

I know that the NFS server is configured to buffer large blocks to combat network issues, so I just knocked up a "fake" NFS mount. I created a docker image for NextPVR, then added NFS volumes to it, and configured them into the container; I set the NFS  mount parameters to use "async" to help buffer the writes. I also set /config in the container to point to /var/opt/nextpvr; now I can shut down the regular NextPVR and start up the container, and it is completely transparent, same recordings, same settings, etc... except that the recording disks are now NFS mounted. In reality, everything is on the same machine, but the disk is being accessed through the nfs server instead of directly, and it takes advantage of NFS buffering and larger writes. With this configuration, the average wait time goes right down to between 10ms  and 20ms, the average write size is up over a meg, and the KBytes written per second is 10 times what it was before.

It seems counter-intuitive, but running over NFS makes the disk work much more efficient for these streaming writes. I'm using a little more memory to allow the nfs server to buffer up the requests and write in larger packets, but memory is something I have plenty of.

I'm sure there may be other ways to fix this, such as reconfiguring the disk drivers, or using a cache file for ffmpeg with the "-cache" flag, but this seems to work well, and is easy to configure.

It would be nice if there were a NextPVR configuration that allowed me to specify the ffmpeg parameters if NextPVR decides to use ffmpeg to write the recordings; I could have probably used an ffmpeg cache file to fix this issue. I considered just replacing ffmpg with an executable shell script wrapper that just adds "-cache" to the list of passed ffpmeg parameters before calling the real ffmpeg, but the container works well enough for now.

[mvallevand]

Glad you figured it out. You can use an extra and use whatever ffmpeg options you want. Remember too you are using an obsolete ffmpeg and sub has to use pretty generic ffmpeg parameters to support from 4 to 7.

Martin

[ballfam]

Thanks Martin. Yeah, I forgot about using an extra, that would probably also solve the issue, allowing me to tune ffmpeg to do better writes.

I'll also update ffmpeg at some point; I just don't like fiddling too much with stuff that is working, it always ends up breaking something.

[ballfam]

In case anyone is interested, I also solved the issue using an extra that runs streamlink. Even though streamlink just uses ffmpeg, it seems to have a couple of advantages:

1. On older Linux versions where ffmpeg will not update with apt (default distro version too old), and where other PPAs install a version that does not have compatible libraries, the only option is to build ffmpeg from source if you want a recent version, but even that is difficult without all the correct dependencies. Streamlink can install with pip, and the latest version seems to work fine on older Linux distros. Although streamlink "uses" ffmpeg, I think it just uses the ffmpeg code, it doesn't actually dynamically link with the ffmpeg shared libraries already on the system, it is statically linked with whatever version of ffmpeg was the latest stable at the time of build. What this means is that installing the latest streamlink provides access to the latest version of the  ffmpeg libraries without actually installing the latest version of ffmpeg.
   
2. Streamlink just calls the ffmpeg libraries, but it is designed specifically for capturing remote streams, which means it is already using the optimal parameters to call the ffmpeg code most effectively with this kind of input
   

I think my original issue may have been a combination of a badly formatted stream, which is not designed for input  to tools like NextPVR, and primitive use of ffmpeg by NextPVR (caused by the fact that it has to use the lowest common denominator to make sure it works everywhere). Streamlink seems to deal much better with badly formatted streams and produces a much smoother result.

Anyway, just switching to Streamlink using an extra solves the issue.