Re-recording existing content

[daneo]

Hi Sub,

long time no hear -- need to send you a beer or two for Christmas!

I'm doing a lot of recording on NPVR, but (mea culpa) mainly using it as a recording back end for Kodi for OTA movies and series in Germany. The filters I am using tend to capture a lot of repeats, and it gets worse once the data has been "cleaned".

Cleaning means that the raw TS files are cut with VideoReDo to remove pre/post padding and commercials, transcoded to H.264 MP4 and sometimes MKV files, then moved over to a bunch of NAS folders for Kodi and distribution to the devices at home.

My problem is that the automated web recording sweep continually picks up movies and content I already have loaded -- how can I setup the system such that local and NAS content is scanned for pre-existing files and not auto-recorded?

For example I have data stored on file shares similar to these:

"J:\Movies1\3 Engel für Charlie - Volle Power\3 Engel für Charlie - Volle Power.mp4"
or
"\\wdmycloud\Movies\Appleseed\Appleseed.mp4"
or
"N:\Movies3\2 Millionen Dollar Trinkgeld (1994)\2 Millionen Dollar Trinkgeld (1994).mkv"

I know NPVR has existing recording crash advoidance logic, but not sure how to apply in the Kodi use case here.

Thanks,
Dane

[mvallevand]

Since the duplicate check is based on the unique I'd avoiding duplicates will be difficult. It might be better to update the file name in NextPVR after cleaning and extending the retention time for avoiding duplicates.

Martin

[daneo]

Hi Martin,

I managed to get a close approximation -- but after the fact instead -- once a recording is created, I use the NPVR native "filename (date)" convention for storage, and have a batch file in postprocessing to check my various drives for a directory of the same name and then mark the directory if found. I could imagine NPVR would perform the same check when starting a recording and bail on recording if found.

I store all temp files on H:\Recordings, and if the directory is in use and can't rename, create a reminder file just in case. Finally it checks the DELETED folder which is for recordings I've sorted out (don't have and don't want again) in case a rerun comes -- it's just full of zero-size directories with the recordings names.

The last call is to try and work better around the logfile rollovers -- so each time a recording finishes, it snaps the logs.

My WISH -- could NPVR do the directory check first, before going to allocate a tuner???

Take a look

Code:

H:
CD Recordings
for /D %%f in (*.*) do (
  if exist "D:\Movies3\%%f" (move "%%f" "x-%%f"
                     if errorlevel 1 echo yes > "x-%%f.txt")
  if exist "G:\Movies2\%%f" (move "%%f" "x-%%f"
                       if errorlevel 1 echo yes > "x-%%f.txt")
  if exist "J:\Movies1\%%f" (move "%%f" "x-%%f"
                       if errorlevel 1 echo yes > "x-%%f.txt")
  if exist "W:\%%f" (move "%%f" "x-%%f"
                       if errorlevel 1 echo yes > "x-%%f.txt")
  if exist "H:\RECORDINGS\DELETED\%%f" (move "%%f" "x-%%f"
                       if errorlevel 1 echo yes > "x-%%f.txt")
  )

call C:\Users\Public\NPVR\logfile.bat

[daneo]

Since the duplicate check is based on the unique I'd avoiding duplicates will be difficult. It might be better to update the file name in NextPVR after cleaning and extending the retention time for avoiding duplicates.

Martin

Hi Martin,

can you please expand a bit on the duplicate checking for recordings? Can you give me some examples of how the DB entries are used for this, or is it based on external data instead?

Thinking more about this -- the database suffers a bit from "groundhog day" as if you plan a recording, then delete it -- but have a filter in place (actor, etc) then the next time the EPG is scanned the entry will be flagged to record. There has to be a way to get a "stuff I already have" data set in place -- like a hash of the entry "Name (year)" which could be validated during recording planning. This way a category of "duplicated avoided" could be created. The current process I have (and I may have a crazy use case) is to put up a simple library of OTA films which I can use at home. The pruning of such is my main problem, as the re-recording of existing content takes up a lot of resources -- and now is being limited to tuner capacity as my scripts pretty much weed out the chaff after recording is completed.

Would it be possible to create a DB entry on the EPG_TABLE which creates a hash of something like the Name (Year) when the EPG is downloaded -- and could then be used against a second table of hashes based on a scan of the local drives? The only thing missing is then some logic to determine if a particular entry has already been recorded before and is in a good state.

Thanks,
Dane

[Graham]

can you please expand a bit on the duplicate checking for recordings? Can you give me some examples of how the DB entries are used for this, or is it based on external data instead?

My experience has been that the duplicate prevention stuff doesn't work for "Quick" (i.e. one-off) recordings. It appears that NextPVR takes the view that I am more clever than it is and that I know what I am doing when I ask for a Quick recording. Sadly, NextPVR is wrong which is why I have umpteen duplicates of movies that I have recorded repeatedly.

[mvallevand]

As I said it is based on the uniqueid field so if that id is in recordings or in recently deleted tables it won't be recorded. The default for recent is 90 days which is too short for my use so I sent it for a large number.

As Graham posted this a recurring recording rule so one off and NEWA recordings don't follow it. If it is a manual rule you might need to update the rule to avoid duplicates

Martin

[daneo]

Now it gets interesting - and closer to where I want to go. Here's some information on combining PowerShell and SQLite

http://ramblingcookiemonster.github.io/S...owerShell/

And if it's that easy, then we would need a few things so that we don't stop on NPVR too much

1) a column in the EPG table for the planned file name without extension so something like "War and Peace (1932)"
2) a column in the EPG table with a "schedule_ignore" flag which can be set externally

This means that we would check the walk EPG with a PoweShell script, see if the Genre is "Movie" and validate the filename against a set of local directories -- if we have HD already, then set ignore flag, if we have on the DNR list, also set ignore flag. Otherwise it's free game for the existing recording scheduling logic.

Looking at the table -- where does the unique_id come from? Is this from SchedulesDirect or a local value? Also, when the EPG is updated -- do you do a delta/compare/update or a replace? Given that the scheduling scan happens after EPG update, it will be tricky to update the data as needed before the scan occurs, unless it can be combined with the postepgupdate scripting.

Would this be possible? I want to avoid a new table where I would have to capture all EPG unique_id entries, create the filenames, and then parse that table -- especially if we would add the ignore flag in the first place.

Sub -- would need your help to extend the EPG_EVENT table with the two fields
Sub/UJB -- would need your help to code the auto record scan to ignore some entries when scanning

Thanks,
Dane

[sub]

Looking at the table -- where does the unique_id come from? Is this from SchedulesDirect or a local value?

The unique_id comes from the EPG source. SchedulesDirect provides one for every show. For other EPG sources, you only get one if the season/episode numbers are supplied. If a unique_id is not clearly available, then the show is treated to be a new episode (since name/description of a show definitely can not be used for this purpose - generic descriptions are used for heaps of shows)

Also, when the EPG is updated -- do you do a delta/compare/update or a replace?

Usually it'll delete and bulk insert everything, since this is a lot quicker and cleaner than having to look at every listing to check for differences etc, and do different actions for each.

Would this be possible? I want to avoid a new table where I would have to capture all EPG unique_id entries, create the filenames, and then parse that table -- especially if we would add the ignore flag in the first place.

Sub -- would need your help to extend the EPG_EVENT table with the two fields
Sub/UJB -- would need your help to code the auto record scan to ignore some entries when scanning

To be honest, I don't really like the sound of this, and it isn't something we've seen other people asking for, so I'd probably prefer not to do this. It also doesn't sound like a very clean way of doing this.

If you're using xmltv for your EPG, your best bet might running a script to manipulate the data before NextPVR imports it. ie, might need to make movies look like season1/episode1, and then NextPVR's existing duplicate logic would avoid recording the same movie twice.

[mvallevand]

@daneo I see your question to UJB "Sub/UJB -- would need your help to code the auto record scan to ignore some entries when scanning"

I think I mentioned that UJB's code doesn't handle duplicates or unique IDs at all. I ended up with 5-10 recordings of the same show in a day so I had to give up using it.

Martin