Disaster Recovery

[ceejayemm]

While carry out some work on another machine I inadvertantly pulled the power plug to my RPI4 which runs NextPVR. In 999 case out of 1000 this wouldn't have been a problem but on this occasion the SD card was corrupted and the PI refused to boot. Having time on my hands, under the UK national COVID lockdown, I decided to rebuild the PI using the new native USB boot feature using a small SSD I had to hand. This all went well and I soon had a working PI again. I have been using NPVR 5.x since January and have a number of recordings that I wanted to reimport into NPVR, this is when my problems started. Due to the way in which the corruption happened I had the recordings (*.ts and *.xml files) stored safely on my NAS but the database and config.xml had been lost and I didn't have a recent export of the recordings. OK, not great problem I thought. I recreated the config via the web app and was ready to start reimporting the recordings. I have some films and a number of TV series recorded so I made a start with the films being individual files. It turns out that the xml file created by NPVR at recording time is NOT quite the format needed to reimport the recording should a problem like mine arise. I needed to edit each xml file to make the necessary changes before the import of each recording could be made. I have done this for most of the films but the thought of changing each series recording xml file is something I will have to do a bit at a time.

I know all the pain of the above is mine - however, with the stable door firmly bolted, I looked around to see if anybody had posted a backup regime (app, script etc) on the forum but couldn't find any. I guess I will need to create a script (to be run via crontab at a suitable time ?) which does the following:

stop the NPVR recording service
copy the database and config.xml file to a safe location
restart the NPVR recording service
runs a recordings export process

I think I can manage to script the first three items but how can I achieve the last (export recordings) ?

Sub - could I make a heart felt plea that you consider adding a facility to a future NPVR version to provide a backup process that allows the the user, if they wish - I do, to save the files needed for a restore like mine above to a safe location of their choosing ? I have attached a file showing the before and after changes to the xml files I had to make to get the recordings to import successfully.

Thanks

Chris

[Graham]

Sub - could I make a heart felt plea that you consider adding a facility to a future NPVR version to provide a backup process that allows the the user, if they wish - I do, to save the files needed for a restore like mine above to a safe location of their choosing ?

There are buttons to export recordings in the Settings -> General screen of the Web App ... Plus, NextPVR automatically creates backup .xml files of the recordings in the data folder after each EPG update.

[ceejayemm]

Graham

Thanks for your quick response. Yes, there is a button in the web app to export recordings, but if you can't get at the web app what do you do then ? Hence my request to add a scheduled backup process to the app. From your info I can now see if I can write a script to copy the database and other pertinent files to a safe location, it would be good if this was documented somewhere. I still think it would be better to include a backup option in the app to help out others who subsequently find themselves in the same situation as me.

I think the xml file produced as part of the recording process could be corrected then that would help too.

Regards.

Chris

[Graham]

I still think it would be better to include a backup option in the app to help out others who subsequently find themselves in the same situation as me.

For the avoidance of doubt ... The data regarding recordings is exported from the database into two backup .xml files in the data folder after every EPG update.  This is different to .xml files created when recordings are created.

[mvallevand]

It is not wise to use the SD drive no exception. They will fail

For me the simplest backup is to save npvr.db3 to an alternate "safe" location no need for sub to do anything. On linux it is very easy to control the tuners to look the same on any new setup.

Martin

[ceejayemm]

I agree, SD Card do fail and this is one of the Pi's greatest weakness and in some respect its greatest strength. Now that it is nearly possible (because its still beta and not yet officially released - why not I don't know) to boot a PI4 from a flash drive, or better still an SSD, may be this will be the change we need to make the PI a better host for NextPVR. In my early testing it certain seems more responsive and even allows me to view recordings via the web app - not that I really want to do this (I know Martin!)

The SD card was being used for the system only, all recording and work spaces were elsewhere BUT the Linux deb installer puts the database and config files on the SD Card with no option to put them anywhere else, other than via changing the config.xml file. You learn by your mistakes, and I am learning.

I don't really want to go delving into the Linux OS to configure things in this day and age, others might but I can make too many other mistakes with a text editor! If Linux ever wants to replace Windows then this mentality has to change. But I am willing to experiment up to a point. A simple location for the database and any other pertinent files in the web app would make life so much simpler. As would the ability to backup these files to another safe location. From what you both have said above it now seems like most of the work has already been done, it just needs the web app to have the ability to make use of this work. Why hide it away ? I appreciate Sub may well have other more pressing things to do hence my original request for this to be available in a future version. If I know it might be forthcoming then I am prepared to wait.

Graham - I appreciate the difference between the recordings export file and the individual programme recording xml files. Why the difference though ? In my case if the individual program recording xml files had been of the same format as the recording export xml file then I could simply have used those which I did have safely tucked away. They are not that different and I doubt it would be a massive job to bring them in line. It then provides another route to restoring a broken system if needed.

The ability to backup an application should not be an onerous task, to paraphrase Martin '... <things> will fail' and it should be simple to set up processes to mitigate this and then restore a system back to a working state should such a situation arise. This application is one of the best I use and as it, hopefully, gets used by more people it needs to cater for both the novice and the expert in equal measures.

Chris

[sub]

To be honest, it doesn't really seem like the sort of thing NextPVR should have to worry about. I can't think of any other applications that have built in data backup functionality in case the file system fails.

If it's a concern, you can use any standard backup program to backup NextPVR's files (along with whatever other files you use choose to backup). You could backup the config.xml and npvr.db3 (or recordings-backup.xml).

[Graham]

I appreciate the difference between the recordings export file and the individual programme recording xml files. Why the difference though ?  In my case if the individual program recording xml files had been of the same format as the recording export xml file then I could simply have used those which I did have safely tucked away. They are not that different and I doubt it would be a massive job to bring them in line. It then provides another route to restoring a broken system if needed.

The individual recording .xml files do not exist when the server runs on Windows ... that data is saved as an alternate data stream (ADS) within the .ts file.  But ... the individual recording .xml files do not contain anything that is not in the database entry for that recording file.  The individual recording .xml files (and the ADS data) exist for some historical reason that is now lost in the mists of time (though Sub might remember).

Are we chasing our tails a bit? ... NextPVR stores all of the data that it uses in npvr.db3 and config.xml ... A backup of those two files is all that is needed to recreate a NextPVR setup.

[ceejayemm]

Graham

We might well be. The only reason for raising the individual program xml files format was because in my situation yesterday it was all I had to try and restore my system. If this is a red herring I am happy to leave it alone.

Sub - I am not asking for anything elaborate. This thread has brought up the fact, not know to me earlier, that the EPG Update process produces a recordings export file. In that case all that needs to be done is to provide a facility copy the database, config file and the recording export file to a safe location. If I could specify the 'safe' location and the process was automated to run on a timer (like the EPG Update is) then that would do just fine. I do think you should provide some sort of process like this if the system can be restored from this set of files.

Chris

[mvallevand]

Why couldn't you just do the backup yourself in your PostUpdateEpg script?

rsync /var/opt/nextpvr/* /location

If you add -r you can even get the metadata. rsync will only copy changed files too.

For restoring video ts recordings I wrote a utility will read the individual recording xml as well as the Windows ADS file (v4 only). In v5 I think all platforms create the XML file.

Martin