Challenge with Postprocessing.bat

[Bobins]

My days of programming batch files are a bit rusty so I think I’m missing something very basic here.

Having just built a new NPVR system with a quad tuner where I could be recording multiple HD programmes concurrently whilst playing back previously recorded HD content, I was concerned about the potential throughput bottleneck and wear & tear on my new 3Tb hard disks.

I decided that a better approach may be to use an SSD as a “high speed” recoding cache and then move the recordings to the hard disk as they complete using the PostProcessing.bat file.

I set the PostProcessing.bat file to call a file Move.bat logging to a log file in the Logs directory. The Move.bat file checks the recording exists in the relevant directory on the SSD, copies it to the new location, checks the new file/directory exists, updates the database then deletes the old recording file.
I’ve proven this works reliably (even handling when the file name contains an “&” which proved tricky in variables in a batch file ).

The other night I noticed that some recordings hadn’t been moved from the SSD cache. I checked my log file and couldn’t find any trace of a request to move these files or any error. When I checked the NPVR logs I find that the problem occurs when multiple recordings finish at exactly the same time. The first recording details are submitted to the PostProcessing.bat and it triggers the move process which completes successfully and logs the results in my log file.

The 2nd (and any other recordings which complete at the same instant) are logged in the NPVR log as calling PostProcessing.bat and passing the correct parameters however there is no record of these 2nd or subsequent calls to Postprocessing.bat actually resulting in the batch file running.

The 1st call to the PostProcessing.bat file would have still been running when the 2nd, 3rd call etc. would have been made. My log file records the 1st running of the PostProcessing.bat file but shows no evidence of the additional calls nor are there any errors in the Windows or NPVR logs.

I’m assuming here that the problem may be that running concurrent instances of PostProcessing.bat isn’t being allowed and the 2nd and subsequent calls are being ignored because it’s already running. If that’s the case how can I fix this?

Any ideas anyone?

[Graham]

My complete stab in the dark would be to use the "start" command to run the file copy and update process so that a new cmd process is spawned to do the copies.

But ... I don't think that any of this is necessary. I would expect a local hard drive to easily cope with the load from four simultaneous HD recordings and one (or more) HD playbacks.

Have you tried recording and playback while looking at % disk active time in Resource Monitor (started from inside Task Manager)?

[sub]

I'm confident that if the nrecord.log shows it running PostProcessing.bat, then it's definitely running it. This stuff is very simple, and hasn't been changed in years. There is lots of people use PostProcessing.bat files - we'd have heard a lot more about it if it wasn't coping with multiple recordings ending at the same time.

It's likely something in your batch is at fault. For your own logging being missing, maybe you're trying to write to the same log file that is open at the same instance in the other running batch file? Maybe temporarily try new log file names for each run on the batch file? (for example, put recording oid in the filename)

[johnsonx42]

I've run into a variety of conditions with multiple simultaneous scripts running, none of which are NPVR's fault. I use a "sleep %2" (or "timeout /T %2" now with windows vista/7) command at the beginning to make each script pause for a number of seconds equal to the channel number; this way the scripts aren't doing their processing at the same time. For situations where I'm logging the std-out of a command that doesn't finish instantly, I direct the output to a unique log file using %3 (recording OID as sub suggested). Here's my current parallelprocessing.bat, which does a bunch of stuff along those lines (as always, I disclaim any suggestion that my batch file is in any way elegant or the right way to do anything):

Code:

cd c:\users\public\npvr\comskip

echo %date%,%time% - %~n1 (%3) recording from channel %2 on tuner %4 >>"%~dp0\recording.log"

Rem list of channels to not run comskip on; PBS channels have no commercials
if "%2" == "27" goto skip
if "%2" == "28" goto skip
if "%2" == "29" goto skip
if "%2" == "50" goto skip
if "%2" == "51" goto skip
if "%2" == "52" goto skip
if "%2" == "53" goto skip
if "%2" == "58" goto skip
if "%2" == "59" goto skip
if "%2" == "60" goto skip
if "%2" == "61" goto skip

goto next

:skip
echo skipping comskip on %~n1; channel %2 in skip list >>"%~dp0\recording.log"
goto continue

:next
echo   ---  sleeping %2 seconds to avoid simultaneous execution >>"%~dp0\recording.log"
sleep %2

:check
tasklist | find /c "comskip" > %3.txt
set /p count= <%3.txt
if %count% geq 3 (
  echo ...%count% comskips running now, waiting 1 minute >>"%~dp0\recording.log"
  sleep 60
  goto check
)
del %3.txt

echo %date%,%time% - invoking comskip on %1 >>"%~dp0\recording.log"
comskip %1 >>"%~dp0\%3.log"
echo %date%,%time% - comskip finished with %1 >>"%~dp0\recording.log"

:continue

REM ok, now we're going to rename the recorded file
REM first sleep for 5 seconds just to make sure file activity is settled, +60 to avoid npvr padding bug
sleep 65

REM check if comskip has run on this file; if not skip to renaming (this is kinda pointless right now, but might be useful later)
:checkcomskip
if not exist "%~dpn1.log" goto makename

REM get rid of excess comskip files... we only need the .edl
del "%~dpn1.log"
del "%~dpn1.txt"
del "%~dpn1.logo.txt"

:makename
REM I used to have a long sequence of commands to build a new name here, but now imagegrablite does all that

REM ok, ready to rename the file... but first check to make sure no one is watching it!
REM try renaming the file to the same name, see if gets an error
echo checking if %1 is open >>"%~dp0\recording.log"
:checkopen
rem first check that the original file is still here; if it's gone the loop will be infinite
if not exist %1 goto abort
rem now try renaming the file to see if it's open
ren %1 "%~nx1"
if errorlevel 1 goto waitandcheck
echo %1 wasn't open >>"%~dp0\recording.log"
goto rename
:waitandcheck
REM if we got here, the rename failed so wait a minute and try again
echo %1 still open, waiting 1 minute >>"%~dp0\recording.log"
sleep 60
goto checkopen
:abort
echo %1 is gone, aborting!!!>>"%~dp0\recording.log"
goto eof

:rename
REM finally, all is clear.  run imagegrablite against the file
echo %date%,%time% - renaming oid %3, file %1 >>"%~dp0\recording.log"
"c:\program files\npvr\imageGrabLite.exe" --oid %3 --rename --zap2it >>"%~dp0\%3.log"

REM now grab the log output and save it for possible later examination
type "c:\users\public\npvr\logs\imageGrabLite.log" >> "%~dp0\recording.log"

:eof

[Bobins]

Thanks guys, I'll do some furtehr testing tonight.

And for the record Sub, I wasn't for a minute suggesting this was a problem with NPVR....I'm sure it's self inflicted with a Windows/batch issue I've not quite got right.

[Bobins]

Problem solved.

The issue was as Sub suggested. I was using an append to the log file assuming that would allow write from different running batches concurrently....FAIL.

The first batch opens the file for write and holds it open preventing other batches accessing therefore they exit without completing the process. The problem was fixed by creating discrete log files for each running batch (named by time/date).

I didn't really want lots of files accumulating in my log directory so just need to figure out a way to get the info back into a single file

[stustunz]

write speed of a standard hard drive at least 150MB/s
amount of data from HD 3MB/s x4 stations =12MB/s

Are my numbers wrong ?

[stustunz]

Its actually less
the maximum amount of data per frequency is 19.39 Mbit/s which is 2.42MB/s x4 =9.68MB/s

but your hard drive may only write at say 80MB/s if its a green one \

[Bobins]

Wouldn't suggest they are....

My target was more to reduce wear & tear on the hard drives due to head thrashing caused by writing 2 or 3 HD recordings at the same time as playing back 1 or 2 HD recordings.

Over the years I've had a few hard disks fail and the more data you have on them the more painful any attempt at recovery. I had a 1Tb drive fail in my home server two weeks ago (Used for photos & Music storage). Typically at any one time there's 1.5Tb - 2Tb on the NPVR drives. (99% not my recordings but the other half's)

Whilst modern drives should be reliable, I've even had "enterprise" class drives fail after 18 months so I'm being ultra cautious. The drives in the NPVR media store are even set to spin down when not in use.

[stustunz]

Ok i just buy a new one every year to record on they are so cheap now