split recording to multiple drives

[Jonsnow]

I have been tinkering with HD HomeRun and NextPVR for a few months now and its been working great, however my usage is become more demanding with more shows now being recorded.

The problem i'm running into is pixelation when using all 4 turners on my HD HomeRun, its not the antenna as all channels have this problem but not when recording 2 tuners. I'm thinking it is the speed of my drives thats causing and wondering if its possible to split the recording to 2 drives to address this and possible HDD space limitations.

It's quite possible its not the hard drives and just aging hardware i'm using as my media backend do to the fact i'm also parallel processing comskip while its recording maxing my CPU.

[sub]

It's actually more likely to be packet loss. This is known to happen with the HDHomeRun devices, and SiliconDust has recommended developers switch from udp (which they originally used), to the newer tcp interface they've provided. NextPVR is still using the udp interface at this stage.

[HarryH3]

It's quite possible its not the hard drives and just aging hardware i'm using as my media backend do to the fact i'm also parallel processing comskip while its recording maxing my CPU.

One easy test to see if comskip is causing the issue is to temporarily disable it and record 4 shows at once, to see if they record properly. Comskip doesn't just max your CPU, it can also dominate Disk I/O if you don't throttle it, especially if it is running multiple instances (like when 4 recordings end at the same time). Multiple instances of comskip running at one time can be quite demanding on system resources. You can force comskip to be less aggressive on Disk I/O by tinkering with the "play nice" settings in comskip.ini. Mine are set as:

Code:

play_nice_start=1900 ; at what time of the day should comskip slow itself down
play_nice_end=2200 ; ending time for self slowing
play_nice_sleep=30

This slows down comskip considerably during prime time, the hours when multiple things are most likely to be recording/viewing/comskip processing, etc.

There is also a script you can add to either PostProcessing.bat or ParalellProcessing.bat, to limit how many instances of Comskip can run simultaneously. I found it on this site many years ago and have been using it ever since. It's quite handy with the ancient Core2Duo system that I use as my NPVR server.

Code:

@echo off

cd C:\Users\Public\NPVR\Comskip

echo %date%,%time% - PostProcessing.bat invoked on %1 recorded from channel %2 >>PostProcessing.log

Rem list of channels to not run comskip on; PBS channels have no commercials
if "%2" == "608" goto skip
Rem if "%2" == "28" goto skip

goto next

:skip
echo skipping comskip on %1; channel %2 because it is in skip list >>PostProcessing.log
goto continue

:next
rem need to add a pause to avoid race conditions
echo   ---  sleeping %2 seconds to avoid simultaneous execution >>PostProcessing.log
sleep %2

rem check how many comskips are running; if 1 or more are running wait 1 minute and check again
rem the third parameter passed in npvr 1.5.36 and later is a unique recording oid number, so it makes a good filename to store the temporary count
:check
tasklist | find /c "comskip" > %3.txt
set /p count= <%3.txt
if %count% geq 1 (
  echo ...%count% comskips running now, waiting 1 minute >>PostProcessing.log
  timeout 60
  goto check
)
del %3.txt

rem continuing now

Rem this trick also makes it possible to have dedicated comskip.ini file for certain channels.
rem if "%2" == "79" goto channel79
Rem use the default comskip.ini for all other channels
rem goto generic

rem :channel79
rem comskip --ini=channel79.ini %1
rem goto continue

:generic
echo %date%,%time% - invoking comskip on %1 >>PostProcessing.log
comskip %1
echo %date%,%time% - comskip finished with %1 >>PostProcessing.log

:continue

:eof

[2leftfeet]

I have been tinkering with HD HomeRun and NextPVR for a few months now and its been working great, however my usage is become more demanding with more shows now being recorded.

The problem i'm running into is pixelation when using all 4 turners on my HD HomeRun, its not the antenna as all channels have this problem but not when recording 2 tuners. I'm thinking it is the speed of my drives thats causing and wondering if its possible to split the recording to 2 drives to address this and possible HDD space limitations.

It's quite possible its not the hard drives and just aging hardware i'm using as my media backend do to the fact i'm also parallel processing comskip while its recording maxing my CPU.

It sounds like a network problem as sub suggests. I have a Prime and a HDHR Quatro on the same gigabit switch (on occasion, I have up to 6 tuners recording at once and have no such issues) - is your network 100 mb/sec or gigabit?

[Jonsnow]

It sounds like a network problem as sub suggests. I have a Prime and a HDHR Quatro on the same gigabit switch (on occasion, I have up to 6 tuners recording at once and have no such issues) - is your network 100 mb/sec or gigabit?

It’s gigabit 100%, when I first set it up I did test recordings all 4 tuners although only checked start and end of the video to make sure it recorded and didn’t notice any artifacts, it has been running since January and only now noticing these artifacts.

[BrettB]

It’s gigabit 100%, when I first set it up I did test recordings all 4 tuners although only checked start and end of the video to make sure it recorded and didn’t notice any artifacts, it has been running since January and only now noticing these artifacts.

My network is also 100% gigabit. But since the fall season began (and hence I have several concurrent recordings going on), I have noticed occassional hiccups in recordings. My signal is from cable and the cable company just ran all new lines all the way from the pole to my cable modem/HDHR location. So I've got great signal strength and am fairly confident the problem isn't the reception. I have 2 HDHR Primes (both with cablecards). I haven't had a chance to dig into the problem more, but I suspect it may be happening when there are multiple streams coming from both Primes concurrently. (The network path between the HDHRs and the NextPVR server does include a gigabit switch and the router.)

It's actually more likely to be packet loss. This is known to happen with the HDHomeRun devices, and SiliconDust has recommended developers switch from udp (which they originally used), to the newer tcp interface they've provided. NextPVR is still using the udp interface at this stage.

Sub: Do you have any plans to try re-introducing the TCP support in NextPVR's HDHR device interface?

[Jonsnow]

One easy test to see if comskip is causing the issue is to temporarily disable it and record 4 shows at once, to see if they record properly. Comskip doesn't just max your CPU, it can also dominate Disk I/O if you don't throttle it, especially if it is running multiple instances (like when 4 recordings end at the same time). Multiple instances of comskip running at one time can be quite demanding on system resources. You can force comskip to be less aggressive on Disk I/O by tinkering with the "play nice" settings in comskip.ini. Mine are set as:

Code:

play_nice_start=1900 ; at what time of the day should comskip slow itself down
play_nice_end=2200 ; ending time for self slowing
play_nice_sleep=30

This slows down comskip considerably during prime time, the hours when multiple things are most likely to be recording/viewing/comskip processing, etc.

There is also a script you can add to either PostProcessing.bat or ParalellProcessing.bat, to limit how many instances of Comskip can run simultaneously. I found it on this site many years ago and have been using it ever since. It's quite handy with the ancient Core2Duo system that I use as my NPVR server.

Code:

@echo off

cd C:\Users\Public\NPVR\Comskip

echo %date%,%time% - PostProcessing.bat invoked on %1 recorded from channel %2 >>PostProcessing.log

Rem list of channels to not run comskip on; PBS channels have no commercials
if "%2" == "608" goto skip
Rem if "%2" == "28" goto skip

goto next

:skip
echo skipping comskip on %1; channel %2 because it is in skip list >>PostProcessing.log
goto continue

:next
rem need to add a pause to avoid race conditions
echo   ---  sleeping %2 seconds to avoid simultaneous execution >>PostProcessing.log
sleep %2

rem check how many comskips are running; if 1 or more are running wait 1 minute and check again
rem the third parameter passed in npvr 1.5.36 and later is a unique recording oid number, so it makes a good filename to store the temporary count
:check
tasklist | find /c "comskip" > %3.txt
set /p count= <%3.txt
if %count% geq 1 (
  echo ...%count% comskips running now, waiting 1 minute >>PostProcessing.log
  timeout 60
  goto check
)
del %3.txt

rem continuing now

Rem this trick also makes it possible to have dedicated comskip.ini file for certain channels.
rem if "%2" == "79" goto channel79
Rem use the default comskip.ini for all other channels
rem goto generic

rem :channel79
rem comskip --ini=channel79.ini %1
rem goto continue

:generic
echo %date%,%time% - invoking comskip on %1 >>PostProcessing.log
comskip %1
echo %date%,%time% - comskip finished with %1 >>PostProcessing.log

:continue

:eof

I was maxing my CPU before, limiting to 2 comskip instances has removed the artifacts and my CPU is now between 40% - 70% which I'm okay with. I'd rather hit my CPU hard so i can watch as soon as 2 are finished recording and processed. The play_nice commands didn't really help much in my case, I may try testing 3 comskip instances with the play_nice command but for now I'm fine with what I have currently.

[HarryH3]

Glad to hear that it's helping. You can play around with the value of the play_nice_sleep command. I found this description of the setting on the compkip forum:

;Amount of miliseconds for each frame the processing should be suspended to reduce CPU load.
;;The effect depends on the processing speed in frames/second. When 10 frames/second are processed then a value of 10 only reduces 10% PCU load. When 100 f/s then the reduction is 50%

All of the Comskip settings and descriptions can be found here: http://www.kaashoek.com/comskip/viewtopic.php?f=2&t=474

[Jonsnow]

Glad to hear that it's helping. You can play around with the value of the play_nice_sleep command. I found this description of the setting on the compkip forum:

;Amount of miliseconds for each frame the processing should be suspended to reduce CPU load.
;;The effect depends on the processing speed in frames/second. When 10 frames/second are processed then a value of 10 only reduces 10% PCU load. When 100 f/s then the reduction is 50%

All of the Comskip settings and descriptions can be found here: http://www.kaashoek.com/comskip/viewtopic.php?f=2&t=474

default is 10 and that works, to lazy to test it as getting the script working took longer than i'd like. The sleep %2 kept failing as it was processing to 57.1 seconds and totally didn't see that and wasted to much time. It also needed to be timeout and not sleep.

[johnsonx42]

default is 10 and that works, to lazy to test it as getting the script working took longer than i'd like. The sleep %2 kept failing as it was processing to 57.1 seconds and totally didn't see that and wasted to much time. It also needed to be timeout and not sleep.

When I originally wrote the script the timeout command didn't exist, so I had to use the sleep command from a downloaded pack of microsoft tools (I think it was called something like windows server resource kit). Looks like I later switched to the Timeout command in one spot, but left the sleep command in the other. Yes, the initial sleep command can introduce undesired delays if you use large channel numbers; there are probably better ways to handle it but sleep %2 always worked fine for me. The sticky in my signature includes a link to the original version of the script and probably some other commentary.