bulk comskip processing script

[baj1]

I made some headway here and thought I'd share.  I have a pretty decent volume of saved recordings, many of which have old comskip files from many versions ago.  I don't know if I'm saying this right, but as I understand it they trip up Kodi/knewc due to the edl_skip_field setting.  It defaults to 0=cut and you will not be able to play portions of the video marked as commercials.  Setting it to a value = 3 will allow playback of the commercial marked portions. 

Anywho, my solution was to re-run comskip on all of these recordings so I started a bash script to handle it in bulk in Linux.  I'm far from a bash guru but was able to piece together some code with a lot of help from other forums and Martin's PostProcessing.sh example.  Here it is:

Code:

#! /bin/bash
# comskip_subfolders.sh
# This script will bulk-process TS files in sub-folders.  Execute in your root Video folder and it will run comskip on all TS files in all the "show" subfolders
# nice eases CPU use, range is -20 (fastest/highest CPU usage) to 19 (slowest/least CPU usage), negative values require sudo

for d in */; do
    echo "Processing Folder $d"
    cd "$d"
    for file in *.ts
    do
      echo "Processing file " $file
      nice -n 19 comskip --ini=/home/<username>/Comskip/comskip.ini "$file" > /dev/null
    done
    cd ..
done

Just paste the code into your favorite text editor and change <username> to your user, or modify the path completely if you installed Comskip elsewhere.  Save the contents to your ~/bin folder and don't forget to chmod +x to make it executable.  If you haven't already you might want to make sure you've set edl_skip_field = 3 in your comskip.ini, or add that line to the end of the ini file if needed.

Change the nice value if you like.  I have a new server with a 6-core 3.6GHz CPU and I was running two instances of the script (in two separate folders) with nice @ "2" and each instance was showing about 240% utilization in System Monitor.  It was processing 1 hour shows in around 100 seconds, although this did vary with the particular show it was processing.

Disclaimer!  As stated I am FAR from a bash guru, use at your own risk!  It worked on my single-depth subfolder structure of my Video (recordings) folder.

I hope someone finds this useful.  This is v1.0, so I'm open to feedback/improvements.

Cheers,
-Brad

[ElihuRozen]

If you wanted to do this from a single command line, the find command might be useful for you (note that this will not display each file as it is processed, which your script does do):

Code:

find . -name "*.ts" -exec nice -n 19 comskip --ini=$HOME/Comskip/comskip.ini "{}" >/dev/null 2>&1 \;

where:
find - command that we are running to find all videos in any subdirectories
. - top level directory to start search.  In this case, the current directory
-name "*.ts" - what to search for.  In this case, any file with a .ts extension
-exec - what to do with each file we find.  In this case, execute the command that follows
nice -n 19 comskip --ini=$HOME/Comskip/comskip.ini - the actual command to execute
"{}" - this is substituted with the actual file name for each file we find
>/dev/null - get rid of all standard output
2>&1 - redirect standard error to standard out.  In this case, get rid of any error messages since standard out is going to /dev/null
\; - this is needed to mark the end of the command that -exec will execute

[VCR58]

I made some headway here and thought I'd share.  I have a pretty decent volume of saved recordings, many of which have old comskip files from many versions ago.  I don't know if I'm saying this right, but as I understand it they trip up Kodi/knewc due to the edl_skip_field setting.  It defaults to 0=cut and you will not be able to play portions of the video marked as commercials.  Setting it to a value = 3 will allow playback of the commercial marked portions. 

Anywho, my solution was to re-run comskip on all of these recordings so I started a bash script to handle it in bulk in Linux.  I'm far from a bash guru but was able to piece together some code with a lot of help from other forums and Martin's PostProcessing.sh example.  Here it is:

Code:

#! /bin/bash
# comskip_subfolders.sh
# This script will bulk-process TS files in sub-folders.  Execute in your root Video folder and it will run comskip on all TS files in all the "show" subfolders
# nice eases CPU use, range is -20 (fastest/highest CPU usage) to 19 (slowest/least CPU usage), negative values require sudo

for d in */; do
    echo "Processing Folder $d"
    cd "$d"
    for file in *.ts
    do
      echo "Processing file " $file
      nice -n 19 comskip --ini=/home/<username>/Comskip/comskip.ini "$file" > /dev/null
    done
    cd ..
done

Just paste the code into your favorite text editor and change <username> to your user, or modify the path completely if you installed Comskip elsewhere.  Save the contents to your ~/bin folder and don't forget to chmod +x to make it executable.  If you haven't already you might want to make sure you've set edl_skip_field = 3 in your comskip.ini, or add that line to the end of the ini file if needed.

Change the nice value if you like.  I have a new server with a 6-core 3.6GHz CPU and I was running two instances of the script (in two separate folders) with nice @ "2" and each instance was showing about 240% utilization in System Monitor.  It was processing 1 hour shows in around 100 seconds, although this did vary with the particular show it was processing.

Disclaimer!  As stated I am FAR from a bash guru, use at your own risk!  It worked on my single-depth subfolder structure of my Video (recordings) folder.

I hope someone finds this useful.  This is v1.0, so I'm open to feedback/improvements.

Cheers,
-Brad

Hi Brad,

Thanks for the bash script! I was searching around for something like this after getting comskip running on my Linux NextPVR server.

I have a question concerning the usage of the script. I have all my shows in a directory called 'Recorded TV',  the full path is /mnt/sda1/'Recorded TV'. So would I just run comskip_subfolders.sh or do I need to add the folder after the command?

Sorry, I'm a newbe to comskip, and bash, and linux.

Thanks, Rex

[baj1]

No worries, I'm still learning too. I made the script to run in what would be your "Recorded TV" folder. If the folder where the script is located is not in your path, you will need to include it when executing the command. I put my bash scripts in a subfolder of my home called "bin". If you did the same, you would execute like this in a terminal:

$ cd /mnt/sda1/Recorded\ TV
$ ~/bin/comskip_subfolder.sh

'~' is a shortcut for your home folder path.

You could always make a set of test folders if you are unsure, but there's really no harm in trying it because if it doesn't find any TS files it doesn't do anything.

[baj1]

If you wanted to do this from a single command line, the find command might be useful for you (note that this will not display each file as it is processed, which your script does do):

Code:

find . -name "*.ts" -exec nice -n 19 comskip --ini=$HOME/Comskip/comskip.ini "{}" >/dev/null 2>&1 \;

where:
find - command that we are running to find all videos in any subdirectories
. - top level directory to start search.  In this case, the current directory
-name "*.ts" - what to search for.  In this case, any file with a .ts extension
-exec - what to do with each file we find.  In this case, execute the command that follows
nice -n 19 comskip --ini=$HOME/Comskip/comskip.ini - the actual command to execute
"{}" - this is substituted with the actual file name for each file we find
>/dev/null - get rid of all standard output
2>&1 - redirect standard error to standard out.  In this case, get rid of any error messages since standard out is going to /dev/null
\; - this is needed to mark the end of the command that -exec will execute

Thanks!  I figured someone would know a more efficient way of doing this, and I do like to learn so thanks for the info.

My 2 instances have been crunching away since late last night and are only on the C's in one folder and the F's in another, so I'm glad I have the echo status lines to show progress.

[mvallevand]

I would have just used sed to fix my comskip.ini files

sed -e 's/0$/3/'g

changes the trailing 0 to 3 on each line

Code:

find folder -name "*.edl" -exec  sed -i -e 's/0$/3/'g {} +

Martin

[baj1]

Thanks Martin. I had done that manually on a couple files and was thinking that would be harder to do, guess not. Should be super fast too since it doesn't have to re-process the TS files.

[baj1]

So I guess I really wanted an edl setting of 2 not 3. With 3 it automatically skips the commercials. we prefer to do it manually because comskip isn't always perfect. Weird that on both LibreELEC and CoreELEC clients that they worked like "2" with a "3" in the .edl files. It was just the local Kodi-knewc client that was autoskipping with the "3".

Simple update to comskip.ini to fix future recordings.

Thanks again Martin, your script worked great for updating the edl settings for the bazillion recordings we already have. :-)