Run Multiple File Conversions On Your 4 Core -Burn Your Processor

[luttrell1962]

I wrote some xbasic code and batch files to process mulitple ffmpeg 2 pass conversions. 3 uses 75-100 cpu time depending on the type of conversion.

xbasic code I had some printing going on but I commented that out you may want to use it to debug this if you choose.

Code:

' ####################
' #####  PROLOG  #####
' ####################
'
PROGRAM    "multiconv"
VERSION    "0.0001"
'
IMPORT    "xst"
'
DECLARE FUNCTION  Entry ()
'
'
' ######################
' #####  Entry ()  #####
' ######################
'
FUNCTION  Entry ()
'
    XstGetCommandLine (@command$)
    XstGetCommandLineArguments (@argc, @argv$[])
'
'########################################################
' Fix input file names from GB-PVR So they can be passed
' to a batch file
'########################################################
'
    DIM file1$[1]
    DIM file2$[1]
    DIM ffmpeg1$[1]
    DIM ffmpeg2$[1]
    DIM ffmpeg3$[1]
    file1$=""
    file2$=""
    iptr=1
XstHideConsole()
'
'***********************************************************
    DO
        IF LEN(file1$) > 0 THEN file1$=file1$+" "
        file1$=file1$+argv$[iptr]
        iptr=iptr+1
    LOOP UNTIL RIGHT$(file1$,1) == CHR$(34) 'Quote mark
    DO
        IF LEN(file2$) > 0 THEN file2$=file2$+" "
        file2$=file2$+argv$[iptr]
        iptr=iptr+1
    LOOP UNTIL RIGHT$(file2$,1) == CHR$(34) 'Quote mark
'************************************************************
'
' Check for first log file then second 2 pass log file if
' If first or second log file does not exist then spawn
' divx2pass.bat else do not spawn third instance. But execute
' and wait for completion of it and all log files then start
' over.
'
'************************************************************
ffmpeg1$=":divx2pass.bat "+file1$+" "+file2$+" one"
ffmpeg2$=":divx2pass.bat "+file1$+" "+file2$+" two"
ffmpeg3$=":divx2pass.bat "+file1$+" "+file2$+" three"
XstChangeDirectory("third party")
    XstGetFileAttributes("one-0.log",@attributes)
    IFZ attributes THEN            'file does not exist
        SHELL(ffmpeg1$)
'            PRINT ffmpeg1$
'            PRINT "Wait Until ProjectX finshes processing file"
    ELSE
        XstGetFileAttributes("two-0.log",@attributes)
            IFZ attributes THEN            'file does not exist
                SHELL(ffmpeg2$)
'                PRINT ffmpeg2$
'                PRINT "Wait Until ProjectX finshes processing file"
            ELSE
                SHELL(ffmpeg3$)                'Wait for third pass to complete
'                PRINT ffmpeg3$
'                PRINT "Wait Until ProjectX finshes processing file"
            END IF
    ENDIF
END FUNCTION
END PROGRAM

Batch file to be called by GB-PVR it has code that waits untils projectx preprocesses the file. At the third instance it waits for it or one of the other 3 process's to complete before returning to GBPVR

Code:

cd "\program files (x86)\Devnz\GBPVR\Third Party"
REM *****************************************************
REM Determine current run instance by created log file
REM******************************************************
IF NOT EXIST one-0.log SET logfile=one-0.log
IF NOT EXIST one-0.log GOTO logfiledone
IF NOT EXIST two-0.log SET logfile=two-0.log
IF NOT EXIST two-0.log GOTO logfiledone
IF NOT EXIST three-0.log SET logfile=three-0.log
IF NOT EXIST three-0.log GOTO logfiledone
:logfiledone
multiconv.exe %1 %2
REM *****************************************************
REM Wait for Project.x to process file
REM******************************************************
:wlogfile
IF NOT EXIST "%logfile%" GOTO wlogfile
rem wait 500 seconds x 3 for projectx to complete
REM *****************************************************
REM Exit if any of the three process's complete
REM******************************************************
:loop
IF NOT EXIST "one-0.log" exit
IF NOT EXIST "two-0.log" exit
IF EXIST "three-0.log" GOTO loop
exit

Multipass ffmpeg batch file. I do preprocessing of the file with projectx. I pass a log file prefix one,two,three so that log files are kept separate.

Code:

javaw -jar d:\projectX.old\projectX.jar %1
echo %date% %time% %1 >> d:\projectx.old\runinst.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\ffmpeg\ffmpeg.exe" -y -i "%~dp1%~n1.m2v" -an -r 30 -f avi -qmax 4 -qmin 1 -threads 8 -s 800x450 -b 2500k -pass 1 -passlogfile %3 -minrate 0k -maxrate 10000k -vcodec mpeg4 -bufsize 3500k -vtag divx nul
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\ffmpeg\ffmpeg.exe" -y -i "%~dp1%~n1.m2v" -i "%~dp1%~n1.ac3" -ab 128k -acodec libmp3lame -ac 2 -r 30 -async 1 -ar 44100 -f avi -qmax 4 -qmin 1 -threads 8 -s 800x450 -b 2500k -pass 2 -passlogfile %3 -minrate 0k -maxrate 10000k -vcodec mpeg4 -bufsize 3500k -vtag divx %2
del "%~dp1%~n1.m2v"
del "%~dp1%~n1.ac3"
del "%~dp1%~n1[1].ac3"
del %3-0.log
IF EXIST "%~dp1%~n1.avi" DEL %1
EXIT

[luttrell1962]

GB-PVR calls multiconvbat.bat that calls multiconv.exe multiconv.exe calls divx2pass.bat.

I have uploaded the all the files here zipped multiconv.zip
http://www.mediafire.com/?2zdjdttwug2

You can use it as long as you name your conversion batch file divx2pass.bat make sure you know that 3 items are passed original filename destfilename ffmpeg 2 pass prefix. You must use 2 pass for this to work. Might as well since you can fun 3 conversion at once.

I upgraded the multiconvbat.bat file to use a timer instead of just spinning forever take less cpu time. The timer code only works with Vista and 7. Search the internet for timer code for xp and replace the line. Make sure you test the timer code as most use choice command and the syntax has changed over different versions of windows.

Code:

@echo off
cd "\program files (x86)\Devnz\GBPVR\Third Party"
REM *****************************************************
REM Determine current run instance by created log file
REM ******************************************************
IF NOT EXIST one-0.log SET logfile=one-0.log
IF NOT EXIST one-0.log GOTO logfiledone
IF NOT EXIST two-0.log SET logfile=two-0.log
IF NOT EXIST two-0.log GOTO logfiledone
IF NOT EXIST three-0.log SET logfile=three-0.log
IF NOT EXIST three-0.log GOTO logfiledone
:logfiledone
multiconv.exe %1 %2
REM *****************************************************
REM Wait for Project.x to process file
REM ******************************************************
:wlogfile
timeout /T 30
IF NOT EXIST "%logfile%" GOTO wlogfile
rem wait 500 seconds x 3 for projectx to complete
REM *****************************************************
REM Exit if any of the three process's complete
REM ******************************************************
:loop
timeout /T 30
IF NOT EXIST "one-0.log" exit
IF NOT EXIST "two-0.log" exit
IF EXIST "three-0.log" GOTO loop
exit

[luttrell1962]

I changed my qmax values to qmin=1 and qmax=2 and I had some failures on conversion. Since ffmpeg creates a zero size avi file the batch file deletes the ts file. It could be a bug in the code but it worked perfectly standalone testing and the other night until I changed the q values. I've rewrote the batch file to check the avi file for zero size and delete it, that way the ts file does not get deleted. I also removed qmax and qmin and substituted -qsame. Added output of the ffmpeg second pass file to a log file in the conversion directory for troubleshooting purposes. It took some research and testing to find a solution for testing filesize.

divx2pass.bat

Code:

javaw -jar d:\projectX.old\projectX.jar %1
echo %date% %time% %1 %3>> d:\projectx.old\runinst.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\ffmpeg\ffmpeg.exe" -y -i "%~dp1%~n1.m2v" -an -r 30 -f avi -sameq -threads 8 -s 800x450 -b 3000k -pass 1 -passlogfile %3 -minrate 0k -maxrate 10000k -vcodec mpeg4 -bufsize 3500k -vtag divx nul
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\ffmpeg\ffmpeg.exe" -y -i "%~dp1%~n1.m2v" -i "%~dp1%~n1.ac3" -ab 128k -acodec libmp3lame -ac 2 -r 30 -async 1 -ar 44100 -f avi -asameq -threads 8 -s 800x450 -b 3000k -pass 2 -passlogfile %3 -minrate 0k -maxrate 10000k -vcodec mpeg4 -bufsize 3500k -vtag divx %2 >> "%~dp1%~n1ffmpeg.log"
del "%~dp1%~n1.m2v"
del "%~dp1%~n1.ac3"
del "%~dp1%~n1[1].ac3"
del %3-0.log
:: Check for zero size avi file
for %%a in ("%~dp1%~n1.avi") do if %%~za==0 del %%a
IF EXIST "%~dp1%~n1.avi" DEL %1
EXIT

Edit : There was a syntax error in the ffmpeg code I fixed it qsame should have been sameq
Edit : Found bug in zero length file detection was
for %%a in (%~dp1%~n1.avi) do if %%~za==0 del "%%a" should be
for %%a in ("%~dp1%~n1.avi") do if %%~za==0 del %%a
it worked with directory names without spaces when I tested it but with directories with spaces it needs quotes around the (in) portion

[luttrell1962]

Seems that the qam recording has some problems lately ffmpeg is reporting this
This is directly off of cable so not sure what is happening.



I think conversion if failing because of this but if I view the ts file it look ok.
Edit : after watching the video I do see some glitches. Maybe my splitter off of cable signal is to low or cable qam is having a problem. Seems mostly the same channel. I will rescan the channels maybe the frequency changed slightly.

[luttrell1962]

Seems if I set my cell phone next to the computer it interferes with the QAM signal so take note.

I've also been playing with xvid conversions. If using xvid cpu utilization increases to about 40% per conversion. I'll just set threads to 6 instead of writing 2 instance code. After I test xvid conversion for a couple of days I'll post the batch file. It does a lot better than mpeg4, smaller files and better quality. But takes longer 16-25fps vs 45-60fps on my machine.

[luttrell1962]

I've changed over to Xvid encoding using mencoder. It is faster. I've tested it for a couple of days now. With mencoder there is no need to run projectx on the file prior to conversion since it seems to realign audio correctly. I added some code to check for small filesize of the avi file and delete it. 10 min recordings would probably fail this test and delete the avi file, but you can change the size to whatever you like.

I've also modified the conversion batch file to output several files capturing pass 1 and pass 2 information. I'm currently working on writing a program to query these files during conversion and display information about the current conversion(s) status (filename fps size pass#). The xbasic program now calls xvid2pass.bat. I could rewrite this program to pass a third parameter to it as to the batch file to call. That would make it more universal. Runinst.txt log file is now stored in the thirdparty directory since I'm no longer using projectX.

You can download mencoder that is tailored for your CPU here
http://oss.netfarm.it/mplayer-win32.php

Edit 03/13/2010 - new version available multiconvxvid0.1.zip
You can download multiconvxvid.zip from
http://www.mediafire.com/?nzqmmyyumrz

xvid2pass.bat- The bitrate might be a little low. Also the threads option does not seem to work. But two instances uses about 90% of my CPU time. It would be easy to modify multiconvbat.bat to wait for conversion 2 to finish before returning to GB-PVR.

Code:

echo %date% %time% %1 %3 >> runinst.txt
echo %1 > %3pass1.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" %1 -oac mp3lame -ovc xvid -xvidencopts bitrate=3000:pass=1:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31:threads=2 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=1280:720 -o nul 1>> %3pass1.txt
echo %1 > %3pass2.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" %1 -oac mp3lame -ovc xvid -xvidencopts bitrate=3000:pass=2:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31:threads=2 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=1280:720 -o %2 1>> %3pass2.txt
del "%3-0.log"
del *.stat
:: Check for zero or small size avi file
for %%a in ("%~dp1%~n1.avi") do if %%~za GTR 400000000 goto done
echo %date% %time% %1 %3 Conversion failed >> runinst.txt
del "%~dp1%~n1.avi"
:done
IF EXIST "%~dp1%~n1.avi" DEL %1
EXIT

Hint of the day: You can get the filename and path by selecting the video file in explorer and then right click selecting properties. Select security tab and highlight the path and filename, copy this. You can paste it into a cmd box for testing conversion batch files. Make sure you put quotes around the path.

[luttrell1962]

First bug is that GB-PVR passed a zero sized ts file, the multconvbat.bat file looped forever waiting for mencoder to create a log file. I put a test in the beginning of the file to check to see if the file exists or is zero size. If so it exits.

Bug #2 I had mencoder fail a couple of conversion with threads=4. I completely removed threads from the code. The website I downloaded it from said that that part was experimental. I think FPS is faster and it uses less cpu time without threads.

Update: I am now logging file conversions to multiconv.log makes more sense than the other filename. You can check this file to see if any conversions fail. I also delete the first output capture file %3pass1.txt ie onepass1.txt. This makes it so I don't have to check the date and time on the file for the status monitor.

I have the status monitor working in a rudimentary mode, using xbasic's console. I am learning about the xbasic gui graphics windows interface, but it will take several days to put something together that looks nice.

Updated files
Multiconvbat.bat

Code:

@echo off
:: Check to make sure the video file exists and is greater than zero else exit
IF NOT EXIST "%1" EXIT
IF "%~z1" EQU 0 EXIT
cd "\program files (x86)\Devnz\GBPVR\Third Party"
:: *****************************************************
:: Determine current run instance by created log file
:: ******************************************************
IF NOT EXIST one-0.log SET logfile=one-0.log
IF NOT EXIST one-0.log GOTO logfiledone
IF NOT EXIST two-0.log SET logfile=two-0.log
IF NOT EXIST two-0.log GOTO logfiledone
IF NOT EXIST three-0.log SET logfile=three-0.log
IF NOT EXIST three-0.log GOTO logfiledone
:logfiledone
multiconvxvid.exe %1 %2
:: *****************************************************
:: Wait for log file to be created
:: ******************************************************
:wlogfile
IF NOT EXIST "%logfile%" GOTO wlogfile
rem
::: *****************************************************
::: Exit if any of the three process's complete
::: ******************************************************
:loop
timeout /T 30
IF NOT EXIST "one-0.log" exit
IF NOT EXIST "two-0.log" exit
IF EXIST "three-0.log" GOTO loop
EXIT

xvid2pass.bat

Code:

rem javaw -jar d:\projectX.old\projectX.jar %1
echo %date% %time% %1 %3 >> multiconv.log
echo %1 > %3pass1.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" %1 -oac mp3lame -ovc xvid -xvidencopts bitrate=3000:pass=1:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=1280:720 -o nul 1>> %3pass1.txt
del %3pass1.txt
echo %1 > %3pass2.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" %1 -oac mp3lame -ovc xvid -xvidencopts bitrate=3000:pass=2:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=1280:720 -o %2 1>> %3pass2.txt
del "%3-0.log"
del *.stat
:: Check for zero or small size avi file
for %%a in ("%~dp1%~n1.avi") do if %%~za GTR 400000000 goto done
echo %date% %time% %1 %3 Conversion failed >> multiconv.log
del "%~dp1%~n1.avi"
:done
IF EXIST "%~dp1%~n1.avi" DEL %1
EXIT

Picture of crude status monitor


When I have the status monitor finished I will upload multiconv.1.zip

[luttrell1962]

I don't know if it's my qam source or the way GB-PVR records TS files but all of a sudden almost all my recordings have the audio off. So I'm back to splitting the file with ProjectX

I had to make some changes to the xvid2pass.bat file also, sometimes the 2 pass log file does not get deleted and my status monitor opens up the txt output files to analyze them so sometimes %3pass1.txt does not get deleted because it is open, I have the batch file test for it and retry deleting.. I was using 1280x720 for output, but it is sort of slow converting. So I changed back to 800x450 and upped the bitrate to 2400. looks pretty good but might up the bitrate to 2600-2800, sometimes faces look a little glitchy

I downloaded the Intel mencoder file because my cpu supports sse2 even though it's a AMD. This increased FPS during encoding by 3-10.

Without threads cpu utilization is only 25% per mencoder process. I tried 2 threads and that seems to increase fps just slightly but uses 27-35% cpu utilization. 4 threads just bombs out mencoder dies, and fps is only about 4 when I was getting 30. So I will probably rewrite the files for 4 instances.

Updated divx2pass.bat file

Code:

javaw -jar \projectX\projectX.jar %1
echo %date% %time% %1 %3 >> multiconv.log
echo %1 > %3pass1.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" "%~dp1%~n1.m2v" -oac copy -ovc xvid -xvidencopts bitrate=2400:pass=1:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=800:450 -o nul 1>> %3pass1.txt
:: Make Sure the file is deleted because sometimes it is open by status monitor
:makesure
del %3pass1.txt
IF EXIST %3pass1.txt GOTO makesure
echo %1 > %3pass2.txt
"D:\Program Files (x86)\Devnz\GBPVR\Third Party\mencoder" "%~dp1%~n1.m2v" -audiofile "%~dp1%~n1.ac3" -oac mp3lame -ovc xvid -xvidencopts bitrate=2400:pass=2:trellis:quant_type=mpeg:qpel:chroma_me:max_bframes=3:vhq=1:min_iquant=1:max_iquant=31:min_pquant=1:max_pquant=31:min_bquant=1:max_bquant=31 -lameopts abr:br=128 -passlogfile "%3-0.log" -ni -ofps 29.97 -vf scale=800:450 -o %2 1>> %3pass2.txt
:: Since the multiconvbat.bat file queries three-0.log sometimes it does not get deleted, make sure it is gone
:makesure2
del %3-0.log
IF EXIST %3-0.log GOTO makesure2
:: Check for zero or small size avi file
for %%a in ("%~dp1%~n1.avi") do if %%~za GTR 400000000 goto done
echo %date% %time% %1 %3 Conversion failed >> multiconv.txt
del "%~dp1%~n1.avi"
:done
del "%~dp1%~n1.m2v"
del "%~dp1%~n1.ac3"
IF EXIST "%~dp1%~n1.avi" DEL %1
EXIT

Edit: I'll rewrite the xbasic program tonight and update multiconvbat.bat file for 4 instances. Will upload and post link multiconvxvid.1.zip

[luttrell1962]

I think all the bugs are worked out now. I've included 4,3,2 instance batch files. On a dual core you should be able to run 2 instances. I've had trouble with the multithreaded version of mencoder. It works but on one file conversion changed from 20fps to 5 mid file, but avi file was fine. I'm going to download the non multithreaded version and try it.

I've included the crude version of the status monitor. To end it you need to run task manager and delete it. Gui version coming soon

Prio is a cool utility that lets you set priority for certain programs in task manager. For example I set javaw (projectx) to low priority since it takes over the disk. It remembers to set priority the next time the program is ran.
Download here v1.99
http://www.prnwatch.com/download.html

multconvxvid0.1.zip is available here
http://www.mediafire.com/?nzqmmyyumrz
See multiconvxvid0.1.txt for instructions

[luttrell1962]

I finished the gui version of the status monitor you can download it here=>statusgui.zip
Picture Running 4 instances. I can convert files on my machine at about 75% of the original recording time @ 800x450. You need multiconvxvid0.1.zip to use it.


I've been playing with Xvid settings and have good quality at fast speeds, but Xvid has a problem with dark backgrounds and blotchyness. I search the internet and found some possible solutions but have not had time to cut a recording and test the setting for quality and speed.