Who can help testing Comskip with buildin ATSC CC decoding?

[erik]

I have added ATSC CC decoding to Comskip but before releasing I need some people to test.
Who is willing to receive a test version of Comskip, set verbose=10 and run Comskip on some recordings from various channels and see if the CC is decoded correctly?
The log file will contain the full CC transcript, it will also dump the XDS (such as V-Chip) data but not yet decode that data.
It should work on .mpg, .ts or dvr-ms files
Can not be combined with setting skip_b_frames != 0
It also contains test code to output a .srt file.

From the changelog

Changes in 0.79.133
- Added functionality to decode ATSC Close Captioning data. Will work on any file (mpg, ts, dvr-ms) that contains this information.
- Added the ability to generate a .srt file, Set output_srt=1 to enable. This is done by integrating ccextractorwin. Send me feedback on what other output formats of ccextractor are usefull.
- Added the ability to dump the user data stream from the mpeg recording. Set output_data=1 to enable. this will generate a .dmp file that could be compatible with ccextractor

[rob11252]

I am. Tell me what to do. I can test on 7 ATSC channels and am recording to dvr-ms. It's not clear for me how do I get hold of this test version of Comskip.

[Deusxmachina]

Sure. I'm also recording to dvr-ms but could make some test runs with something else if it matters. Hmm, now I wonder if Cyberlink mux not syncing the audio/video properly would also not sync the CC properly.

[erik]

Take build 133
http://www.kaashoek.com/files/comskip79_133.zip

make sure you have in your comskip.ini file
skip_b_frames=0
Then look at the detect_method setting and make sure it contains the 16=closed captions method.
If you don't know how to check set
detect_method=255
but remember to set it back after testing!
and set
verbose=10

Then drag and drop a couple of recordings on comskip.exe extracted from the build 133 zip
Does not matter if its dvr-ms, mpg or ts. If You regular comskip does process the recordings this one should also be able to do so.
If you are using dvr-ms and don't want to wait for comskip to start due to its regular "process during recording" delay you may set
dvrms_live_tv_retries=1
otherwise comskip may wait 300 seconds before it starts to do some real work.

When everything is done look at the generated .log file with the same name as the recording.
Open the file and look at the end for "Closed caption transcript"
Beneath there should be the full transcript of the close caption in the recording.
The transcript should be readable and make sense. If its garbled I need a short part of the recording to do some more tests.
Then look for lines starting with XDS, they come in pairs. The first with numbers and the second letters like
XDS: 1 83 d3 4f ce 49 43 20 58 80 8f
XDS: SONIC_X y
If you see them I like you to copy all the lines with XDS into a mail (or post in this thread)
Then you may have set
output_srt=1
or
output_smi=1
and see if there have been generated .srt or .smi file and see if they have the correct content.
If you do not know what the correct content should be, don't bother, as I also don't know

If there is no CC and no XDS set
output_data=1
and see if a .dmp file gets generated when processing the recording through comskip.
If no .dmp file is generated the recording does not have a user data stream and so no CC or XDS.
If there is a dmp file but no CC or XDS in the log file I like to get the first say 10kBytes of the .dmp file.

[erik]

I forgot to mention that once CC is working you can use the comskip.dictionary file to add show and/or commercial identification words to improve the commercial detection quality.

[Deusxmachina]

One show done, two more in-progress. Figured I'd post some stuff as I go.

First show is "Chappelle Show," which the CC log looks fine, except I have GBPVR set to record five minutes afterwards, which happened to be "Girls Behaving Badly," and everything in that five minutes looks fine except for this:

985) S: 59901 E: 59923 L: 31 I'M A GIRL BEHAVING BADLY.
986) S: 59927 E: 59986 L: 42 BUHQVYNW RATLY.P T/
987) S: 59987 E: 60027 L: 35 I'M A GIRL bEHóVING BADLY.T@P
988) S: 60028 E: 60044 L: 19 BEHAVING$BCDNY>
989) S: 60047 E: 60082 L: 13 @ú YES, ]
990) S: 60083 E: 60085 L: 2 4/
991) S: 60086 E: 60595 L: 63 $@ P@@@ @@@@ @@@@@@@@@@@@@@ @@@@@@@P@@@@@@@@@P@@@P@P@P@P@P
992) S: 60996 E: 61033 L: 58 GIRLS BEHAVING BADLYIS BEGINNING ITS FOURTH SEASON,

The other two shows' CC looks fine. Maybe the above is just from a reception glitch or something strange.

Looks like I'll email the XDS lines. There's a ton of them.

The other two shows had no XDS. Emailed the ones from from this show.

[erik]

In case the CC data in the log file looks wrong.
Did you try and set
output_smi=1
and look at the generated .sami file to see if the ccextractor decoder did a better job?
The absence of XDS data is disappointing.
Good xds data according to the standard would greatly simplify show/commercial classification

[erik]

The XDS data you mailed me contains

XDS: 1 85 68 6d 8f 16 0 0 0 0 0
XDS: hm

XDS: 1 85 68 5d 8f 26 0 0 0 0 0
XDS: h] &

XDS: 1 85 68 6d 8f 16 0 0 0 0 0
XDS: hm

XDS: 1 85 68 cd 8f b6 0 0 0 0 0
XDS: hM 6

XDS: 1 85 is V_Chip data. Can you see if that data is transmitted shortly after show/commercial borders?

[Deusxmachina]

XDS: 1 85 is V_Chip data. Can you see if that data is transmitted shortly after show/commercial borders?

If you mean the "TV14" and similar logo that appear in the corner of the screen for a few seconds, it shows up at the start of the show but not after commercials. But it also does the same on the same show/different episode that was comskipped with a different version of Comskip.

I added srt=1 and smi=1 in. The files from a few shows look like they probably should look. Below is the paste of that glitch spot from the smi file. Everything else looks fine, so must just be an anomaly. That show also took three times as long to Comskip as other shows.

Also emailed you xds from another show that happened to have some.


<SYNC start="2003403"><P class="UNKNOWNCC">
Á <br>
BUHQVYNW RATLY.P <br>
I'M A GIRL <br>
bEHóVING BADLY.T@P T/
</P></SYNC>
<SYNC start="2004603"><P class="UNKNOWNCC">&nbsp</P></SYNC>

<SYNC start="2004704"><P class="UNKNOWNCC">
@ú <br>
YES, ]4/
</P></SYNC>
<SYNC start="2021720"><P class="UNKNOWNCC">&nbsp</P></SYNC>

<SYNC start="2036403"><P class="UNKNOWNCC">
<i>GIRLS BEHAVING BADLYIS</i> <br>
BEGINNING ITS FOURTH SEASON,
</P></SYNC>

[erik]

Build 134 (tomorrow) willl list the frame numbers where the XDS data is found.
I hope that the XDS v-chip data is always transmitted at the start of a show segment.
There is also program length data in the XDS information.
That could also be a very good clue that some parts belong together
Wait for build 134 to do some more testing and sending me an entire log file.

I only received one mail with XDS data from the log file.
Is that correct?