Trying to suppress Chinese Characters in TVxB

[mvallevand]

in hex (I think). It might be that when the sed script takes off the first instance on the line, the remaining characters then appear to our text renderers as extended ascii instead of Chinese. Removing all instances might be the magic required.

sed works character by character, the chinese utf-8 3 byte codes are all 0x80 or greater which is why this in theory will work. If the file is somehow converted to utf-16 or unicode all bets are off

Martin

[jksmurf]

You guys are both AWESOME, thanks so much for the help. I've been struggling with this for a year!

1. Get SED for Windows working. Make sure the above sed command syntax does what is desired.
Try this command

Code:

sed -e "s/[\x00-\x1F\x7F-\xFF]//g" tbv1.html > tvb.html

Done (on the XML, I agree much easier on the xml for my limited capabilities!). And it works using the above from the later post! Strips the Chinese Characters right out. Truly GREAT!

2. Configure TVxb to produce an xmltv file named "temp.xmltv".

Done.

3. Put a file named "prepost" in the \TVxb\bin directory because, per section 2.3.4 of the TVxb manual, "For security reasons a flag file called prepost (with no extension) must be created in the \TVxb\bin folder before the pre or post commands will work. (The file can be empty.) If this file does not exist, then the pre- and post-commands are deactivated."

Done.

4. Put a postcommand in the TVxb ini file like this
postcommand="sed -e s/[\x00-\x1F\x7F-\xFF]/ / temp.xmltv > mylistings.xmltv"
(I am not sure of how to correctly put the quotes in that line, or if/where they are necessary.)

Ok, Done.

5. Let GBPVR (or NPVR) process the "mylistings.xmltv" file as it would normally.

OK, Ran it. Need to play around with a batch file, rather than a command line... I think?

Code:

WARNING: Post-command batch file "xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml" can only run from the C:\Utils\TVXB\TVXB04Test\bin\folder.
.......: Edit the TVxb.ini file and correct the postcommand= item
WARNING: Post-command batch file ""C:\Utils\TVXB\TVXB04Test\bin\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"" was not found.
.......: Post-command did not run.

This what I used:

Code:

postcommand="C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"

Put a line in your epg update batch file to delete the temp.xmltv file if you want to save some hard drive space.

Not too concerned about this it recreates new one every time anwyay.

Thanks guys, 99% there!

k.

[mvallevand]

OK, Ran it. Need to play around with a batch file, rather than a command line... I think?

Code:

WARNING: Post-command batch file "xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml" can only run from the C:\Utils\TVXB\TVXB04Test\bin\folder.
.......: Edit the TVxb.ini file and correct the postcommand= item
WARNING: Post-command batch file ""C:\Utils\TVXB\TVXB04Test\bin\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"" was not found.
.......: Post-command did not run.

I think you are correct that it want a batch file so in C:\Utils\TVXB\TVXB04Test\bin create something like striphigh.bat and create it with just the working command line, update the ini file to use postcommand=striphigh.bat and finally keep your fingers crossed.

Martin

[markbb1]

Need to play around with a batch file, rather than a command line... I think?

This what I used:

Code:

postcommand="C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"

k.

I think you could remove the quotes around "s/[\x00-\x1F\x7F-\xFF]//g" and eliminate the ambiguity that is causing the errors. Obviously the postcommand is being interpreted as two or three independent strings:

"C:\Progra~1\GnuWin32\bin\sed -e "
s/[\x00-\x1F\x7F-\xFF]//g
" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"

Try it with only the outermost quotes.

[markbb1]

sed works character by character, the chinese utf-8 3 byte codes are all 0x80 or greater which is why this in theory will work. If the file is somehow converted to utf-16 or unicode all bets are off

Martin

It can work character by character, but I read in the documentation somewhere that the search and replace functions, by default, only find/replace the first instance in the line. There are ways to make it only do the first, or only the last, or any of the ones in between, or any combination. The "g" is the simpest way to get all of them in the file.

It appeared from the changes caused by your original syntax that it was only removing the first byte from the line, or possibly the first byte from each 3-byte string. Anyway, I am glad the "g" took care of it.

[jksmurf]

Success!!

First off, unfortunately none of the combinations I tried with or without the quotes worked (but thanks for the suggestion!)

Code:

postcommand="C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml"

Code:

postcommand="C:\Progra~1\GnuWin32\bin\sed -e s/[\x00-\x1F\x7F-\xFF]//g xmltvHKBTVTemp.xml > xmltvHKBTV.xml"

Code:

postcommand=C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" xmltvHKBTVTemp.xml > xmltvHKBTV.xml

Code:

postcommand=C:\Progra~1\GnuWin32\bin\sed -e s/[\x00-\x1F\x7F-\xFF]//g xmltvHKBTVTemp.xml > xmltvHKBTV.xml

BUT, the batch file in the bin dir DID work.

striphigh.bat contents (for other future users)

Code:

C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml > C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTV.xml

Code:

Executing: Post-command
Postcommand: C:\Utils\TVXB\TVXB04Test\bin\striphigh.bat
Status: Finished!

Yippee!

As an aside, I am now ironically getting a few oddities like

"()" where there used to be (太子珠寶鐘錶特約 but I'll work it out.
I tried TVxB's left clip on "()" and Substitution="()", but of course the () don't exist until AFTER the SED command and lclip on just ")" takes too much of the title.

Thanks again fellers,

k.

[mvallevand]

You can stack up your sed replacements, try adding -e s/()//g; after the first one.

Martin

[jksmurf]

So striphigh.bat would contain?

Code:

C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml > C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTV[color=red]Temp[/color].xml

C:\Progra~1\GnuWin32\bin\sed -e "[color=red]s/()//g[/color]" C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml > C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTV.xml

k.

[mvallevand]

One line should do it.

Code:

C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" -e "s/()//g" C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml > C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml

Martin

[jksmurf]

Yep that works

Code:

C:\Progra~1\GnuWin32\bin\sed -e "s/[\x00-\x1F\x7F-\xFF]//g" -e "s/()//g" C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTVTemp.xml > C:\Utils\TVXB\XML\TVXB04XML\xmltvHKBTV.xml

(just xmltvHKBTVTemp.xml > xmltvHKBTV.xml though)