TVGOS epg plugin for US OTA

[pvruser]

Just wondering if any of the developers had the time/patience to write a TVGOS plugin.

if not, what exactly would i need to decode the TVGOS data stream? i did a dump from my CBS station in TSReader Lite and it has the TVG1 & TVG2 streams. i even dumped some data from them but it appears to be non-human readable. is it a proprietary encrypted format or can i decode it easily with some known algorithm?

and of course how can i integrate this as a plugin for GBPVR so that when i run my epg update script it switches to CBS, gets the data dump, optionally converts it to some XML format and loads it into the database?

i'm currently using a customized version of zap2xml perl script and i'd like to avoid the issues that all page scrapers have...

thanks!

[rcy]

Just wondering if any of the developers had the time/patience to write a TVGOS plugin.

if not, what exactly would i need to decode the TVGOS data stream? i did a dump from my CBS station in TSReader Lite and it has the TVG1 & TVG2 streams. i even dumped some data from them but it appears to be non-human readable. is it a proprietary encrypted format or can i decode it easily with some known algorithm?

and of course how can i integrate this as a plugin for GBPVR so that when i run my epg update script it switches to CBS, gets the data dump, optionally converts it to some XML format and loads it into the database?

i'm currently using a customized version of zap2xml perl script and i'd like to avoid the issues that all page scrapers have...

thanks!

I've been looking for the same thing for a couple of weeks now. Just today I found the ATSC standard for the guide data in the ATSC data stream. You can find it here:

http://www.atsc.org/standards/a_65cr1_with_amend_1.pdf

I assume this is what the DTVPal Dvr uses for it's guide data. (it is OTA only.)They talk about it over at AVSForum. Not all areas are transmitting the data as I understand. I'm in the LA area, so KCBS is supposed to be transmitting this data.

Hope this helps!

[whurlston]

They are only required to transmit about 6 hours and most only transmit 6-12. Cable doesn't usually transmit it so it's really not worth the effort to grab it.

[rcy]

They are only required to transmit about 6 hours and most only transmit 6-12. Cable doesn't usually transmit it so it's really not worth the effort to grab it.

I believe you're right about the PSIP data, which I assume is what the ATSC document refers to after scanning it. But for devices like the DTV Pal (OTA Dvr with no monthly fee like TIVO), they receive 7 days of info via TVGOS transmitted via a single local station. (typically the CBS affiliate) This method was used by VCRs, DVD recorders, and the Sony HD Dvr via analog TV Guide service. (TVGOS) There is now a digital version of this being transmitted in certain markets, with all markets to have data transmitted at the changeover. (my understanding)

But I could be wrong. There doesn't seem to be a lot of clear info on the topic.

[rcy]

Seems the two are different, at least according to wikipedia - TVGOS:

"The new service, launched in the United States in 2006, is entirely digital, using the ATSC digital television standard. While this will significantly improve the service, older systems will no longer be able to download listings once analog shutdown is done in early 2009.[dated info] http://www.macrovision.com/dtv/10053.htm According to this thread from Macrovision analog program guides will continue to be available.

It should be noted that the Program and System Information Protocol (PSIP) within the ATSC standard already allows for each station to send out its own EPG information, and in the U.S. the FCC in fact requires stations to do so for OTA (but not cable, unless they are rebroadcasting an OTA station that provides the PSIP data). However, actual implementation is rather spotty, and usually very minimal -- stations often don't have full descriptions (or even correct information) on the current show, much less the next 3 hours required of them, or the 16 days worth of guide information that can be transmitted. Additionally, ATSC tuners are not required to even show the EPG, only the very limited information for the current show on the current channel.

While the TV Guide service also requires software in the TV or other device, and licensing fees or royalties to be paid to Gemstar TV Guide, it also offers a more complete solution for broadcast television, more like digital satellite or digital cable."

But I'd expect that there is no "open" standard that documents how it works if royalties have to be paid for the info. So while the data is probably there, without the standard for translating it, the point is probably moot.

[whurlston]

Ah yes, was talking about the PSIP data, sorry. The TVGOS is something else but it is a proprietary format and you would need to pay the liscensing fees or hack it which will probably be pretty difficult. Far too many alternatives already exist to make the effort worth it.

[pvruser]

yeah i found some comments somewhere about the TVGOS feed containing images and i did a capture on that PID which indeed contains JPG and PNG images. however the stuff between the images looks unreadable, even though there are clear patterns which look like "headers" of some sort. i even found XML in clear text but only to create menus and load up the channel images or ads, nothing containing any show name.

i was hoping someone knows how to grab the EPG text and dates and show IDs since that's what we really need for a working XMLTV file... i'm no expert at file headers (PNG and JFIF are clear text so it's easy to grab them) so if anyone knows what method is used to compress all the real EPG data that would make a fascinating topic

BTW i found this in another place i hope it helps someone to find the key:

The first program listings to be included in the TVGOS data are for Day 1 (today),
followed by Day 2 (tomorrow). The next program listings to be included are Day 8, not Day 3.
Days 3 and 4 are next, followed by Day 7. Finally, Days 5 and 6 are included. [1, 2, 8, 3, 4, 7, 5,
6] While the entire 8 days of program listings may not be included in the TVGOS data
transmission, Days 1, 2, and 8 are always included. This non-chronological method provides
two important functions. First, with Days 1 and 2 always being transmitted, any program
changes to a broadcaster’s schedule are updated. (For example, if a sporting event’s postseason
series ends early, the originally scheduled games will be substituted with alternate
programming.) Secondly, the complete 8-day program listings will always be stored in the
TVGOS system memory. (The last day, Day 8, is always included in the download.)

[whurlston]

Can you capture a full TS stream about a minute or two long and make it available to download? I'd be interested in seeing it. Don't expect anything though, I'm not sure I can take on another project at the moment.

[pvruser]

Can you capture a full TS stream about a minute or two long and make it available to download? I'd be interested in seeing it. Don't expect anything though, I'm not sure I can take on another project at the moment.

a minute or two will not be enough... i have collected a 120 meg file over 12 hours for that particular PID plus a few smaller ones over a few hours. the transfer is very slow and most of the time it's just what appears to be images... i'll see what site i can find for uploading files and send the link.

[pvruser]

I've been looking for the same thing for a couple of weeks now. Just today I found the ATSC standard for the guide data in the ATSC data stream. You can find it here:

http://www.atsc.org/standards/a_65cr1_with_amend_1.pdf

I assume this is what the DTVPal Dvr uses for it's guide data. (it is OTA only.)They talk about it over at AVSForum. Not all areas are transmitting the data as I understand. I'm in the LA area, so KCBS is supposed to be transmitting this data.

Hope this helps!

after skimming that PDF i think they are using Huffman for the text of the EPG part. they have some tables listed in the PDF. however they also mention custom Huffman tables optimized for specific parts of the EPG which can be implemented by vendors...