Zap2it Closing Down

[cpa]

Its fine, post the name.

Dam you be fast! ctpvr

This is without a doubt the most responsive community I've seen.

[cpa]

If you find away to extract that data into the common xmltv file format, then you'll be able to use it in GB-PVR

Where could I get a sample of a common xmltv format? The data is in a access file. I should be able to export using access right?

[sub]

I would gamble that any PVR app you bought for three dollars is one of the culprits that is violating Zap2It's non-commercial licensing mentioned on their website.
http://docs.tms.tribune.com/tech/tmsdata...etter.html

I could not tell you. How would I find out if that is the case?

They were in violation of the zap2it license for a long time, but I remember they went legit late last year. They dont have digital channel listings at this stage though.

[sub]

Where could I get a sample of a common xmltv format? The data is in a access file. I should be able to export using access right?

Its a pretty simple file format: http://xmltv.org/wiki/xmltvfileformat.html

[cpa]

Its a pretty simple file format: http://xmltv.org/wiki/xmltvfileformat.html

Much thanks, I'll be baaack if I getting to work :confused:

[pwtenny]

I'd like to note, if this hasn't been discussed already, that scraping web sources is problematic if program descriptions (and you care about them) aren't displayed on the same page as the show grid (Titan: no, zap2it: no, TV Guide, no). My typical lineup of less than 130 channels for 14 days gives me a little over 31,000 programs. If you get a grid that displays all your channels for 6 hours, you'll need 56 HTTP requests to get the grid and another 31,000 - one for every program to get its description.

If you can get each description in exactly one second, which I don't think is realistic, you'll be looking at 8.5 hours minimum and enough requests to get you banned from the server and possibly threatened legally. Even if it can be done in 1/3rd the time, that's still nearly three hours of hammering that webserver (even faster) which is even more likely to get you yelled at.

I for one don't generally care about descriptions enough for this to be a problem, but for those that do care, it's not going to happen.

As for schedule scraping, I got half way through writing a perl scraper for zap2it's current public interface in just over an hour, outputting pseudo-XMLTV compatible xml. Don't worry too hard, that will come out soon from probably a dozen different people or more, it's just grunt work.

I don't see getting more than that anytime soon though. This well and truly sucks.

[sub]

I'd like to note, if this hasn't been discussed already, that scraping web sources is problematic if program descriptions (and you care about them) aren't displayed on the same page as the show grid (Titan: no, zap2it: no, TV Guide, no). My typical lineup of less than 130 channels for 14 days gives me a little over 31,000 programs. If you get a grid that displays all your channels for 6 hours, you'll need 56 HTTP requests to get the grid and another 31,000 - one for every program to get its description.

If you can get each description in exactly one second, which I don't think is realistic, you'll be looking at 8.5 hours minimum and enough requests to get you banned from the server and possibly threatened legally. Even if it can be done in 1/3rd the time, that's still nearly three hours of hammering that webserver (even faster) which is even more likely to get you yelled at.

I for one don't generally care about descriptions enough for this to be a problem, but for those that do care, it's not going to happen.

As for schedule scraping, I got half way through writing a perl scraper for zap2it's current public interface in just over an hour, outputting pseudo-XMLTV compatible xml. Don't worry too hard, that will come out soon from probably a dozen different people or more, it's just grunt work.

This is pretty much what I said a few pages back. I even used 30,000 as an example of numbers of web pages hits and how I got banned for doing exactly this. After the initial load you could drop back to scrapping just the next 36hrs or so of listings, which drops the number down quite a bit - but its still too high. This is why I was suggesting other options for propagating the info in chunks via usenet or p2p, though this probably steps over a legal line (though with all the people willing to share movies and music online, sharing a few TV listings might not be a concern to many).

[betlit]

let me get this straight - the effort here is to write some kind of parser that will parse web page(s) into XML info of EPG?
But:
1) What about the source provider "not liking" this (see 2)
2) What happens when the source provider decides to change the format/look/etc of the pages?

this two points are the main problems of most xmltv grabbers. so far i wrote three grabbers (tv_grab_ch, tv_grab_ch_bluewin, tv_grab_ch_search). two of them got blocked by the source provider after a couple of months. tv_grab_ch_search (http://tv.search.ch) seems to tolerate it (as e.g. tvtoday.de does since years).

a change of the layout usually means a change of the grabber. as most authors of the grabbers use them for themselves, things like that are spotted and taken care of fast.

[dragonfiregum]

I assume a xmltv grabber cant draws data from the cable provider like a cable box loads its own epg?
Or is it possible to draw straight from your cable provider? i mean my cable box knows the schedule somehow right?
I understand this is probably encrypted... but if you are paying for cable (which you obviously should be if you watching those channels ) that data comes with your subscription to cable. Pulling it for tv watching/personal use should be legal as long as your not redistributing it. Is there any ways to draw data from the line then?
If not maybe a shot in the dark but yahoo seems intrested in tivo. Maybe they would offer a monthly fee or something for data. But i have no idea how to contact them as major companys like that dont seem to like to include contact info anywhere.

[pwtenny]

This is pretty much what I said a few pages back. I even used 30,000 as an example of numbers of web pages hits and how I got banned for doing exactly this. After the initial load you could drop back to scrapping just the next 36hrs or so of listings, which drops the number down quite a bit - but its still too high. This is why I was suggesting other options for propagating the info in chunks via usenet or p2p, though this probably steps over a legal line (though with all the people willing to share movies and music online, sharing a few TV listings might not be a concern to many).

Yeah, I got my number wrong though.

Downloading EPG data from Tribune...
Downloaded 17258kB in 308 seconds.
(re)processing EPG data.
MESSAGE: Your subscription will expire: 2007-09-06T07:57:54Z
(re)processed 36176 programs.

Since the only way to get data into GBPVR now is through XMLTV files, I have a few questions relating to that. XMLTV's format seems to consist of the two following things:

<channel id="I10021.labs.zap2it.com">
<display-name>254 AMC</display-name>
<display-name>254 AMC DITV518:-</display-name>
<display-name>254</display-name>
<display-name>AMC</display-name>
<display-name>AMC</display-name>
<display-name>Satellite</display-name>
</channel>

For a channel, and..

<programme start="20070616013000 " stop="20070616020000 " channel="I30017.labs.zap2it.com">
<title lang="en">Nets Magazine</title>
<desc lang="en">Highlight show of the New Jersey Nets; interviews with players and coaches.</desc>
<category lang="en">Sports talk</category>
<category lang="en">Basketball</category>
<category lang="en">Newsmagazine</category>
<category lang="en">Limited Series</category>
<episode-num system="dd_progid">SH539087.0000</episode-num>
<audio>
<stereo>stereo</stereo>
</audio>
<subtitles type="teletext" />
</programme>

..for a program. Which elements from this are required for GBPVR to be able to parse the file? There isn't much you can get from scraping beyond a single display-name for a channel and title+category+start/end+channel for a program. Even though Tribune isn't due to shut off service for another two months, I plan to have a zap2it scraper done by the weekend so that I don't have to worry about it.