MultiSite Incremental XMLTV Epg Grabber

[WG++Maker]

I don't know if this is the right spot to introduce this but ....

I have made an advanced /multisite/incremental EPG grabber which is available for some time and in use by a growing number of users.

Some background:
* I live in a rural area on the Canary Islands where the internet connections are just dialup speed (no ADSL or 3G). On occasion I can use a satellite downlink . Lets say the speed varies between 2-50kB/s.
* Under these conditions I have been using several epg grabbers, but it always took 'hours' to just the daily update.
* The above was the reason I decided to make my own epg grabber which limits downloading to an absolute minimum without loosing richness of data. With it, the daily download for 25 channels in full detail takes about 10 minutes (sat downlink) to 1 hour (in dial-up)

Some features:
* Incremental grabbing. It only grabs what is new or what needs updated because it is changed or missing.
* Can grab from multiple tvguide sites in one run. Each site is programmable through a SiteIni file (As of now there are about 50 of them available for nearly all European countries and a few beyond that). These siteinifiles can be made by the user with the help of debugging modes and the documentation. Of course I am prepared to help or make one if requested. Look here for a complete list of supported tvguide sites.
* The output follows the xmltv standard.
* Most xmltv elements are supported, like start and stop times, title (+ original title), sub-title, category, rating, star-rating, credits (director, actor, presenter, writer, composer, producer), (production)date, episode, subtitles, premiere, previously-shown, video aspect and video quality. All of which are just a choice of the user and programmable per channel.
* All kinds of other options, like, days to grab, one-show-update-only and site-index-page-only (less detail but super fast) and xmltv-element-value filtering and modification.
* Written in C# , executable less that 1MB. Tested for XP, Vista and W7. It runs in command mode, no GUI , no drivers to install. FreeWare, Absolutely non commercial, guaranteed free of malware or virus .

For more information WebGrab+Plus
To download the installation package WebGrab+PlusV1.0.7Beta
To download all the siteini files SiteIni.Pack.7
To have a look at the documentation Documentation.V1.0.7Beta.pdf

[Reddwarf]

Very interesting! I'll have to have a closer look when I have a spare moment

[jksmurf]

I've been using TVxB since around 2005, but as you point out development has stopped. It is VERY configurable once you know how, but I'd always be keen to try something else especially if it's quick. There's also a site I can't get past using TVxB and the devleoper doesn't answer emails.

What I do though is use TV Guide sites of the Cable or FTA providers, it's not like a TVGuide site specially made for a country, with neat data ready for parsing, you really have to do a lot of trial and error ways to parse the data, some sites took me a long time to coinfigure...then they changed the format!

If you like, I could send you the tvxb.inis and the site I'd like to grab , and ask you to have a look at some of these to see if your grabber would work on it? Don't want to take too much of your time though.

k.

[Reddwarf]

Would it work on a site like this? I'm attaching the complete webpage.

[WG++Maker]

Would it work on a site like this? I'm attaching the complete webpage.

Sorry for the late response.....

VG.NO was not so difficult to grab! I have a siteini file (connects the site to WebGrab+Plus) ready.
I am still doing some last changes like adding episode and presenter.

If you want to try before I make it public, send me an email with a request (my email address is on the cover of the documentation)

A few examples of the result:

<programme start="20110628200000 +0100" stop="20110628204500 +0100" channel="NRK1">
<title lang="nb">Joanna Lumley: Draumen om Nilen (2) ®</title>
<desc lang="nb">(Joanna Lumley: Jewel in the Nile) Britisk dokumentarserie. I denne serien følger vi den kjente komikeren Joanna Lumley på hennes reise langs Nilen og dets utspring. Vi møter menneskene, den lokale tradisjonen og ting hun fascineres av i området. I de øde ørkenområdene i det nordlige Sudan fins det ingen hoteller. Men for den oppdagelsesreisende britiske komikeren er det andre ting som vekker interesse. Hun møter en mann som har overlevd et krokodilleangrep, og i Karima får hun se Nubias skatter. Nubia er den lite kjente svarte Faraoen fra et av de eldgamle kongedømmene i Afrika. På veien mot Khartoum hilser hun på en nomadefamilie og får vite hvordan livet arter seg for dem.(n)</desc>
<category lang="nb">Dokumentar</category>
</programme>

<programme start="20110628143000 +0100" stop="20110628154500 +0100" channel="NRK2">
<title lang="nb">Dr. Caligaris kabinett</title>
<desc lang="nb">(Das Cabinet des Dr. Caligari) Tysk grøsser fra 1920. Den hemmelighetsfulle dr. Caligari viser frem søvngjengeren Cesare på markedet i en liten by. Caligari har søvngjengeren helt i sin makt, og om nettene blir det begått flere mord i den lille byen.(n)</desc>
<credits>
<director>Robert Wiene.</director>
<actor>Conrad Veidt</actor>
<actor>Werner Krauss</actor>
<actor>Lil Dagover</actor>
<actor>Rudolf Lettinger</actor>
</credits>
<category lang="nb">Film</category>
<date>1920</date>
</programme>

<programme start="20110628013500 +0100" stop="20110628022500 +0100" channel="TV2">
<title lang="nb">Numbers (2) ®</title>
<desc lang="nb">Amerikansk krimserie. Spesialagent Don Eppes har rekruttert sin bror til FBI i håp om at han kan løse kriminalsaker fra et annet perspektiv. Broren Charlie er nemlig et matematisk geni, som ved hjelp av formler og algoritmer løser krimgåter. HD. Don og teamet hans blir bedt om å bistå etter at flere FBI-agenter blir drept i en skuddveksling med bankranere.(n)</desc>
<category lang="nb">Serie</category>
<video>
<quality> HD</quality>
</video>
</programme>

WG++Maker .... Jan

[WG++Maker]

I've been using TVxB since around 2005, but as you point out development has stopped. It is VERY configurable once you know how, but I'd always be keen to try something else especially if it's quick. There's also a site I can't get past using TVxB and the devleoper doesn't answer emails.

What I do though is use TV Guide sites of the Cable or FTA providers, it's not like a TVGuide site specially made for a country, with neat data ready for parsing, you really have to do a lot of trial and error ways to parse the data, some sites took me a long time to coinfigure...then they changed the format!

If you like, I could send you the tvxb.inis and the site I'd like to grab , and ask you to have a look at some of these to see if your grabber would work on it? Don't want to take too much of your time though.

k.

Yes .. please send me the sites you're interested in. I like to know which of them I can grab with WG++


thanks WG++Maker ... Jan

[Reddwarf]

I'v sent a couple of emails but I'm still waiting for a response...

[WG++Maker]

Hi Reddwarf,

sorry , but I am unaware of email from you that still needs response.
Maybe they slipped my attention due to the holiday period.
Can you please resend your last mail?
Again, sorry

WG++Maker ... Jan

[Reddwarf]

Hi Reddwarf,

sorry , but I am unaware of email from you that still needs response.
Maybe they slipped my attention due to the holiday period.
Can you please resend your last mail?
Again, sorry

WG++Maker ... Jan

Ok, I'v sent you a new email

[WG++Maker]

A new version of WebGrab+Plus (V1.0.8 Beta) is available:

It now supports 66 tvguide sites and comes in two installation variants:

-- The standard .msi variant is available here: (http://www.servercare.nl/Docs/WebGrab+Pl...etaMsi.rar)
-- and the ClickOnce variant here: (http://www.servercare.nl/Docs/WebGrab+Pl...ckOnce.rar)
(the ClickOnce installation was use for all the previous versions and is kept available for those user that want to continue using utilities or batch scripts that rely on the file structure that is typical for this type of installation)

Supported sites for the following countries :

Belgium(2), Brazil, Croatia, Czech Republic, Denmark(2), Finland, France(3), Greece, Germany(2), HongKong, Hungary, Israel, Italy(5), Litauen, Netherlands(5), Norway(2), Poland(2), Portugal(2), Romania(3), Serbia, Singapore, Slovakia, Republic South Africa(2), Spain(4), Sweden(3), Switserland(2), Turkey, UK(6) and
International Networks like: DirectTV(North America), OSN (Libanon, Saudi Arabia, North African Countries etc) Discovery, TvGuide.com (N/M/S-America and Caribbean), BBC, Australia Network and RadioAustralia

A complete list of supported tvguide sites can be had at: (http://www.servercare.nl/Docs/SiteIni-List-Pack8.pdf)

For detailed information for users and siteini developers read the documentation: (http://www.servercare.nl/Docs/Documentat....8Beta.pdf)

For support, siteini request and more information (http://www.servercare.nl/Lists/Posts/Pos...a371&ID=98)
or just mail (my mail address is on the cover of the documentation)

Wg++Maker .... Jan