MultiSite Incremental XMLTV Epg Grabber

[WG++Maker]

Hi, attached the inis's I use for TVxB, which has worked well so far. The inis have the URL's plus all post process parsing etc that I do on the XML. Took a lot of work!

Your ini's seem to be sites that have good schedules, whereas mine are just local TV websites. Let me know if you need any other info.

k.

Thanks for the sites ... they are certainly the most easy to do (Chinese) but I will try as soon as I have some time ...

WG++Maker . . Jan

[Reddwarf]

There is a remaining issue with vg.no : The site normally lists the Norwegian title on the index page and (if any) the title in the original language on the show detail page. This causes the program to show updates if these two titles are different and , as it is arranged now, the xmltv title to be the one from the detail page. If you don't want it that way , it is easy to change ... if so .. let me know.
wg++maker ... Jan

Yes can you please change it, the way it is now it gives me issues with recurring recordings. Thanks in advance

[WG++Maker]

Hi Reddwarf,

I have updated vg.no
* The title from the detail page is disabled
* Some small changes in the credits

WG++Maker ... Jan

[Turas]

hi,

i'm trying to use the grabber on my windows home server (win2003 server based)
i installed the .net4 client runtimes. but i still get an exception when starting the grabber for the first time.
do i have to install additional runtime-packages?

Code:

C:\Program Files\ServerCare\WebGrab+Plus>"WebGrab+Plus.exe"
    WebGrab+Plus version 1.0.8 Beta -- Jan van Straaten

many thanks to Paul Weterings whose WebGrab was a great inspiration
----------------------------------------------------------------------

File C:\Documents and Settings\All Users\Application Data\Team MediaPortal\Media
Portal TV Server\xmltv\tvguide.xml not found, creating a new one ..

Update requested for - 23 - out of - 23 - channels for 7 day(s)
Update mode - set per individual channel


      i=index  .=same  c=change  g=gab  r=replace  n=new

Das Erste-ard.de updating, using site TVTODAY.DE, mode incremental

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or ass
embly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d5
0a3a' or one of its dependencies. The system cannot find the file specified.
   at WebGrab.Grab.FetchHtmlPage(String url, Boolean index, SiteIni siteini, Str
ing type, Config conf)
   at WebGrab.Program.Main(String[] args)

[WG++Maker]

Hi Turas,

I have no experience with running the program in Server 2003, but see no reason why it shouldn't work.

I assume you used the MSI installer version. If I am right, the changes are that there are some dependancies (.net references that WG++ needs) missing. (I just found out that the MSI installer version doesn't automatically install the missing dependancies, which I assumed).

I think the best you can do is (the idea behind this to force the installation of the necessary .net components by using the ClickOnce installation version) :

1. Uninstall the current version which has these problems (control panel)

Install the other installer version 'ClickOnce' because that does automatically install missing .net dependancies :
2. download http://www.servercare.nl/Docs/WebGrab+Pl...ckOnce.rar and
3. run the setup in the deploy folder (not the install.bat!)
4. If any dependancy is missing it will be installed.
5. If this version is installed properly, a test run is started automatically. If you are convinced all is ok you can stop it at any time.
6. Now you have a choice: You can keep this version and used it ....
.... or if you prefer the other version (MSI), you can uninstall the ClickOnce version (control panel) and install the MSI version again (run setup in the deploy folder, not the install.bat !!). Now all the dependancies should be in place, so it should work ...

Sorry for the troubles .. and .. I hope this works ... WG++Maker ... Jan

[Turas]

hi,

installing one-click version worked - all dependencies are in place now. test-grabbing started as expected.
after uninstalling one-click and re-installing msi version, everything is fine.
thx and br turas

[WG++Maker]

A new version update (1.0.9) is available @ServerCare.nl

It offers for the user:
- Support for 80 TvGuide sites World Wide
- Introduction of the option to load a cookie file for sites that require this.

And for the SiteIni developer:
- A lot effort has gone into improving, extending and bug fixing of the scrub engine, allowing to capture a wider range of TvGuide websites.
- Further completion of the operation calculate, adding time and date calculations.

Supported sites for the following countries (see the full list of supported sites):
Australia, Belgium(2), Brazil, Croatia(2), Czech Republic, Denmark(2), Finland, France(3), Germany(2), Greece, HongKong, Hungary, Iceland(3), Israel, Italy(5), Litauen, Netherlands(6), New Zealand, Norway(2), Poland(3), Portugal(3), Romania(3), Serbia(2), Singapore, Slovakia, Slovenia(3), Republic South Africa(2), Spain(6), Sweden(3), Switserland(2), Turkey, UK(6) and International Networks: DirectTV(North America), OSN (Libanon, Saudi Arabia, North African Countries etc), Discovery, TvGuide.com(N/M/S America and Caribbean), BBC, Australia Network and RadioAustralia.

Download links:
MSI installer package (installs the whole lot, program, all the siteini files, documentation)
SiteIni.Pack (apart from the siteini files it gets you the channellist files for nearly all the available sites)
Full list of supported tvguide sites.
Documentation only.

WG++Maker ... Jan

[WG++Maker]

[12-April-2012] WebGrab+Plus Version update V1.1.0
This version adds an optional postprocessor that will add IMDb data (from IMDb.com) automatically to the xmltv output file. It does that in 4 steps:
1. Select shows from the xmltv input as candidate for postprocessing based on configurable selection settings
2. Tries to match the selected shows with IMDb entries, based on configurable matching settings.
3. Collect IMDb data for matched shows (Data collected : title, starrating, plot, description, review, viewer comments)
4. Merge of the collected IMDb data with the source xmltv data based on configurable allocation and presentation settings.

It also maintains a local mdb datafile that enables to re-use the data without timeconsuming reconnection with IMDb.com for the same show.

Many thanks to Jacon Berggren and Karsten Kugel for the apha testing and all the usefull suggestions for improvements.

Apart from this new postprocessor, the functionality of the EPG is improved with the option to grab from a subdetail html page. This can be used if certain epg data is on a available on a different html page than the two already available (index - and detail page)

[12 April 2012] Site.Ini.Pack10.0
Now containing siteini files for more than 100 Tvguide sites.
New siteini files for : Austria, Denmark, Germany, Hungary, Ireland, Poland, Sweden, UK and Spain (Basque) and updates for 22 siteini's.
Many thanks again to Willy de Wilde for contributing a lot of the new sites and updates.

Visit:
http://www.servercare.nl/Lists/Posts/Vie...aspx?ID=98

Download links:
The program:
http://www.servercare.nl/Docs/WebGrab+PlusV1.1.0MSI.rar
The siteini pack:
http://www.servercare.nl/Docs/SiteIni.Pack.10.rar
The list of all supported tvguide sites:
http://www.servercare.nl/Docs/SiteIni-List-Pack.10.pdf
The documentation:
http://www.servercare.nl/Docs/Documentation.V1.0.9.pdf
A small utility that automatically installs the required .net versions for the msi installer version:
http://www.servercare.nl/Docs/Install%20...encies.rar

WG++Maker ... Jan

[agerdin]

Hi Jan,

The ini file of my grabber has the following code which I think formats the rating 6.2/10 to 3/5 (the grabber tv.nu has the IMDB rating).

Code:

*
* starrating , format 3.5/5
index_starrating.modify {calculate(not "" format=F1)|10 /}
index_starrating.modify {addend(not "")|/5}
*index_starrating.modify {remove|.0}  * if enabled 3.0/5 changes to 3/5
*

How can I change this if I would like to keep the original rating format 6.2/10.

Thanks

\\Agerdin

[WG++Maker]

Hi agerdin,

as tv.nu.ini is now configured, it grabs the starrating only from the index page. This is the one indicated with a number and text like 'bra' or 'minder bra' and not the IMDb starrating that is shown on the detail page. Because most epg data is on the indexpage it was decided not to link to the detail page. As a resukt, the grabbing is very fast but unfortunately it misses the IMDb starrating as a consequence.

I advice to use the latest version of the program V1.1.0. That allows to use a MDB postprocessor that automatically grabs IMDb data and inserts it in the xmltv output file. Apart from IMDb starrating it can add 'original title' 'plot' 'IMDBdescription' 'review' and 'viewer -comment'. All optional and configurable.

As example a possible listing of 'The perfect man' which I just grabbed with V1.1.0 , postprocessor enabled. As you can see it adds the IMDb starrating, but you can also just replace the tv.nu starrating with the IMDb starrating.


<programme start="20120421113000 +0100" stop="20120421133000 +0100" channel="TV3">
<title lang="sv">The perfect man</title>
<desc>The Perfect Man... [plot: Teenager Holly Hamilton is tired of moving every time her single mom Jean has another personal meltdown involving yet another second-rate guy. To distract her mother from her latest bad choice, Holly conceives the perfect plan for the perfect man.. an imaginary secret admirer who will romance Jean and boost her shaky self-esteem. When the virtual relationship takes off, Holly finds herself having to produce the suitor, borrowing her friend's charming and handsome Uncle Ben as the face behind the e-mails, notes and gifts. Holly must resort to increasingly desperate measures to keep the ruse alive and protect her mom's newfound happiness, almost missing the real perfect man when he does come along.]
FILM - AMERIKANSK ROMANTISK KOMEDI FRÅN 2005. Tonårstjejen Holly Hamilton har tröttnat på att flytta varje gång hennes ensamstående mamma Jean gör slut med någon av sina nollor till pojkvänner. För att distrahera modern från hennes senaste dåliga val, hittar Holly på en hemlig beundrare som ska uppvakta Jean och stärka hennes sviktande självförtroende. När den virtuella romansen tar fart måste Holly fixa fram en kavaljer. Hon lånar därför en väninnas stilige farbror - Ben, för att ha ett ansikte bakom alla mail. Men för att hålla fasaden uppe och upprätthålla moderns nyvunna lycka tvingas Holly ta till alltmer desperata åtgärder.
[imdb description: Directed by Mark Rosman. With Hilary Duff, Heather Locklear, Aria Wallace, Chris Noth. Teenager Holly Hamilton is tired of moving every time her single mom Jean has another personal meltdown involving yet another second-rate guy...] (n)</desc>
<credits>
<director>Mark Rosman</director>
<actor>Hilary Duff</actor>
<actor>Heather Locklear</actor>
<actor>Chris Noth</actor>
<actor>Carson Kressley</actor>
<actor>Mike O'Malley</actor>
</credits>
<category lang="sv">film</category>
<date>2005</date>
<star-rating system="tv.nu">
<value>2.0/5</value>
</star-rating>
<star-rating system="imdb">
<value>5.1/10</value>
</star-rating>
</programme>

WG++Maker .. Jan