New XMLTV Enhancer and more...

[zehd]

In WizWebHelper.DLL there is an ImdbScraper and TheTvDbScraper function that will retrieve show information when passed the show title and episode.

It's still a work in progress and may not retrieve all the info you are after, but if you decide to use let me know what's missing and I will add.

Attached is the help file for the Wiz DLLs, refer to the WizWebHelper namespace section.

I was able to see how you queried the tvdb.com for episodic series in your C source. Wrote my VB6 module for that...

I wonder how to get movie information. Maybe I need to query imdb for that...

[JavaWiz]

I was able to see how you queried the tvdb.com for episodic series in your C source. Wrote my VB6 module for that...

I wonder how to get movie information. Maybe I need to query imdb for that...

Yes, basically that's what I do. IMDB or EPGuide for movie info, TheTvDb for Tv series/episodes.

[zehd]

Yes, basically that's what I do. IMDB or EPGuide for movie info, TheTvDb for Tv series/episodes.

I'm having a hard time getting the URL that you construct to find and exact match for a movie title with imdb. Looking at your code, I see URLs that get a close match, but list several alternatives.

I realize that you're using the HTMLAgility dll to help get through the scraped html what do you search for on the page?

[JavaWiz]

I'm having a hard time getting the URL that you construct to find and exact match for a movie title with imdb. Looking at your code, I see URLs that get a close match, but list several alternatives.

I realize that you're using the HTMLAgility dll to help get through the scraped html what do you search for on the page?

In SearchForMovie():

• Construct base movie name (remove tokens surrounded by [] () etc (ie filterMovieName())
  
• Determine search type (ie Title, Also Known As AKA or Episode)
  
• Construct URL based on type:
  Code:

  [SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].TITLE:[/SIZE]
[SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
[SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
[SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].AKA:[/SIZE]
[SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;site=aka;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
[SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
[SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].EPISODE:[/SIZE]
[SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;ttype=ep;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
[SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
• Make Http call with target URL
  
• Check if it is a Detail page (ie exact hit) XPath query looking for h1 sections
  Code:

  [SIZE=1][COLOR=#0000ff]private[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]bool[/COLOR][/SIZE][SIZE=1] isDetailPage([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1] html)[/SIZE]
[SIZE=1]{[/SIZE]
[SIZE=1][COLOR=#2b91af]HtmlNodeCollection[/COLOR][/SIZE][SIZE=1] nodes = ProcessXPathQuery(html, [/SIZE][SIZE=1][COLOR=#a31515]"//h1"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] (nodes == [/SIZE][SIZE=1][COLOR=#0000ff]null[/COLOR][/SIZE][SIZE=1])[/SIZE]
[SIZE=1][COLOR=#0000ff]return[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]false[/COLOR][/SIZE][SIZE=1];[/SIZE]
[SIZE=1][COLOR=#0000ff]else[/COLOR][/SIZE]
[SIZE=1][COLOR=#0000ff]return[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]true[/COLOR][/SIZE][SIZE=1];[/SIZE]
[SIZE=1]}[/SIZE]
• If so process html and retrieve values and place in MovieInfo object
  
• If not check if it is a list
  Code:

  [SIZE=1]exactMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Exact Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1]partialMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Partial Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1]approxMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Approx Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1]popularMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Popular Titles']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
  If so, if popular or exact match, grab detail URL from ArrayList (detailURL)
  Code:

  [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].popularMatches.Count == 1)[/SIZE]
[SIZE=1]{[/SIZE]
[SIZE=1][COLOR=#008000]// title|url[/COLOR][/SIZE]
[SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].LoggingActive)[/SIZE]
[SIZE=1]StatusNotification([/SIZE][SIZE=1][COLOR=#a31515]"SearchForMovie()-Looks like we have a POPULAR match"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1]detailURL = parseUrlFromTitleUrlEntry(([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1])popularMatches[0]);[/SIZE]
[SIZE=1]}[/SIZE]
[SIZE=1][COLOR=#0000ff]else[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].exactMatches.Count == 1)[/SIZE]
[SIZE=1]{[/SIZE]
[SIZE=1][COLOR=#008000]// title|url[/COLOR][/SIZE]
[SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].LoggingActive)[/SIZE]
[SIZE=1]StatusNotification([/SIZE][SIZE=1][COLOR=#a31515]"SearchForMovie()-Looks like we have an EXACT match"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
[SIZE=1]detailURL = parseUrlFromTitleUrlEntry(([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1])exactMatches[0]);[/SIZE]
[SIZE=1]}[/SIZE]
• If detailURL is populated, attempt to retrieve movie info from that URL
  Code:

  [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#008000]/*!stopProcessing && */[/COLOR][/SIZE][SIZE=1] detailURL != [/SIZE][SIZE=1][COLOR=#0000ff]null[/COLOR][/SIZE][SIZE=1])[/SIZE]
[SIZE=1]RetrieveMovieByURL(detailURL);[/SIZE]
[SIZE=1][COLOR=#008000]//ProcessDetailHtml(detailBuffer);[/COLOR][/SIZE]
• If movieInfo object is populated, then you've got a hit, if not, check the ArrayLists, those are possible matches (ie ExactMatches, PopulareMatches, PartialMatches, Approx Matches).

If I missed the point, let me know and I'll try to explain further.

[zehd]

In SearchForMovie():

Thank you. Your explanation makes it much clearer for this VB6 guy...

I'll see what I can do.

[JavaWiz]

Thank you. Your explanation makes it much clearer for this VB6 guy...

I'll see what I can do.

I know where your comming from. I was a heavy VB5/6 programmer several years back.

To get a handle on the XPath syntax I went thru the following tutorial... http://www.w3schools.com/xpath/xpath_intro.asp

Additionally, attached is a little tool (XPathTester) I wrote to help figure out the correct XPath query commands for parsing html pages (ie IMDB and TheTvDb). It wasn't meant to be production ready, so you may find some quirks with it. If you need the source, let me know...

• Input file: Local html file or URL to be parsed
  
• Query: Saved query file or manual query entry to be executed
  
• Show Children: Shows parent AND children nodes
  
• Wrap Text: says it all...

[zehd]

I know where your comming from. I was a heavy VB5/6 programmer several years back.

To get a handle on the XPath syntax I went thru the following tutorial... http://www.w3schools.com/xpath/xpath_intro.asp

Additionally, attached is a little tool (XPathTester) I wrote to help figure out the correct XPath query commands for parsing html pages (ie IMDB and TheTvDb). It wasn't meant to be production ready, so you may find some quirks with it. If you need the source, let me know...

• Input file: Local html file or URL to be parsed
  
• Query: Saved query file or manual query entry to be executed
  
• Show Children: Shows parent AND children nodes
  
• Wrap Text: says it all...

Well I have to admit that I have tried really hard to work with xml parsing in vb6, and I found my old approach to parse line by line with file i/o to be much faster on the really large xmltv files. And the set series module uses that approach. So I'm going to try the same with the Get Movie module.

If it just isn't working I'll let you know...

[zehd]

Here's the latest Enhancer.... (snip)

Latest update to software is found at:

http://gbpvr.com/pmwiki/pmwiki.php/Utility/UltraXMLTV

[agerdin]

You can now choose what order for the Description Append tags. Each tag is separated by a space. It's optional, you can remove them if you want. Some of the tags already include (), but most do not. You can add your own punctuation here. You can also add Carriage Returns. Don't break or modify the tags and be sure you leave the square brackets. If you get messed up, you can reset the box and start over.

Zehd,

All looks good and I really like the ability to change the order, add your own literals and add carriage returns. However, since I am adding literals and carriage returns they still appear even for shows that does not have that tag. What I want is to ignore the literal and/or carriage return if the tag is not found for that show. I also would like the option to remove the () from the tag <date>. See some Ultra-XMLTV Enhancer generated examples below.

Thanks for a cool app.

\\Agerdin

Code:

<programme start="20071215024000 +0100" stop="20071215033500 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>CSI</title>
    <desc>NÃ¥gra fÃ¥gelskÃ¥dare hittar en korp med ett människoöga i munnen och spÃ¥ren leder till en frÃ¥nskild man och hans tvÃ¥ barn.


Actors: William Petersen, Marg Helgenberger, Gary Dourdan, George Eads, Jorja Fox, Paul Guilfoyle.
År: (2002)
Aspect:
Avsnitt: Del 12 säsong 3 </desc>
    <duration units="minutes">55 min</duration>
    <episode-num system="xmltv_ns">2 . 11 .</episode-num>
    <category>series</category>
    <category>Drama</category>
  </programme>

Code:

<programme start="20071215120000 +0100" stop="20071215133000 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>Spirit - Hästen frÃ¥n vildmarken</title>
    <desc>Amerikanskt animerat familjeäventyr frÃ¥n 2002. Handling: Spirit är en vild ung mustang som beger sig ut pÃ¥ en dramatisk och actionfylld resa för att utföra det omöjliga, nämligen att rädda sitt hemland och Ã¥terfÃ¥ sin frihet. PÃ¥ sin farofyllda och spännande resa genom Amerikas majestätiska vildmark finner Spirit oväntat en vän i en ung Lakotakrigare. Original...

Directed by: Lorna Cook.

År: (2002)
Aspect:
Avsnitt: </desc>
    <duration units="minutes">90 min</duration>
    <category>movie</category>
    <category>Movies</category>
  </programme>

Code:

<programme start="20071210011000 +0100" stop="20071210060000 +0100" channel="svt1.svt.se" clumpidx="0/1">
    <title>Sändningar frÃ¥n SVT24</title>
    <desc>



År:
Aspect: 4:3
Avsnitt: </desc>
    <duration units="minutes">290 min</duration>
  </programme>

[zehd]

Zehd,

All looks good and I really like the ability to change the order, add your own literals and add carriage returns. However, since I am adding literals and carriage returns they still appear even for shows that does not have that tag. What I want is to ignore the literal and/or carriage return if the tag is not found for that show. I also would like the option to remove the () from the tag <date>. See some Ultra-XMLTV Enhancer generated examples below.

Thanks for a cool app.

\\Agerdin

Code:

<programme start="20071215024000 +0100" stop="20071215033500 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>CSI</title>
    <desc>NÃ¥gra fÃ¥gelskÃ¥dare hittar en korp med ett människoöga i munnen och spÃ¥ren leder till en frÃ¥nskild man och hans tvÃ¥ barn.


Actors: William Petersen, Marg Helgenberger, Gary Dourdan, George Eads, Jorja Fox, Paul Guilfoyle.
År: (2002)
Aspect:
Avsnitt: Del 12 säsong 3 </desc>
    <duration units="minutes">55 min</duration>
    <episode-num system="xmltv_ns">2 . 11 .</episode-num>
    <category>series</category>
    <category>Drama</category>
  </programme>

Code:

<programme start="20071215120000 +0100" stop="20071215133000 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>Spirit - Hästen frÃ¥n vildmarken</title>
    <desc>Amerikanskt animerat familjeäventyr frÃ¥n 2002. Handling: Spirit är en vild ung mustang som beger sig ut pÃ¥ en dramatisk och actionfylld resa för att utföra det omöjliga, nämligen att rädda sitt hemland och Ã¥terfÃ¥ sin frihet. PÃ¥ sin farofyllda och spännande resa genom Amerikas majestätiska vildmark finner Spirit oväntat en vän i en ung Lakotakrigare. Original...

Directed by: Lorna Cook.

År: (2002)
Aspect:
Avsnitt: </desc>
    <duration units="minutes">90 min</duration>
    <category>movie</category>
    <category>Movies</category>
  </programme>

Code:

<programme start="20071210011000 +0100" stop="20071210060000 +0100" channel="svt1.svt.se" clumpidx="0/1">
    <title>Sändningar frÃ¥n SVT24</title>
    <desc>



År:
Aspect: 4:3
Avsnitt: </desc>
    <duration units="minutes">290 min</duration>
  </programme>

I will remove all of the () and add the option to include the duration type and other labels. I will remove all of the modifiers and just add a couple to the Description string. (more for demonstration purposes. A person can edit the string and take out what they don't like... It will be about a week, or a bit more.