NextPVR Forums
  • ______
  • Home
  • New Posts
  • Wiki
  • Members
  • Help
  • Search
  • Register
  • Login
  • Home
  • Wiki
  • Members
  • Help
  • Search
NextPVR Forums Public Developers v
« Previous 1 … 34 35 36 37 38 … 93 Next »
New XMLTV Enhancer and more...

Thread Closed 
 
  • 0 Vote(s) - 0 Average
New XMLTV Enhancer and more...
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#71
2007-12-08, 02:56 PM
JavaWiz Wrote:In WizWebHelper.DLL there is an ImdbScraper and TheTvDbScraper function that will retrieve show information when passed the show title and episode.

It's still a work in progress and may not retrieve all the info you are after, but if you decide to use let me know what's missing and I will add.

Attached is the help file for the Wiz DLLs, refer to the WizWebHelper namespace section.

I was able to see how you queried the tvdb.com for episodic series in your C source. Wrote my VB6 module for that...

I wonder how to get movie information. Maybe I need to query imdb for that...
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
JavaWiz
Offline

Posting Freak

Jacksonville, FL. USA
Posts: 2,522
Threads: 141
Joined: Dec 2006
#72
2007-12-08, 03:38 PM
zehd Wrote:I was able to see how you queried the tvdb.com for episodic series in your C source. Wrote my VB6 module for that...

I wonder how to get movie information. Maybe I need to query imdb for that...
Yes, basically that's what I do. IMDB or EPGuide for movie info, TheTvDb for Tv series/episodes.
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#73
2007-12-08, 04:57 PM
JavaWiz Wrote:Yes, basically that's what I do. IMDB or EPGuide for movie info, TheTvDb for Tv series/episodes.

I'm having a hard time getting the URL that you construct to find and exact match for a movie title with imdb. Looking at your code, I see URLs that get a close match, but list several alternatives.

I realize that you're using the HTMLAgility dll to help get through the scraped html what do you search for on the page?
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
JavaWiz
Offline

Posting Freak

Jacksonville, FL. USA
Posts: 2,522
Threads: 141
Joined: Dec 2006
#74
2007-12-08, 09:03 PM
zehd Wrote:I'm having a hard time getting the URL that you construct to find and exact match for a movie title with imdb. Looking at your code, I see URLs that get a close match, but list several alternatives.

I realize that you're using the HTMLAgility dll to help get through the scraped html what do you search for on the page?
In SearchForMovie():
  • Construct base movie name (remove tokens surrounded by [] () etc (ie filterMovieName())
  • Determine search type (ie Title, Also Known As AKA or Episode)
  • Construct URL based on type:
    Code:
    [SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].TITLE:[/SIZE]
    [SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
    [SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
    [SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].AKA:[/SIZE]
    [SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;site=aka;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
    [SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
    [SIZE=1][COLOR=#0000ff]case[/COLOR][/SIZE][SIZE=1][COLOR=#2b91af]SEARCH_TYPE[/COLOR][/SIZE][SIZE=1].EPISODE:[/SIZE]
    [SIZE=1]targetURL = scIMDB_BASEURL + [/SIZE][SIZE=1][COLOR=#a31515]"/find?s=tt;ttype=ep;q="[/COLOR][/SIZE][SIZE=1] + [/SIZE][SIZE=1][COLOR=#2b91af]HttpUtility[/COLOR][/SIZE][SIZE=1].UrlEncode(baseMovieName);[/SIZE]
    [SIZE=1][COLOR=#0000ff]break[/COLOR][/SIZE][SIZE=1];[/SIZE]
  • Make Http call with target URL
  • Check if it is a Detail page (ie exact hit) XPath query looking for h1 sections
    Code:
    [SIZE=1][COLOR=#0000ff]private[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]bool[/COLOR][/SIZE][SIZE=1] isDetailPage([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1] html)[/SIZE]
    [SIZE=1]{[/SIZE]
    [SIZE=1][COLOR=#2b91af]HtmlNodeCollection[/COLOR][/SIZE][SIZE=1] nodes = ProcessXPathQuery(html, [/SIZE][SIZE=1][COLOR=#a31515]"//h1"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] (nodes == [/SIZE][SIZE=1][COLOR=#0000ff]null[/COLOR][/SIZE][SIZE=1])[/SIZE]
    [SIZE=1][COLOR=#0000ff]return[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]false[/COLOR][/SIZE][SIZE=1];[/SIZE]
    [SIZE=1][COLOR=#0000ff]else[/COLOR][/SIZE]
    [SIZE=1][COLOR=#0000ff]return[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]true[/COLOR][/SIZE][SIZE=1];[/SIZE]
    [SIZE=1]}[/SIZE]
  • If so process html and retrieve values and place in MovieInfo object
  • If not check if it is a list
    Code:
    [SIZE=1]exactMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Exact Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1]partialMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Partial Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1]approxMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Titles (Approx Matches)']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1]popularMatches = GetTitleAndLinksInSection(html, [/SIZE][SIZE=1][COLOR=#a31515]"//p[b='Popular Titles']"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    If so, if popular or exact match, grab detail URL from ArrayList (detailURL)
    Code:
    [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].popularMatches.Count == 1)[/SIZE]
    [SIZE=1]{[/SIZE]
    [SIZE=1][COLOR=#008000]// title|url[/COLOR][/SIZE]
    [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].LoggingActive)[/SIZE]
    [SIZE=1]StatusNotification([/SIZE][SIZE=1][COLOR=#a31515]"SearchForMovie()-Looks like we have a POPULAR match"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1]detailURL = parseUrlFromTitleUrlEntry(([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1])popularMatches[0]);[/SIZE]
    [SIZE=1]}[/SIZE]
    [SIZE=1][COLOR=#0000ff]else[/COLOR][/SIZE][SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].exactMatches.Count == 1)[/SIZE]
    [SIZE=1]{[/SIZE]
    [SIZE=1][COLOR=#008000]// title|url[/COLOR][/SIZE]
    [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#0000ff]this[/COLOR][/SIZE][SIZE=1].LoggingActive)[/SIZE]
    [SIZE=1]StatusNotification([/SIZE][SIZE=1][COLOR=#a31515]"SearchForMovie()-Looks like we have an EXACT match"[/COLOR][/SIZE][SIZE=1]);[/SIZE]
    [SIZE=1]detailURL = parseUrlFromTitleUrlEntry(([/SIZE][SIZE=1][COLOR=#0000ff]string[/COLOR][/SIZE][SIZE=1])exactMatches[0]);[/SIZE]
    [SIZE=1]}[/SIZE]
  • If detailURL is populated, attempt to retrieve movie info from that URL
    Code:
    [SIZE=1][COLOR=#0000ff]if[/COLOR][/SIZE][SIZE=1] ([/SIZE][SIZE=1][COLOR=#008000]/*!stopProcessing && */[/COLOR][/SIZE][SIZE=1] detailURL != [/SIZE][SIZE=1][COLOR=#0000ff]null[/COLOR][/SIZE][SIZE=1])[/SIZE]
    [SIZE=1]RetrieveMovieByURL(detailURL);[/SIZE]
    [SIZE=1][COLOR=#008000]//ProcessDetailHtml(detailBuffer);[/COLOR][/SIZE]
  • If movieInfo object is populated, then you've got a hit, if not, check the ArrayLists, those are possible matches (ie ExactMatches, PopulareMatches, PartialMatches, Approx Matches).
If I missed the point, let me know and I'll try to explain further.
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#75
2007-12-09, 02:23 AM
JavaWiz Wrote:In SearchForMovie():

Thank you. Your explanation makes it much clearer for this VB6 guy...

I'll see what I can do.
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
JavaWiz
Offline

Posting Freak

Jacksonville, FL. USA
Posts: 2,522
Threads: 141
Joined: Dec 2006
#76
2007-12-09, 02:34 AM
zehd Wrote:Thank you. Your explanation makes it much clearer for this VB6 guy...

I'll see what I can do.
I know where your comming from. I was a heavy VB5/6 programmer several years back.

To get a handle on the XPath syntax I went thru the following tutorial... http://www.w3schools.com/xpath/xpath_intro.asp

Additionally, attached is a little tool (XPathTester) I wrote to help figure out the correct XPath query commands for parsing html pages (ie IMDB and TheTvDb). It wasn't meant to be production ready, so you may find some quirks with it. If you need the source, let me know...
  • Input file: Local html file or URL to be parsed
  • Query: Saved query file or manual query entry to be executed
  • Show Children: Shows parent AND children nodes
  • Wrap Text: says it all...
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#77
2007-12-09, 02:45 AM
JavaWiz Wrote:I know where your comming from. I was a heavy VB5/6 programmer several years back.

To get a handle on the XPath syntax I went thru the following tutorial... http://www.w3schools.com/xpath/xpath_intro.asp

Additionally, attached is a little tool (XPathTester) I wrote to help figure out the correct XPath query commands for parsing html pages (ie IMDB and TheTvDb). It wasn't meant to be production ready, so you may find some quirks with it. If you need the source, let me know...
  • Input file: Local html file or URL to be parsed
  • Query: Saved query file or manual query entry to be executed
  • Show Children: Shows parent AND children nodes
  • Wrap Text: says it all...

Well I have to admit that I have tried really hard to work with xml parsing in vb6, and I found my old approach to parse line by line with file i/o to be much faster on the really large xmltv files. And the set series module uses that approach. So I'm going to try the same with the Get Movie module.

If it just isn't working I'll let you know...
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#78
2007-12-09, 07:47 AM (This post was last modified: 2008-01-08, 01:11 AM by zehd.)
Here's the latest Enhancer.... (snip)

Latest update to software is found at:

http://gbpvr.com/pmwiki/pmwiki.php/Utility/UltraXMLTV
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
agerdin
Offline

Senior Member
Posts: 692
Threads: 96
Joined: Sep 2004
#79
2007-12-10, 11:27 AM
zehd Wrote:You can now choose what order for the Description Append tags. Each tag is separated by a space. It's optional, you can remove them if you want. Some of the tags already include (), but most do not. You can add your own punctuation here. You can also add Carriage Returns. Don't break or modify the tags and be sure you leave the square brackets. If you get messed up, you can reset the box and start over.

Zehd,

All looks good and I really like the ability to change the order, add your own literals and add carriage returns. However, since I am adding literals and carriage returns they still appear even for shows that does not have that tag. What I want is to ignore the literal and/or carriage return if the tag is not found for that show. I also would like the option to remove the () from the tag <date>. See some Ultra-XMLTV Enhancer generated examples below.

Thanks for a cool app.

\\Agerdin

Code:
<programme start="20071215024000 +0100" stop="20071215033500 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>CSI</title>
    <desc>NÃ¥gra fÃ¥gelskÃ¥dare hittar en korp med ett människoöga i munnen och spÃ¥ren leder till en frÃ¥nskild man och hans tvÃ¥ barn.


Actors: William Petersen, Marg Helgenberger, Gary Dourdan, George Eads, Jorja Fox, Paul Guilfoyle.
År: (2002)
Aspect:
Avsnitt: Del 12 säsong 3 </desc>
    <duration units="minutes">55 min</duration>
    <episode-num system="xmltv_ns">2 . 11 .</episode-num>
    <category>series</category>
    <category>Drama</category>
  </programme>

Code:
<programme start="20071215120000 +0100" stop="20071215133000 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>Spirit - Hästen frÃ¥n vildmarken</title>
    <desc>Amerikanskt animerat familjeäventyr frÃ¥n 2002. Handling: Spirit är en vild ung mustang som beger sig ut pÃ¥ en dramatisk och actionfylld resa för att utföra det omöjliga, nämligen att rädda sitt hemland och Ã¥terfÃ¥ sin frihet. PÃ¥ sin farofyllda och spännande resa genom Amerikas majestätiska vildmark finner Spirit oväntat en vän i en ung Lakotakrigare. Original...

Directed by: Lorna Cook.

År: (2002)
Aspect:
Avsnitt: </desc>
    <duration units="minutes">90 min</duration>
    <category>movie</category>
    <category>Movies</category>
  </programme>

Code:
<programme start="20071210011000 +0100" stop="20071210060000 +0100" channel="svt1.svt.se" clumpidx="0/1">
    <title>Sändningar frÃ¥n SVT24</title>
    <desc>



År:
Aspect: 4:3
Avsnitt: </desc>
    <duration units="minutes">290 min</duration>
  </programme>
zehd
Offline

Posting Freak
Posts: 5,119
Threads: 249
Joined: Feb 2006
#80
2007-12-11, 06:47 AM
agerdin Wrote:Zehd,

All looks good and I really like the ability to change the order, add your own literals and add carriage returns. However, since I am adding literals and carriage returns they still appear even for shows that does not have that tag. What I want is to ignore the literal and/or carriage return if the tag is not found for that show. I also would like the option to remove the () from the tag <date>. See some Ultra-XMLTV Enhancer generated examples below.

Thanks for a cool app.

\\Agerdin

Code:
<programme start="20071215024000 +0100" stop="20071215033500 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>CSI</title>
    <desc>NÃ¥gra fÃ¥gelskÃ¥dare hittar en korp med ett människoöga i munnen och spÃ¥ren leder till en frÃ¥nskild man och hans tvÃ¥ barn.


Actors: William Petersen, Marg Helgenberger, Gary Dourdan, George Eads, Jorja Fox, Paul Guilfoyle.
År: (2002)
Aspect:
Avsnitt: Del 12 säsong 3 </desc>
    <duration units="minutes">55 min</duration>
    <episode-num system="xmltv_ns">2 . 11 .</episode-num>
    <category>series</category>
    <category>Drama</category>
  </programme>

Code:
<programme start="20071215120000 +0100" stop="20071215133000 +0100" channel="kanal5.se" clumpidx="0/1">
    <title>Spirit - Hästen frÃ¥n vildmarken</title>
    <desc>Amerikanskt animerat familjeäventyr frÃ¥n 2002. Handling: Spirit är en vild ung mustang som beger sig ut pÃ¥ en dramatisk och actionfylld resa för att utföra det omöjliga, nämligen att rädda sitt hemland och Ã¥terfÃ¥ sin frihet. PÃ¥ sin farofyllda och spännande resa genom Amerikas majestätiska vildmark finner Spirit oväntat en vän i en ung Lakotakrigare. Original...

Directed by: Lorna Cook.

År: (2002)
Aspect:
Avsnitt: </desc>
    <duration units="minutes">90 min</duration>
    <category>movie</category>
    <category>Movies</category>
  </programme>

Code:
<programme start="20071210011000 +0100" stop="20071210060000 +0100" channel="svt1.svt.se" clumpidx="0/1">
    <title>Sändningar frÃ¥n SVT24</title>
    <desc>



År:
Aspect: 4:3
Avsnitt: </desc>
    <duration units="minutes">290 min</duration>
  </programme>

I will remove all of the () and add the option to include the duration type and other labels. I will remove all of the modifiers and just add a couple to the Description string. (more for demonstration purposes. A person can edit the string and take out what they don't like... It will be about a week, or a bit more.
Frank Z
[COLOR="Gray"]
I used to ask 'why?' Now I just reinstall...
[SIZE="1"]______________________________________________
Author: ZTools: ZProcess, MVPServerChecker; UltraXMLTV Enhancer, Renamer, Manager; [/SIZE]
[/COLOR]
« Next Oldest | Next Newest »

Users browsing this thread: 3 Guest(s)

Pages (39): « Previous 1 … 6 7 8 9 10 … 39 Next »
Jump to page 


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to extract M3U8 and get matching XMLTV guide data from NPVR almightyj 0 2,933 2018-10-23, 07:24 AM
Last Post: almightyj
  XMLTV Channel Number Option gdogg371 12 6,050 2018-07-18, 04:32 PM
Last Post: sub
  Merge xmltv data with xslt Graham 4 3,226 2014-01-02, 12:45 PM
Last Post: Graham
  XMLTV data mvallevand 6 2,371 2011-04-16, 03:08 AM
Last Post: sub
  Free XMLTV EPG scraper dneprrider 214 94,129 2010-03-29, 04:54 AM
Last Post: dljones8053
  XmlTv Structure psycik 2 2,033 2008-12-29, 10:53 PM
Last Post: psycik
  Content Provider: How to publish XMLTV data? skoka123 6 3,125 2008-09-03, 05:48 AM
Last Post: skoka123
  XMLTV Questions -Oz- 34 9,499 2008-03-24, 01:19 AM
Last Post: zehd
  I-xmltv updated for YahooXMLTv (adds a new show marker and more) Jim_ 78 22,953 2008-01-02, 10:33 AM
Last Post: tvshowman
  xmltv zap2it & msn scraper merge tvshowman 2 1,684 2007-12-12, 03:48 AM
Last Post: tvshowman

  • View a Printable Version
  • Subscribe to this thread
Forum Jump:

© Designed by D&D, modified by NextPVR - Powered by MyBB

Linear Mode
Threaded Mode