UltraXMLTV Enhancer Version 2 - Beta

[zehd]

On a side note....

I remember one of the cool features I read about Tivo a long time ago (though never experienced it) is a search phrase, that would record ANYTHING that matched the phrase.

The example I heard was Type in "Barbra Streisand" and anything with Barbra Streisand in the title, sub_title or description, would be scheduled to be recorded.

That mans you could have a 'standing order' that would auto schedule recordings as they come up, as opposed to the current Search feature which only displays all events already in the EPG.

I'm thinking of adding this into Ultra, as Ultra can watch when something comes in and automatically mark it to be recorded...

Of course you probably don't want a search phrase of 'the'

[TomR]

There was a day not to long ago, thta I could throw a series name at theTVdb.com like "Aardvarks taste funny" (the eg assumes the show doesn't really exist) and I'd get a response back from theTVdb.com server, lickety split with a blank or empty xml file

I would then mark that show as Wrong Data and then never ask to check that again.

Well I got tired of that. Now it waits for a full time out (mine is set for 15 seconds) but even longer doesn't give me a empty xml file.

So I looked at keeping track of all the ids for all the shows I try...

And something cool is resulting from this. I end up with a transferable text file tat lists all the shows with their IDs. Anything that doesn't have an ID, is marked No Data in the list.

That means that I could spend an hour or two filling this list, and send the file along with my zip file, and then you guys won't need to check those names ever again.

If you have shows that I have not checked, then it will append to the list, and you can save it somewhere...

I already have a fairly current file from LewE, so I'll be checking his file. If TomR wants to send me his HUGE file, I can run that through and you won't have to spend the time again...

Where do you want them sent? They're to big to attach (16 MB zipped).

I've been sucessful in getting UltraXMLTV torun from both my c: drive and USB stick.

When you get the files, I'm having a problem with Episode/Season showing up for American Idol (and possibly other shows but I haven't noticed yet).

In version 1 of UltraXMLTV I use to get this information (something like S8E19). But with the latest version (2.0.3355.23809 ) I am not getting information. It'ss now showing up like 821/822.

I am getting proper S/E info for other shows. I checked my XML_CACHE folder and the is no longer any files for American Idol. I also checked the _app.log and I do not find any reference to American Idol.

Has something changed that would prevent the S/E info for some shows?

[zehd]

Has something changed that would prevent the S/E info for some shows?

Yup! There was definitely some luck where Ultra1 got some shows, that I have no idea how to do again. There's a whole bunch if if then else happening once a show is found. '24' is anoter example of a show where getting SeasEP to work is hit or miss

They are both from the Fox network, and believe I've looked at that too.

Anyway for now, I'm letting some shows go, if they require very special one-off handling.

I have to deal with the TVdb kinda pulling the rug out from me...

I'll PM my email address to you. You can send an XMLTV file ether.

[TomR]

Just wanted to get back to you with some stats on running from a USB stick...

I follow your instruction and put everything on the USB stick and ran from there. The 1st time it took a couple of hours. I sure this was related to all of the fetching it had to do for the XML_CACHE folder.

Now this is the 2nd day of running from the stick. This was all done on a highspeed connection at work.

Here are some stats:

Original XML file is 100MB

Fetches: 31
Queries: 22298
Responses: 16016
Ended: 17928
Wrong/No Data: 21620

This Download: 165.26 KB

Processing Time : 00:50:32

While the time is a little longer than I was hoping, it still isn't too bad.

I might get adventurous and give it a try at home using a dial up connection. With only 31 fetches (165.26 KB), hopefully it will be done in about 1 hour. I'll let you know what kind of results I get on the dial up.

Did I see somewhere that you were working on something that might help speed this process up the whole process (not just by reducing the number of fetches)?
Tom

[zehd]

Just wanted to get back to you with some stats on running from a USB stick...

I follow your instruction and put everything on the USB stick and ran from there. The 1st time it took a couple of hours. I sure this was related to all of the fetching it had to do for the XML_CACHE folder.

Now this is the 2nd day of running from the stick. This was all done on a highspeed connection at work.

Here are some stats:

Original XML file is 100MB

Fetches: 31
Queries: 22298
Responses: 16016
Ended: 17928
Wrong/No Data: 21620

This Download: 165.26 KB

Processing Time : 00:50:32

While the time is a little longer than I was hoping, it still isn't too bad.

I might get adventurous and give it a try at home using a dial up connection. With only 31 fetches (165.26 KB), hopefully it will be done in about 1 hour. I'll let you know what kind of results I get on the dial up.

Did I see somewhere that you were working on something that might help speed this process up the whole process (not just by reducing the number of fetches)?
Tom

Although I did not see a huge decrease in speed with simple read/write of the xml files in cache, it does sound like that might be your bottle neck.

It's possible that with 31 fetches at Dialup, it won't be that bad at all...

I will be releasing something that will not Require Ultra to seek out Series xmls from theTVdb.com. Although the series files are very small (most of the real data is in the Episodes xml), theTVdb.com seems to be freezing my program whenever it is asked if a series exists

(It seems a lot of the other grabbers are also complaining about this)

Basically its where Ultra will download a master SeriesID list, and bypass asking theTVdb.com anything at all...

Now Ultra won't get lost in the Internet when a show doesn't exist...

[leaded1]

I got Ultra2 up and running yesterday. Looks great! My processing time dropped from about 8 hours with Ultra1 to ~30minutes. Very cool. This definitely looks like a keeper. No problems with registration or installation once I installed in the directory you mentioned in your instructions. New show detection looks good for all of the shows I've got on Recurring recordings. Season and Ep numbers are working for almost all of the shows I'm interested in. I had to play around with a the timeout duration in order to get some of them to download the first time [notably The Office (US)].

I'm Looking forward to the features you've hinted at.
Thanks,
Leaded1

[zehd]

No problems with registration or installation once I installed in the directory you mentioned in your instructions.

Thanks. Just so you know, I do want to make it so people can put it anywhere they want. I just have a few things I'm trying to troubleshoot, and keeping to a default directory makes sense for now

[zehd]

OK, I haven't released the very newest version yet, but I want to get started on makeing some notes. I hope that it will make enough sense WITHOUT you seeing it, that you may be able to offer some suggestions.

Problem: the TVdb.com responds very very slowly when there is a request for a series ID, based on a series name. It's even slower when you query a show that doesn't even exist in the database.

Solution: a master list that contains all the Series Names and Series IDs.

WOW! OK, If the list would be complete, it would be huge I'm sure. And I've searched if there even was a downloadable list, but nah!!!

New Ultra has a button to download latest Master list (hosted on my server space) and merge it into your own SeriesID list.

As Ultra cycles through each of the records, it asks the list for a SeriesID based on a Series Name. It's either a valid number or the string "No Data"

If it's No Data, it skips and carries on. If it's Valid it checks if it needs to get a copy of the Episode XML (That part hasn't changed in speed)

---

Now the XML_Cache doesn't have Series files or Wrong.Data files. The data is maintained in the _SeriesIDs.txt list (actually in memory as an array, for speed.)

---

If you come across a show that is not in the list, then Ultra will try and get the information and then add it to the list. It shouldn't have to get that info ever again.

---

Now, due to theTVdb.com causing grief, sometimes, even valid shows, will seem like it has 'No Data'. And it can be a lot of fun and work to get the correct number added to the list. Sometimes manually, sometimes several retires (we can go into that later)

What if I have 'Boston Legal=No Data' but you have 'Boston Legal=74058' and it took a bit of work too, maybe lots of time on the net to get that stupid number. How can I share in that information?

There is also a button that will upload your _SeriesIDs.txt to my server space. I can download it, merge it into the master and then upload the master.

Everyone else can then Download the Master file again, and we'll all be up to date...

---

I don't want this to be automatic. I don't want this to be mandatory. It's a little bit like the torrent idea, if you're going to download, you should upload too...

---

Just so you know, when I merge, I compare records. If I have No Data, and your file has a real ID (Good work!) I save the Real ID, for all to use from the future.

---

The goal is to have accurate information regarding if a Series Exists or not. If it does, we should have the real ID, if it doesn't let's not waste any time, let's know it doesn't by saying "No Data"...

[LewE]

Solution: a master list that contains all the Series Names and Series IDs.

WOW! OK, If the list would be complete, it would be huge I'm sure. And I've searched if there even was a downloadable list, but nah!!!

Would there be any significant difference if there were different lists for different countries/regions? You'd know better than I how many series there are world-wide and how many of these are pretty much local to certain areas/continents.

New Ultra has a button to download latest Master list (hosted on my server space) and merge it into your own SeriesID list.

As Ultra cycles through each of the records, it asks the list for a SeriesID based on a Series Name. It's either a valid number or the string "No Data"

If it's No Data, it skips and carries on. If it's Valid it checks if it needs to get a copy of the Episode XML (That part hasn't changed in speed)

---

Now the XML_Cache doesn't have Series files or Wrong.Data files. The data is maintained in the _SeriesIDs.txt list (actually in memory as an array, for speed.)

---

If you come across a show that is not in the list, then Ultra will try and get the information and then add it to the list. It shouldn't have to get that info ever again.

---

Now, due to theTVdb.com causing grief, sometimes, even valid shows, will seem like it has 'No Data'. And it can be a lot of fun and work to get the correct number added to the list. Sometimes manually, sometimes several retires (we can go into that later)

What if I have 'Boston Legal=No Data' but you have 'Boston Legal=74058' and it took a bit of work too, maybe lots of time on the net to get that stupid number. How can I share in that information?

There is also a button that will upload your _SeriesIDs.txt to my server space. I can download it, merge it into the master and then upload the master.

Everyone else can then Download the Master file again, and we'll all be up to date...

Since you mention this would be an array in memory at run time I assume that the file won't be that big that uploading and downloading won't be a problem, even for diak-up customers.

I don't want this to be automatic. I don't want this to be mandatory. It's a little bit like the torrent idea, if you're going to download, you should upload too...

How about they can download the current version as part of the download of the program. Subsequent downloads could be tied to their uploading their locally generated file. This could be set by configuring Ultra by the user for whether periodic updates are to be retrieved.

Just so you know, when I merge, I compare records. If I have No Data, and your file has a real ID (Good work!) I save the Real ID, for all to use from the future.

If it were to be automatic, you could have a Rev Level built into the current version of the file. When a user updates their local file with a new entry that they discovered their file could be that rev level flagged as modified. Then, when Ultra runs it could query your server with its local copy's Rev Level and get back from your server that it is current or get a download of a fresh copy. If Ultra has a locally modified version it could upload it to your server as part of this daily processing.

The goal is to have accurate information regarding if a Series Exists or not. If it does, we should have the real ID, if it doesn't let's not waste any time, let's know it doesn't by saying "No Data"...

If these comments make sense, GOOD. If not, just write it off to my having gone off on a tangent due to misunderstanding.

[zehd]

Would there be any significant difference if there were different lists for different countries/regions? You'd know better than I how many series there are world-wide and how many of these are pretty much local to certain areas/continents.

How about they can download the current version as part of the download of the program. Subsequent downloads could be tied to their uploading their locally generated file. This could be set by configuring Ultra by the user for whether periodic updates are to be retrieved.

If it were to be automatic, you could have a Rev Level built into the current version of the file. When a user updates their local file with a new entry that they discovered their file could be that rev level flagged as modified. Then, when Ultra runs it could query your server with its local copy's Rev Level and get back from your server that it is current or get a download of a fresh copy. If Ultra has a locally modified version it could upload it to your server as part of this daily processing.


If these comments make sense, GOOD. If not, just write it off to my having gone off on a tangent due to misunderstanding.

I appreciate your thoughts...

So far with a combined list of my shows and YOURS, the text file is 40k. That's not bad. Though if it were automatic, and my program takes off with a larger user base, I would need to worry about the bandwidth load on Whurstlon's server...

As far as importing. It's fast painless and non destructive. So it wouldn't be a problem to do it all the time, except for the bandwidth. Yes, I could have date checking built in so it won't actually download and import unless it's a newer version online.

If it turns out that I see a dramatic decrease in speed or power when the file (in memory) is so unwieldy, I have another couple of ideas up my sleeve. But I won't want to worry until that happens...

AS far as the languages/countries, well, being From Canada, I have been checking mnay French shows too. Making sure the accents etc are working, and there really isn't all that many non English (UK and US) shows that I've seen.

That said, a 'show- to- id' is unique, and so whatever you throw at me, if it exists, you'll see it displayed...

I will need to worry about the actual ISO codes further down the line though.

(hmmm, maybe that's why I haven't seen two many foreign shows, (maybe I should be setting the lang to 'fr' and I'll see more) (AHHHH Get out of my brain)