UltraXMLTV Enhancer Version 2 - Beta

[TomR]

OK, I haven't released the very newest version yet, but I want to get started on makeing some notes. I hope that it will make enough sense WITHOUT you seeing it, that you may be able to offer some suggestions.

Problem: the TVdb.com responds very very slowly when there is a request for a series ID, based on a series name. It's even slower when you query a show that doesn't even exist in the database.

Solution: a master list that contains all the Series Names and Series IDs.

WOW! OK, If the list would be complete, it would be huge I'm sure. And I've searched if there even was a downloadable list, but nah!!!

New Ultra has a button to download latest Master list (hosted on my server space) and merge it into your own SeriesID list.

As Ultra cycles through each of the records, it asks the list for a SeriesID based on a Series Name. It's either a valid number or the string "No Data"

If it's No Data, it skips and carries on. If it's Valid it checks if it needs to get a copy of the Episode XML (That part hasn't changed in speed)

---

Now the XML_Cache doesn't have Series files or Wrong.Data files. The data is maintained in the _SeriesIDs.txt list (actually in memory as an array, for speed.)

---

If you come across a show that is not in the list, then Ultra will try and get the information and then add it to the list. It shouldn't have to get that info ever again.

---

Now, due to theTVdb.com causing grief, sometimes, even valid shows, will seem like it has 'No Data'. And it can be a lot of fun and work to get the correct number added to the list. Sometimes manually, sometimes several retires (we can go into that later)

What if I have 'Boston Legal=No Data' but you have 'Boston Legal=74058' and it took a bit of work too, maybe lots of time on the net to get that stupid number. How can I share in that information?

There is also a button that will upload your _SeriesIDs.txt to my server space. I can download it, merge it into the master and then upload the master.

Everyone else can then Download the Master file again, and we'll all be up to date...

---

I don't want this to be automatic. I don't want this to be mandatory. It's a little bit like the torrent idea, if you're going to download, you should upload too...

---

Just so you know, when I merge, I compare records. If I have No Data, and your file has a real ID (Good work!) I save the Real ID, for all to use from the future.

---

The goal is to have accurate information regarding if a Series Exists or not. If it does, we should have the real ID, if it doesn't let's not waste any time, let's know it doesn't by saying "No Data"...

What about a flag to automatically upload the user’s _SeriesIDs.txt file and another flag to automatically download a new Master file? This way you could be sure to get more information to add to the master file and the user could be sure that they always have the latest master file. The down side could be you might get flooded with copies of _SeriesIDs.txt files.

What about having UltraXMLTV automatically check and display a message that a newer master file exists and ask if it should be downloaded? The user would not be prompted depending on the flag the set above.

What about linking the 2 processes together? If you download a master file then your _SeriesIDs.txt file would also be uploaded.

Just my 2 cents.

Tom

[TomR]

I got Ultra2 up and running yesterday. Looks great! My processing time dropped from about 8 hours with Ultra1 to ~30minutes. Very cool. This definitely looks like a keeper. No problems with registration or installation once I installed in the directory you mentioned in your instructions. New show detection looks good for all of the shows I've got on Recurring recordings. Season and Ep numbers are working for almost all of the shows I'm interested in. I had to play around with a the timeout duration in order to get some of them to download the first time [notably The Office (US)].

I'm Looking forward to the features you've hinted at.
Thanks,
Leaded1

zehd, don't you just love it when things work as you intended?

Maybe LewE and I should stop breaking thing on you.

[zehd]

What about a flag to automatically upload the user’s _SeriesIDs.txt file and another flag to automatically download a new Master file? This way you could be sure to get more information to add to the master file and the user could be sure that they always have the latest master file. The down side could be you might get flooded with copies of _SeriesIDs.txt files.

What about having UltraXMLTV automatically check and display a message that a newer master file exists and ask if it should be downloaded? The user would not be prompted depending on the flag the set above.

What about linking the 2 processes together? If you download a master file then your _SeriesIDs.txt file would also be uploaded.

Just my 2 cents.

Tom

Well, I'm still a little timid with the whole Upload thing... I know how spyware can do nasty stuff, and as I have said elsewhere, I don't trust a helluvalot on the net...

As far as Auto downloading, for now, I won't worry about date checking, and I can have it auto download and auto upload by option.

I won't get flooded. I'll just have your very latest file (overwrites on the server)

I will have the 'Upload as Master File' Button too, but it will be password protected...

After a few of you get the hang of it, I'll give you guys a password, so that you can upload a updated master file, that really anyone can make, so long as you know how to NOT F^(&^$(^ with the data...

[LewE]

Well, I'm still a little timid with the whole Upload thing... I know how spyware can do nasty stuff, and as I have said elsewhere, I don't trust a helluvalot on the net...

After a few of you get the hang of it, I'll give you guys a password, so that you can upload a updated master file, that really anyone can make, so long as you know how to NOT F^(&^$(^ with the data...

Good point. Since you are talking about a plain text format, you can never know what sort of damage users may have done to their files from viewing/editing with a text editor although I don't think viruses can be transmitted via plain text files. You would, however, have to add a lot of validity checking to your input process when you deal with uploaded files.

You may want to consider a proprietary formatted file (your own design) to eliminate users modifying them.

[TomR]

Just wanted to get back to you with some stats on running from a USB stick...

I follow your instruction and put everything on the USB stick and ran from there. The 1st time it took a couple of hours. I sure this was related to all of the fetching it had to do for the XML_CACHE folder.

Now this is the 2nd day of running from the stick. This was all done on a highspeed connection at work.

Here are some stats:

Original XML file is 100MB

Fetches: 31
Queries: 22298
Responses: 16016
Ended: 17928
Wrong/No Data: 21620

This Download: 165.26 KB

Processing Time : 00:50:32

While the time is a little longer than I was hoping, it still isn't too bad.

I might get adventurous and give it a try at home using a dial up connection. With only 31 fetches (165.26 KB), hopefully it will be done in about 1 hour. I'll let you know what kind of results I get on the dial up.

Did I see somewhere that you were working on something that might help speed this process up the whole process (not just by reducing the number of fetches)?
Tom

I tried the USB stick at home with a dial up connection. I used the same XML file as I did at work. I wen to the XML_CACHE folder and delete the files that were downloaded today. This was to ensurer the the data was similar for each test.

Stats:
Original XML file is 100MB

Fetches: 30
Queries: 22298
Responses: 16015
Ended: 17927
Wrong/No Data: 21621

This Download: 94.17 KB

Processing Time : 01:00:22


Pretty much just as expected. It only took marginally longer with the dial up connection. But it's nice to know that poor dial up user (like myself) can do the enhancing.

The kill would be in grabbing the orginal XML file. Since my file is about 100MB it would take me 15+ hours and I don't see me doing that. :eek:

[TomR]

Well, I'm still a little timid with the whole Upload thing... I know how spyware can do nasty stuff, and as I have said elsewhere, I don't trust a helluvalot on the net...

If there was an obvious flag tahat allow you to select uploading (set off by default) it may get around the idea of spyware. E.g. the users would have to select it in order to upload it.

I won't get flooded. I'll just have your very latest file (overwrites on the server)

I'm not sure I understand why ther would only be 1 file. If LewE and myself both upload the file would you not have 2 files that need to be processed? Therefore, when to get you get to 1,000,000 users uploading files :eek: , would you not have 1,000,000 files that would need to processed? Would the merge process be manuallly initiated by you or would it be automatic? Maybe this would not really be an issue due to the size of the files being uplaod.

[zehd]

If there was an obvious flag tahat allow you to select uploading (set off by default) it may get around the idea of spyware. E.g. the users would have to select it in order to upload it.

I'm not sure I understand why ther would only be 1 file. If LewE and myself both upload the file would you not have 2 files that need to be processed? Therefore, when to get you get to 1,000,000 users uploading files :eek: , would you not have 1,000,000 files that would need to processed? Would the merge process be manuallly initiated by you or would it be automatic? Maybe this would not really be an issue due to the size of the files being uplaod.

Believe me, if I'm at 1,000,000 users, I'll have a web service that receives and auto processes uploaded data. Or I'll have the help do it...

Yes, you and LewE would upload separate files, and then there would be two. And yes I'd have to import them both one at a time and manually.

But the list would be incrementing, getting more and more accurate, and it would actually beome a smaller job as more of the TV shows get identified.

200 stations and there's still crap on TV...

[zehd]

Release: 2.0.3357.31501

OK. There are are few design elements that are still quite messy. I just wanted to get this out so I can have you test it and I can see the mechanism work.

LewE, I based the included _SeriesIDs.txt file on yours and my data. Yours is maybe two days old. So when you go to enhance you should have only a few fetches.

TomR, this is why I wanted your xml file (BTW if you zip, It's much smaller I'm sure) With over 100,000 records, you're going to be having some extra fetches indeed. (Or you could PM me your zip code and lineup and I can mc2xml myself a copy)

First off, don't delete or backup anything. I especially designed this to handle legacy. Copy the Ultra.exe, Replace.ini and seriesID.txt files into the Ultra folder.

Start Ultra, Click the Cache tab, and click "Download Series IDs and import'. Because the file that you download is the same as the one included in the package, not much will happen. I just want to be sure things are going to work...

Next on the Main tab. Click Go (Commit or not, doesn't matter)

LewE should have this done in a few minutes. TomR, I have no idea, though it shoud be pretty good. (I DON'T WANT TO HEAR ANYTHING ABOUT DIAL-UP)

During the first 'Go', Ultra will search your XML_cache folder and automatically add any programs that have been marked "Wrong Data" into the list as "No Data". It will then do some house keeping and delete the file types that are not needed anymore. (You won't need any Wrong Data or .. Series.xml files either.

OK. Then Ultra starts to go through your records and compare the shows to the list that's in memory. As it finds something that isn't in the list, it will force a fetch and see if there's an ID or No Data. This is the part that is gonna kill us all...

theTVdb.com is so freaking slow, that Ultra freezes up while it is getting told to F off by the server...

You may get a few entries that were never detected correctly in previous Ultra versions, you may get stuff that you KNOW has Ep info, yet it's marked "No Data" Well some of that might be me, but really most of it is the server taking so long, that the timeout comes and goes and I just Mark it No Data, so I can move on and get some sleep.

Yeah you could set the Web Timeout to 2 minutes. But I have noticed that when the server is sure it wants to mess with you, it will wait till however long you want to wait, before it tells you to F off (Did I say the server is making me mad?)

When it finishes. Please Upload (Cache tab >> Upload for Sharing) And then let me know you've done that , so I can look at what you got and then merge it into the master file.

---

Now, there is a possibility that the TVdb.com has singled me out (with my IP, cause I've been pestering the server a lot ) (I really don't think so. There are lots of people complaining about it this weekend, and still not fixed)

But IF you are downloading new fetches at a freakishly fast pace (You can check the WebResonse log to tell you about how fast you queries are happening. I want you to have a go at testing all the shows marked No Data and maybe some of the shows can be fixed.

What you do is on the Cache Tab, click Purge All No Data, and then on Main tab, click Go. Let it run through. Upload the results, Let me know...

I will merge the content, and we will all be better off...


Good Luck

[zehd]

If you get some shows marked No Data, there are ways to update the files. But currently it's pretty manual, and I'd like you to NOT play with the data files just yet...

[zehd]

One more thing. If you're running, and it's in the middle of downloading, Keep clicking the Stop button until the Web Time out occurs. That will cause a graceful stop.

I have killed the program many a time as well, but Closing gently makes sure the new file is capped up nicely