TVGids.nl actively stopping grabbers?

[syncpuls]

I noticed that as of today I cannot grab data from the Dutch tvgids.nl website. I'm using TVxb, which is one of the -friendly- tools (it tries to keep the traffic on the website in mind).

TVxb is using wget (a port from Linux to download web pages). I noticed that apperantly tvgids.nl is not happy with browsers identifying as wget... as soon as I change the agent-name I can download pages again!

example:

C:\Program Files\TVxb\bin>wget http://www.tvgids.nl -t 1
--21:15:34-- http://www.tvgids.nl/
=> `index.html'
Resolving http://www.tvgids.nl... 195.144.10.130
Connecting to http://www.tvgids.nl[195.144.10.130]:80... connected.
HTTP request sent, awaiting response...
Read error (No such file or directory) in headers.
Giving up.


C:\Program Files\TVxb\bin>wget http://www.tvgids.nl -t 1 -U --user-agent="Internet Explorer"
--21:15:40-- http://www.tvgids.nl/
=> `index.html'
Resolving http://www.tvgids.nl... 195.144.10.130
Connecting to http://www.tvgids.nl[195.144.10.130]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

[ <=> ] 28,342 --.--K/s

21:15:40 (294.44 KB/s) - `index.html' saved [28342]

As you can see I'm impersonating "internet explorer" in the second download attempt, and yes: I'm allowed!

Unfortunately I can only do this manually, and don't know how to 'fix' TVxb.... Any suggestions?

[syncpuls]

Well, that took a whopping 15 minutes to figure out...

In the
C:\Documents and Settings\All Users\Application Data\TVxb\ini

folder is a 'wgetrc' file

It appears wget has a undocumented option that can be used here; simply add:

user_agent="Internet Explorer"
At the end of the file.

Now tvgids.nl thinks you're an internet explorer user.

I hope this info gets to someone who needs it.

[mikenl]

Thanks for the info.
Setting up tvxb now, was using xmltv, i got the same errors there.
Was thinking for a solution until i read your post, thanks !

[syncpuls]

You're welcome...

Grin... If it affects xmltv as well as TVxb I guess a lot of Dutch PVR users are browsing the web right now... ;-)

Looks like I've done my 'good job' for the day. ;-)

The only reason I can think of that TVGids.nl is doing is, is that they are getting tired of gabbers (I guess I can't blame them).

To all: Please consider this when configuring your grabber! Let's not configure them 'greedy' and hog bandwidth, since this action by TVgids.nl shows us that we might be burning our own fingers when abusing their servers.

As a side note: I'd be more than happy to get a decent Dutch EPG subscription for a price similar to what I'm now paying for my regular TV guide (that I just don't use any more). I wish TVGids.nl would provide that!

[JanBoeren]

A service as provided by the german tvmovie.de would be great! I have a subscription there for the german tv stations and they provide 14 days EPG complete with programm information! They provide NL1, 2 and 3 too! but no commercial NL stations.

[HtV]

It seems like amontillado's grabber is working aswell. It's still downloading on my server but it seems ok. I hadn't noticed there was a problem, I'm on holiday for 6 days, I'm glad I read it here. Luckily I was able to vnc in to my server.
let's hope everything works out.
I second the demand for a cheap (free is always better, we are dutch hey ) epg source. Let's hope something happens in this area,

Hans

[HtV]

Well, it's still downloading. It shows a lot of errors so I think I was to optimistic.:o

Hans

[syncpuls]

I stopped using Amontadillo's grabber some time ago, I don't remember exactly why; but I seem to recall some stuff not downloading properly. It wouldn't surprise me if it works if he used another identifier than 'wget [versionnumber]'

I wonder what the trouble would be in setting up a Dutch xml-epg website, I guess TV stations are pretty difficult about their programming data.

[-=TJ=-Stefan]

I still use Amontadillo's grabber because it can grab 6 days ahead from the tvgids.nl. But it seems I have to install another. Got a grabber in my mail from someone. Will give it a test and will let you know. Otherwise it seems we have to use TVxb. No option found in Amontadillo's grabber to impersonate a browser

[syncpuls]

Hi Stef, after our previous conversation I dug a bit deeper into that (the more than 3 days thing) It appears that indeed it is possible to get that kind of information from tvgids.nl, but tvxb simply doesn't use those (search) pages.

I wonder if it's possible to get tvxb to do that too, by configuring it differently. Maybe when I have more spare time after my second exam... ;-)

Do I understand correctly from your post that amontillado's grabber is now also giving problems?

(yep, it's amontillado's, not amontadillo's; I looked it up. It's apparently named after some kind of sherry, and E.A Poe wrote a short story about it; its all about some guy being buried alive. Creepy if you ask me, but at least I've now been able to disprove being a culture-barbar.)