Need helping fetching a web page using c#

[Jeff]

Does anyone know how to use c# to call a web site and enable Javascripts to run locally?

I am trying to update the weather plug-in to use the new msnbc format and one think I have noticed is that I can no longer figure how to directly access the weather web page by US zip code. I now have to use area codes even for US cities. I ave been trying to figure out a way to automatically map zip codes to area codes.

What I have discovered so far is that from my browser I can do it by typing in the URL http://weather.msn.com/search.aspx?weasearchstr=12121, where 12121 is the zip code.

In code I have tried to do the same but it does not work.

Code:

// See if this as a new style code or an old style US zip code
Stream stream = null;
if (this.areaCode.Length == 5)
   stream = new WebClient().OpenRead("http://weather.msn.com/local.aspx?wealocations=wc:" + this.areaCode);
else
   stream = new WebClient().OpenRead("http://weather.msn.com/search.aspx?weasearchstr="+this.areaCode);
                    
StreamReader streamReader = new StreamReader(stream);
string webPage = streamReader.ReadToEnd();
stream.Close();

The webpage that comes back includes an error message in ot that says “A display error has occurred. To use this feature, you must allow the use of Javascript. To fix this error, try changing your security settings so that the use of Javascript is enabled.”

Does anyone know how to do this?

Thanks

Jeff

[reven]

off the top of my head, you might want to try and look at setting a header so it tricks msnbc into thinking you do support javascript so it sends the page.

[Jeff]

Sounds promising, but I don;t know how to do that? Cna you point me in the right direction?

Thanks

[reven]

something like

Code:

HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(strURL);
objRequest.Headers.Add("HeaderName: value");

so just do a search on "javascript" header, or "http headers" or something, and find the header to set javascript, like to set the contenttype you would use something like

Code:

HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(strURL);
objRequest.Headers.Add("Content-Type: text/html");

im hoping that http headers can specify if a browser can support java script,its the only way i know how a server would get this information (well from a request to start with).

[reven]

hey i found this via google

Most Internet Explorer versions list a "Mozilla" (Netscape) version first in their User-Agent line, with the real browser version listed parenthetically. This is done for compatibility with JavaScript, where the User-Agent header is sometimes used to determine which JavaScript features are supported. Also note that this header can be easily spoofed, a fact that calls into question the reliability of sites that use this header to "show" market penetration of various browser versions.

so yeah just do the same, so somthing like

Code:

HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(strURL);
objRequest.Headers.Add("User-Agent: Mozilla/5.0");

[Jeff]

I found out how to fix it. I switched to a seperate request/response and set the keep alive paramter to false. it is true by default and for some reason the conneciton was getting closed.

Code:

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(weatherURL);
request.KeepAlive = false;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

Jeff