HTML parser and/or regular expressions resources

[McBainUK]

Title says it all. Links and tips appreciatied

[Spartan]

I use something in my sports scores plugin which allows you to use XPath expressions on html documents -- very handy.

See here: http://www.codeplex.com/htmlagilitypack

[McBainUK]

Got any code examples? Be great to have something to start me off...

[Spartan]

You can download sports scrores plugin source code from the wiki...

[HTPCGB]

When I wrote the cnnDynSource, I used a free program called Expresso to help me develop and test regex's fairly quickly. It avoids the whole compile, run and test cycle which slows you down significantly.

Here's an example of a method that downloads a web page, scrapes it for hyperlinks and prints them in a textbox control:

Code:

HttpWebRequest theRequest;
HttpWebResponse theResponse;
Stream theStream;

url = "http://www.gbpvr.com";
theRequest = (HttpWebRequest)WebRequest.Create(jsonUrl);
theResponse = (HttpWebResponse)theRequest.GetResponse();
theStream = theResponse.GetResponseStream();
            
string htmlCode = new StreamReader(theStream).ReadToEnd(); //converts the stream to a string

Regex theRegex = new Regex(@"href="(?<hyperlink>.*?)"");

MatchCollection mc = theRegex.Matches(htmlCode); //runs the regex on the string and puts all the matches into an array

for (int i = 0; mc.Count; i++)
{
     textBox1.AppendText("/r/n" + mc[i].Groups["hyperlink"].Value);
}

This requires System.Net, System.Io, System.Text.RegularExpressions and a textbox named textBox1.

The basic form of the regex that I used is (href=".*?"), minus the brackets.
Basically, it looks for (href=") followed by a bunch of characters (denoted by the ".") which is then followed by a (").

A "." represents a single character. It is followed by two modifiers ( "*" and "?"). "*" tells it to look for multiples of the preceding item. "?" tells it to look for as few of the item as possible (once one match is found it stops looking, avoids infinite loops in large documents).

"(?<hyperlink>" and "?)" are part of what's called a capture group. Everything in between the two ends is captured into a group with the name "hyperlink" which is later accessed from the MatchCollection.

I apologize for any mistakes in the code and hope that my alternation from brackets to quoatations in the explanation wasn't too annoying.

[McBainUK]

Many thanks to you both

The basic form of the regex that I used is (href=".*?"), minus the brackets.
Basically, it looks for (href=") followed by a bunch of characters (denoted by the ".") which is then followed by a (").

A "." represents a single character. It is followed by two modifiers ( "*" and "?"). "*" tells it to look for multiples of the preceding item. "?" tells it to look for as few of the item as possible (once one match is found it stops looking, avoids infinite loops in large documents).

"(?<hyperlink>" and "?)" are part of what's called a capture group. Everything in between the two ends is captured into a group with the name "hyperlink" which is later accessed from the MatchCollection.

This is what I needed - a dummies guide. Think it will be very useful for the Cinema listings plugin's web scraper

[Ted the Penguin]

lemme find you a regex site has much more info.

also, if you want to just try out some regex stuff, do it in perl, since all you have to do is run it (of course with perl installed)

http://www.regular-expressions.info/
http://www.regular-expressions.info/reference.html