BTW, I had a thought. Since some themes and site templates are really common, perhaps the engines could use what they know about ‘footprinting templates to ID spam’ in order to reduce processing required for many sites. The script would query a DB of known footprints and see if it’s a theme whose blocks are already identified; if not, it would start froms scratch.

What do you think?

Nice. Freakin. Post.

Likewise as I mentioned in the article, it would be in many cases simpler once blocks have been defined to rely on language for accurate answers, internal navigation will often be referred to as menu either directly in text or as an ID like wise the easiest way to identify a blog roll is to pull from the DOM the group of links that come after the term “Blog Roll” or “Friends Links” or some other derivation, of course this is presuming the site you are looking at is indeed in the language you are using I’m not familiar with the Chinese for “Blog roll” and am unlikely to write a program to look for it.

For example I today applied this technique on a rudimentary paid link detector, looking for links in pages which did not fit in with the content of the block or those of the over all site. It wasn’t hugely accurate but certainly accurate enough to see the use in the technique.

1) for sites on eg wordpress and blogspot, the templates are well known and widely used, so case based reasoning will identify many of the blocks based on an understanding of the popular templates, if you can sniff them out;

2) when calculating link density, you can probably strip out em, i, b, strong etc tags (leaving their contents) to simplify the dom;

3) for identifying the major content block in a page, eg the news story in a news page, then the block with some increasing function of lowest link density and longest text character count is possibly the one you want; there probably needs to be some minmax reasoning going here - eg i may want the the largest amount of text content at as low a level in the dom hierarchy as possible (ie i want the content div not the main=content+banner block)

4) stuff like adsense etc can be sniffed and maybe remived from the dom to simplify processing;

5) where an internal navigation block is identified, I wonder whether you can start to look for site based templates (which is i guess what dapper tries to do when you give it several related pages) and then simplify the analysis on this basis - eg building a weighted page that maybe gives ‘density confidence’ values (or mean+sd density scores) of common blocks across linked pages from the same site, maybe at the same depth in the URL path?

I love a good dog walk

When i posted the bookmark, my thought was to use something like link density to try and guess at two sorts of block based on link density, where link density is measured really simply as something like (number of a nodes in a subtree of the DOM)/(number of non-text nodes in the subtree), or even just (number of alpha chars of link text in subtree)/(number of alpha chars in subtree)

1) intrasite navigation blocks (high link density and high proportion of those links pointing to the local domain)
2) blogrolls (cribbed from a ‘blogroll’ title, class or id attribute value or based on high link density and heuristic that counts wordpress.com, blogspot.com etc. ALternatively, run autodection on the links to see if they all return RSS feeds that look like canonical blog feed urls from major platforms, or feedburner feeds…

You’ve got me really intrigued about pursuing this a bit further now