Search Engines: Three to Beware

Ever since Google and Yahoo! set the gold standard for how search engines use other people’s content, there have been other sites and services that have sought to push the envelope.

But following the controversy over the RSS aggregator Shyftr, it is worthwhile to take a look at three other sites that are making widespread use of blog content, how they are doing so and what the potential implications are.

After all, Shyftr is not the only site republishing your feed in full or large part, it is only one of several. Some of which may raise even more red flags than the original Shyftr itself did.
BlogDimension

Blogdimension LogoBackground: BlogDimension calls itself “The Web 2.0 search engine”. It is fundamentally a blog search engine that functions similar to a Technorati or an Icerocket. Users search for terms and are delivered a list of relevant blog entries. Users can also search for audio and video.

Why it is Controversial: In addition to offering standard search results, the site offers a “cached” link that displays the full feed as it is scraped from the RSS entry. The scraped entries are available to the search engines, there is no robots.txt, the cached link is not nofollowed and the page itself is not marked “noindex”, and are served with many different ads.

To make matters worse, the cached pages do not follow the standard of a clickable title, instead offering only a very small attribution in the footer, and they modify the original articles by stripping out images/ads and other formatting.

All in all, the site manages to avoid doing nearly all of the things that made Google Cache a fair use and does not offer a clear opt-out system or a way to prevent caching.

Blogdimension’s Response:Blogdimenion addressed many of these issues in a recent blog entry detailing a conflict they had had with another blogger.

They start off by saying that “The story is very simple. There is a new trend from some bloggers to treat search engines as thieves.” This seems unduly callous considering that it is their search engine that violates standard industry practices and goes against recent legal rulings.

However, they do go on to say that they are working to address the bug that causes “nocache” to not work and are are also considering only caching short portions of the original entries. Considering that there are currently no standards for designating “nocache” in RSS feeds, especially in cases where the feed is hosted off-site, that may be the best approach.

My Opinion: Blogdimension pushes the envelope significantly both legally and ethically. It crosses lines that a search engine should not cross and goes beyond being just a tool for locating information and, instead, is also trying to profit from the work of bloggers by displaying ads next to full content.

With a few modifications, Blogdimension could be a decent search engine. But as a site that is hosted in the EU, namely France, it is under the gun of even more stringent notice and takedown provisions than here in the U.S.

Fixing the service to both be a good neighbor and fit more comfortably with industry practices should be simple, but it is of critical importance.

http://www.plagiarismtoday.com/2008/04/17/search-engines