Whether search is 90% solved or whether the last 10% will take 90% of the effort, - either or both according to Marissa Meyer - there is a lot of improvement to be had in search. If you honestly want to find authoritative information about a topic that’s been over-SEOed like ‘ring tones’ or ‘mortgages’ – or you’re searching for a semantically challenging term like ‘bush’ - or you’re looking for something particularly esoteric – Google leaves you with a lot of cruft to wade through.
Particularly in technology and the Internet – there is no such thing as a permanent monopoly – eventually someone will challenge Google in core search and start taking their market share.
It sure won’t be Cuil – now that they are in self-destruct mode. It won’t be Powerset now that their assets have been assimilated. But it just may be Gnip – Eric Marcoullier’s ping server to rule them all. In this recent interview with Om Malik, Eric humbly calls Gnip’s service ‘commodity work’ that takes away some logistical headaches for web service developers. But what Gnip is really doing is fundamentally changing the nature of aggregation and indexing on the web from a pull model to a push model.
Search engines today send out spiders to actively crawl the web and pull content into the index – at so great a cost in overhead that crawling is generally considered to be a powerful barrier to entry in the search market.
Contrastingly, if you subscribe to a blog, you get pushed a notification whenever that blog is updated – with a push model, there is no need to scan through every blog you like to read to find out which ones have been updated as a spider would. This push model provides enough of an advantage that webmasters will use Google Adwords & Adsense as a way to ping Google and get new content or sites indexed more quickly than simply waiting for the spider.
This is not a new idea – back in the circa 2000 era, visionary Seattle startup 360powered had the same idea – and even managed to perfect an interesting patent enumerating this architecture. Unfortunately, 360powered fell victim to the dot com crash and their IP ended up auctioned off in bankruptcy.
In a Gnip world, every website would have a feed – whenever content changes – the index gets pinged. Simultaneously, the overhead of crawling is distributed out to the edge, removing the burden from the search engine, and the delay for new content to get indexed goes to zero.
Gnip is nothing less than a fundamental cornerstone of the next generation of dominant search.