Keywords, natural language and search

Seamus McCauley has written an interesting piece on "Keywords vs. 'natural language'":

Keyword-based search works so well, so intuitively, because if you watch people interact with other people you'll see them watch out for the keywords too. Google's trick wasn't to invent anything - it was just to pay attention to how we were doing things already and then train computers to do it too. Which really isn't good news for all the people striving to make search work more like "natural language".

As usual, Seamus's observations are most thought provoking, and he's quite right to point out just how serviceable keyword based search has turned out to be for very many use cases. However, I wonder if he isn't rather glossing over the extraordinarily richness and precision of natural language in implying that we can dispense with it for search. My cursory reading around natural language syntax has left me with a strong suspicion that the fuzzy and interlinked hierarchical syntactical structures that researchers are devining within natural language syntax are what enable us to resolve the meaning of individual words within the complex webs of semantic and social contexts that we use to order our understanding and perceptions of the world (try typing that sentence into Google and see just what kind of insights it comes up with!).

Of course, web search predicated on the matching of isolated keywords to unstructured and semi-structured web pages does clearly provide massive value, as Google's extraordinary success proves. Unlike in the case of a pidgin speaker of a language, who might get their basic needs met to an extent with a few keywords and phrases, but who will struggle to understand much of the information they elicit with their questions, native speakers who use keyword based search are able to use their fluent language skills to sift through and analyse the rich language of their search results. Also, it turns out that very many combinations of two or more keywords are sufficient to narrow down search results with at least a tolerable degree of accuracy.

However, when it comes to a question like "what highbrow movies that my friends rate are showing on Sunday evening in the town where I am holidaying?", keywords alone are clearly never going to cut the search mustard. The rich set of highly-personal social and semantic contexts that this phrase evokes implies the need for a very different approach to search than that of brute keyword crunching, if we are to make search into something that is not only intuitive but also truly personalised.

So, while keywords have enabled the creation of incredibly valuable search services like Google, let's not throw the rich-personalised search baby out with the natural language bathwater. To understand natural language would be to go a long way to understanding how we perceive and understand our world—and that is surely going to be key for the continued evolution of search.

STOP PRESS:

The friend I had arranged to meet at the British Library turned up at the British Museum (we spent ten minutes phoning each other trying to work out why each couldn't find the other!). Oh well... ; )