A Big Day for Blog Friends

I feel like a bit like a trapeze artist at the moment, arcing through the air between swings. (Admittedly I feel like a trapeze artist very definitely in a metaphorical sense only, as I put my back out yesterday and am hobbling around the flat!)

We took the current Blog Friends service down a few minutes ago, and are now working furiously to get Blog Friends v1 Beta ready for prime time—hopefully sometime later today.

So whether you are an existing or would-be user of Blog Friends, please bear with us: we very much hope the wait will be more than worthwhile.

And the view up here is amaaaaaaaaaazing! ; )

[Cross-posted from The Blog Friends Blog]

FOI automation—a distant pipedream

Andrew at IMPACT blog points us to this press release from the Information Commissioner's Office (my bold):

Freedom of Information is fast becoming a fixed feature of 21st century democracy and should not be seen as a battle ground between public bodies and the people, according to the Information Commissioner, Richard Thomas. Speaking at the annual FOI Live conference in London on 24 May, the Information Commissioner [stated] that the transparency and accountability brought by Freedom of Information reinforce good government, and should not be seen as a threat. However, he will also stress that those using FOI must act responsibly.

Readers will know that I believe that the trend towards information transparency is ineluctable, so I quite agree with the Information Commissioner on that point.

But what of the misuse of the FOI that his last sentence touches on? Of course, if search technologies could step up to automate the resolution of FOI requests properly (which it falls so far short of doing right now), the problem of human resources in government and business being stretched by nuisance FOI requests would disappear.

However, to achieve such automation, we would need to develop search technologies that can mediate the individual, richly-structured, socially and semantically contextualised ways people think and express themselves. And even Google is light years away from achieving that goal.

In the meantime, it is hard to see how government and business—and indeed the legal system—can avoid the overhead of having human beings sort through FOI requests written in natural language in order to make judgement calls over the validity of those requests.

Keywords, natural language and search

Seamus McCauley has written an interesting piece on "Keywords vs. 'natural language'":

Keyword-based search works so well, so intuitively, because if you watch people interact with other people you'll see them watch out for the keywords too. Google's trick wasn't to invent anything - it was just to pay attention to how we were doing things already and then train computers to do it too. Which really isn't good news for all the people striving to make search work more like "natural language".

As usual, Seamus's observations are most thought provoking, and he's quite right to point out just how serviceable keyword based search has turned out to be for very many use cases. However, I wonder if he isn't rather glossing over the extraordinarily richness and precision of natural language in implying that we can dispense with it for search. My cursory reading around natural language syntax has left me with a strong suspicion that the fuzzy and interlinked hierarchical syntactical structures that researchers are devining within natural language syntax are what enable us to resolve the meaning of individual words within the complex webs of semantic and social contexts that we use to order our understanding and perceptions of the world (try typing that sentence into Google and see just what kind of insights it comes up with!).

Of course, web search predicated on the matching of isolated keywords to unstructured and semi-structured web pages does clearly provide massive value, as Google's extraordinary success proves. Unlike in the case of a pidgin speaker of a language, who might get their basic needs met to an extent with a few keywords and phrases, but who will struggle to understand much of the information they elicit with their questions, native speakers who use keyword based search are able to use their fluent language skills to sift through and analyse the rich language of their search results. Also, it turns out that very many combinations of two or more keywords are sufficient to narrow down search results with at least a tolerable degree of accuracy.

However, when it comes to a question like "what highbrow movies that my friends rate are showing on Sunday evening in the town where I am holidaying?", keywords alone are clearly never going to cut the search mustard. The rich set of highly-personal social and semantic contexts that this phrase evokes implies the need for a very different approach to search than that of brute keyword crunching, if we are to make search into something that is not only intuitive but also truly personalised.

So, while keywords have enabled the creation of incredibly valuable search services like Google, let's not throw the rich-personalised search baby out with the natural language bathwater. To understand natural language would be to go a long way to understanding how we perceive and understand our world—and that is surely going to be key for the continued evolution of search.

STOP PRESS:

The friend I had arranged to meet at the British Library turned up at the British Museum (we spent ten minutes phoning each other trying to work out why each couldn't find the other!). Oh well... ; )

Team dream

Alan Graham writes about the continued role for humans in mediating online advertising; Euan Semple talks about how personal recommendations are often a better bet than search for finding good stuff online.

The thing is, people are still a whole lot better than machines at understanding the deeper meanings of information, by perceiving its resonance across a web of learned personal, cultural and social associations. In other words, people are able to imbue information with identity—much as they do one another.

On the other hand, algorithms don't need lunch breaks, don't get sick, moody or unmotivated, and don't kid themselves that they are being "objective" in their judgements when they have big chips on their shoulder and axes to grind (this latter representing a significant clouding of transparency in human communication).

Well ok then, how about a partnership, of people plus algorithms? A dream team in the game of information sharing and discovery! Oh bother, I guess a buzzword for that concept has already been trademarked.

The concept's realisation, on the other hand, is only just beginning to come about. This dizzying little video glosses the story so far: