WolframAlpha launched at 7pm CST yesterday, against much Twitter hype. If it wasn't tweeted by Peter Kim I probably would not have been interested in yet another search engine claiming to have Googly potential. I went through the WolframAlpha demonstration video, and my interest peaked. After having used it a bit with several adhoc as well as recommended search terms, I realise that, as a database of intentions, WolframAlpha is still a very young infant in the search business, albeit an infant with a potentially high IQ.
One example that started me on this unscientific evaluation was the search for 'Universities near Hounslow, UK', something a little generic, but still near-enough to natural language to be human readable and express the intention of my search. Given the video demo and instructions on the site, it also gave the impression that WolframAlpha would make mince meat of this search. It was very sad to see that it again gave me a 'Wolfram|Alpha isn't sure what to do with your input' screen.
Modifying the search to include a more popular location, 'Universities near Cambridge' resulted in a similar screen. By comparison, Google's seach result returned a mini-map from Google Maps with several univerisities listed, along with their names as website addresses. WolframAlpha failed this search 'litmus test' whereas Google shined brightly.
The second test used was 'September 11, 2001.' Indeed, under noteable events WolframAlpha did list 'World Trade Center destroyed' as one of its results, the result I was searching for. Google, by comparison also had the wikipedia entry for that sad day as its first link. Good litmus test results for both, nice!
A search for a more obscure date of 'September 14, 1979' produced more useful information in WolframAlpha though, since information on a noteable event was there, and the relevancy of its statistical information like moon-phase and sunrise/sunset times seemed a little more appropriate to the possible intention of the search. By comparison Google's relevancy ranking was a bit off to my intention, with links to an archived issue of Science Magazine being its first choice result. 'All technology stories on 14th September 1979 in the Guardian UK' was in the top ten results still around number 5, so still not too far off, given the obscure nature of the search and my unspecific intention with this search.
The tie-breaker for me was using a not-too-popular but still known date, UK Prime Minister Gordon Brown's birthday of 20 February 1951. Here Google floundered, the Wikipedia link to Gordon Brown was listed in the top 10, but was not number one. By comparison, in WolframAlpha's Noteable Events listing Gordon Brown's birthday was listed, along with actor Edward Albert. The Edward Albert birthday was a result that was NOT in Google's top ten search, but did seem to have a strong relevance to search intention that Google missed, even though it was not my search intention. This is a toss-up, but given my opinion that the intention of a search for a date would be noteable events. Given the missing of Edward Albert's birthday from the test results, I'm not sure if Google passed this litmus test.
One nitpick is that WolframAlpha's noteable events listing is always a bit of scroll down the page, the first result was actually the time difference from today. This is probably a tribute to its Mathematica underpinnings, since it is billed as more of a number-cruncher and statistical analyser than a relevancy-finder, which is more Google's forte.
The final litmus test I used was two stocks, 'MSFT AAPL'. That's Microsoft and Apple's stock names, for those unaware. Here WolframAlpha did work as advertised, bringing up very relevant information about each company, and a 'optimal portfolio return' for each. By comparison, Google continuously listed links to stock alerts for each from the previous day's trading. One additional result that came in with Google was an article from May 1st this year on 'What Microsoft Can Learn From Apple', something that may not have been my intention, but was still relevant, and a good link to me, as a business reader. If I were a stock analyst though, I'm not sure how I would have reacted. Both guys passed this litmus test, however where Google offered variety, WolframAlpha offered more purposeful computational knowledge.
However, this should not say WolframAlpha is a Google-killer (yet). Searches for less mathematically formulated queries, such as my name, or 'Ratio of Men To Women in London' seemed to confuse WolframAlpha's supposedly powerful natural language interpreter, just as my initial 'Universities in Hounslow, UK' search did. Google, on the other hand, had no problem understanding it, and directing the results of my search to the intention, which was the national statistics site for the UK.
So although the score so far seems to lead to WolframAlpha being gifted in quite a few ways, it still has a long way to go to even start to be in Google's league. I don't think is quite WolframAlpha's mission yet to be the successor to Google's crown as search engine king, but it should be the BHAG (Big Hairy Audacious Goal) of all search engines anyway.
As I write this, Google actually links 'BHAG' directly to the Wikipedia article on it, whereas WolframAlpha thought I was asking about somewhere in Pakistan. I guess that just reinforces my conclusion, that if you follow the script given, WolframAlpha works decently, however as a database of intentions, WolframAlpha has a long way to go still to reach the Google standard.
