This is Sci7’s review of an article by Lori Lorigo, Bing Pan, Helene Hembrooke, Thorsten Joachims, Laura Granka, and Geri Gay to be published in the Journal “Information Processing and Management”, these comments and observations are made on a corrected proof “Article in Press”. The authors are from Cornell and Stanford Universities, and the research was partially funded by Google.

The researchers used eye tracking technology to assess how users interpret the information on Google’s search results page. Their results indicate that gender and task do affect the way in which users evaluate search results as revealed both by their eye movements and other actions.

Key general findings (as selected by Sci7)

• Only in one fifth of cases were users observed to read the results in the order in which they are presented
• On average three of the results on a search engine results page were read
• No subjects went beyond the third results page.
• Most of the subjects viewed (not clicked) the first and second results quite equally.
• Subjects clicked on the first result most of the time.
  Though the fraction of users clicking the first result was clearly recorded by the researchers, this interesting statistic did not make the paper.
• On more than half of the search sessions, users chose to revise the query terms without clicking on any link.
  This emphasises the importance of information, such as the first 90 or so characters of the title, META description and URL, which are often included on the search engine results page.

This work utilises a simple and well established broad classification of the types of tasks that users may be trying to achieve when using a search engine, these classes are:

• Navigational - The search engine is being used to help a user get to a specific page that the user already knows exists and is aiming for
• Transactional - The search engine is used as a step in an online transaction eg. The user wants to make a purchase / reservation / send a message.
• Informational - The search engine is used assist a user find information that they are looking for.

The scale of this research, as with many studies on online behaviour appears tiny, this paper is based on the results derived from only 23 individuals, ~400 Google queries, and ~600 Google results pages.

The researchers wrote navigational and informational tasks to be completed by their student test subjects. Navigational tasks were to get to a particular page eg. “Find the homepage of Michael Jordan, the statistician”, and informational tasks required the discovery online of the answers to factual questions such as: “Where is the tallest mountain in NY located?”.

Two minutes were given to answer each query, eye tracking and transaction log data were recorded. Eye-tracking hardware and software was from A.S.L . Three different types of “looking” at ten regions of interest on the screen, corresponding to each of the natural search results on the results page were recorded. The visual interaction with each region being either a saccade (very brief look less than 50 ms), fixation (at least 200 ms on a region) , or a scanpath (a sequence of fixations) was recorded. Pupil dilation was also recorded as a measure of the interest a subject was taking in a particular page. The accuracy of the equipment did not enable determination of if the subject was reading the snippet, title, url or other element of each result. There was no mention in the paper of Google adverts, this is an odd omission.

ANOVA and Chi squared analysis was used to determine if results obtained for the different gender and task classes were significantly different. These statistical measurements do not take into account the sample size relative to the number of individuals conducting web searches in the real world when assessing relevance - any significance found only refers to the small experimental sample. While task success rates were not significantly different based on either task or gender, the manner in which the search engine results page was interpreted varied. Actions in which there was a significant (at the 0.01 level) difference between males and females included:

• Females clicked on the second link twice as often as males. (Males clicked the second link 7% of the time compared to females’ 14.5%.)
• Males viewed more results pages than females.
• Females repeatedly viewed the same result more often than men.
• Males were more likely to click on the lower ranking links (links 7,8,9,10) on a results page, these include those scrolling was required to reveal in the test environment, though males were more likely than females to follow link 6 too - though the difference wasn’t statistically significant - suggesting “Males are more likely to click lower ranking links”, is a more defensible conclusion, than “Males are more likely to click links they have scrolled to reveal”.

General additional observations included : “Males look at a greater number of abstracts and also spend more time examining the result page in general [than females]”.

In terms of differences between tasks, no “transactional” tasks were evaluated, while Sci7 can understand that it may have been impractical (expensive) to get test subjects to make an online purchase for example, and many other transactional tasks require registration / signup etc. the reasons for this omission should have been discussed in the paper. Such work could have been quite informative and could have looked at questions such as at what stage the different groups used search engines during the transaction.

The results for the navigational tasks were as intuitively expected:

“Intuitively, a greater proportion of time is spent on Google result pages for navigational tasks since navigational questions do not require much additional scrutiny on web documents outside of Google; once the appropriate query result is found, one immediately has the answer for the navigational query.”

The statements in the above quote are backed up by the statistical data presented.

Interestingly the paper currently under review refers to earlier work which estimated that in 2000, 1 in 28 of all web page views were of search engine results pages. Reference: Jansen, B. J., & Pooch, U. (2000). Web user studies: a review and framework for future work. Journal of the American Society of Information Science and Technology, 52(3), 235–246.

The academic home page of the first author Lori Lorigo (It is normal academic practice that the first author is the one who has done the bulk of the work):
http://www.hci.cornell.edu/people/lorigo.htm

Sci7 has emailed Lori Lorigo inviting her to comment on this article, specifically on:

1. The omission of any mention of Google Ads. Were users instructed not to click ads?. Where the ads present during the experiment?
2. Did the Google funding come with any conditions? - Is this the reason ads were not mentioned / discussed?
3. Why transactional queries were not investigated.
4. What is the fraction of users who clicked, the first link, the second link etc. ?

DOI link to reviewed paper: 10.1016/j.ipm.2005.10.001

At the time of writing the author’s home and publications page are the only results on Google for a search for the article title, suggesting Sci7 is the first to critically review or cite this work online:

Sci7’s web-optimization services are informed by an awareness of online trends as well as published and proprietary research.

Technorati Tags: Google | Google+Rank | Usability | Search.

This entry was posted on Thursday, December 8th, 2005 at 7:17 pm and is filed under Resources. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.