How well does result relevance predict session satisfaction?

By incorporating additional properties of the query itself, this paper constructs a model which predicts user satisfaction even more accurately than relevance alone, and finds that this relationship is surprisingly strong.

Abstract

Per-query relevance measures provide standardized, repeatable measurements of search result quality, but they ignore much of what users actually experience in a full search session. This paper examines how well we can approximate a user's ultimate session-level satisfaction using a simple relevance metric. We find that thisrelationship is surprisingly strong. By incorporating additional properties of the query itself, we construct a model which predicts user satisfaction even more accurately than relevance alone.