Web-collaborative filtering: recommending music by crawling the Web

A CF spider is described that collects from the Web lists of semantically related entities that can be used by existing CF algorithms by encoding them as `pseudo-users' and collecting useful data.

Abstract

We show that it is possible to collect data that are useful for collaborative filtering (CF) using an autonomous Web spider. In CF, entities are recommended to a new user based on the stated preferences of other, similar users. We describe a CF spider that collects from the Web lists of semantically related entities. These lists can then be used by existing CF algorithms by encoding them as `pseudo-users'. Importantly, the spider can collect useful data without pre-programmed knowledge about the format of particular pages or particular sites. Instead, the CF spider uses commercial Web-search engines to find pages likely to contain lists in the domain of interest, and then applies previously proposed heuristics to extract lists from these pages. We show that data collected by this spider are nearly as effective for CF as data collected from real users, and more effective than data collected by two plausible hand-programmed spiders. In some cases, autonomously spidered data can also be combined with actual user data to improve performance.