Detecting near replicas on the Web by content and hyperlink analysis

The presence of replicas or near-replicas of documents is very common on the Web. Whilst replication can improve information accessibility for the users, the presence of near-replicas can hinder the effectiveness of search engines. We propose a method to detect similar pages, in particular replicas and near-replicas, which is based on a pair of signatures. The first signature is obtained by a random projection of the bag-of-words representation of the page contents. The second signature, referred to as Hypelink Map, is computed by a recursive equation which exploits the connectivity among the Web pages to encode the page context. The experimental results show that on the given dataset replicas and near-replicas can be detected with a precision and recall of 93%.