An extensive study of Web robots traffic

Maria Carla Calzarossa, Daniele Tessera, Maria Calzarossa, Luisa Massari

Risultato della ricerca: Contributo in libroContributo a convegno

5 Citazioni (Scopus)

Abstract

The traffic produced by the periodic crawling activities of Web robots often represents a good fraction of the overall websites traffic, thus causing some non-negligible effects on their performance. Our study focuses on the traffic generated on the SPEC website by many different Web robots, including, among the others, the robots employed by some popular search engines. This extensive investigation shows that the behavior and crawling patterns of the robots vary significantly in terms of requests, resources and clients involved in their crawling activities. Some robots tend to concentrate their requests in short periods of time and follow some sorts of deterministic patterns characterized by multiple peaks. The requests of other robots exhibit a time dependent behavior and repeated patterns with some periodicity. We represent the traffic as a time series modelled in the frequency domain. The identified models, consisting of trigonometric polynomials and Auto Regressive Moving Average components, accurately summarize the behavior of the overall traffic as well as the traffic of individual robots. These models can be easily used as a basis for forecasting.
Lingua originaleEnglish
Titolo della pubblicazione ospiteProc. iiWAS2013
Pagine410-417
Numero di pagine8
DOI
Stato di pubblicazionePubblicato - 2013
EventoThe 15th International Conference on Information Integration and Web-based Applications & Services (iiWAS2013) - Vienna
Durata: 2 dic 20134 dic 2013

Convegno

ConvegnoThe 15th International Conference on Information Integration and Web-based Applications & Services (iiWAS2013)
CittàVienna
Periodo2/12/134/12/13

Keywords

  • Web log analysis
  • Web robot
  • Web traffic characterization
  • crawling pattern,
  • time series analysis

Fingerprint

Entra nei temi di ricerca di 'An extensive study of Web robots traffic'. Insieme formano una fingerprint unica.

Cita questo