Abstract
The traffic produced by the periodic crawling activities of Web robots often represents a good fraction of the overall websites traffic, thus causing some non-negligible effects on their performance. Our study focuses on the traffic generated on the SPEC website by many different Web robots, including, among the others, the robots employed by some popular search engines. This extensive investigation shows that the behavior and crawling patterns of the robots vary
significantly in terms of requests, resources and clients involved in their crawling activities. Some robots tend to concentrate their requests in short periods of time and follow some sorts of deterministic patterns characterized by multiple peaks. The requests of other robots exhibit a time dependent behavior and repeated patterns with some periodicity. We represent the traffic as a time series modelled in the frequency domain. The identified models, consisting of trigonometric polynomials and Auto Regressive Moving
Average components, accurately summarize the behavior of the overall traffic as well as the traffic of individual robots.
These models can be easily used as a basis for forecasting.
Lingua originale | English |
---|---|
Titolo della pubblicazione ospite | Proc. iiWAS2013 |
Pagine | 410-417 |
Numero di pagine | 8 |
DOI | |
Stato di pubblicazione | Pubblicato - 2013 |
Evento | The 15th International Conference on Information Integration and Web-based Applications & Services (iiWAS2013) - Vienna Durata: 2 dic 2013 → 4 dic 2013 |
Convegno
Convegno | The 15th International Conference on Information Integration and Web-based Applications & Services (iiWAS2013) |
---|---|
Città | Vienna |
Periodo | 2/12/13 → 4/12/13 |
Keywords
- Web log analysis
- Web robot
- Web traffic characterization
- crawling pattern,
- time series analysis