Abstract
In this study, we introduce two novel features: the consecutive sequential request ratio and standard deviation of page request depth, for improving the accuracy of malicious and non-malicious web crawler classification from static web server access logs with traditional data mining classifiers. In the first experiment we evaluate the new features on the classification of known well-behaved web crawlers and human visitors. In the second experiment we evaluate the new features on the classification of malicious web crawlers, unknown visitors, well-behaved crawlers and human visitors. The classification performance is evaluated in terms of classification accuracy, and F1 score. The experimental results demonstrate the potential of the two new features to improve the accuracy of data mining classifiers in identifying malicious and well-behaved web crawler sessions.
