Robot Hunter: or precisely what I thought I wouldn't be doing when I became a librarian
-
Upload
ucd-library -
Category
Technology
-
view
92 -
download
0
Transcript of Robot Hunter: or precisely what I thought I wouldn't be doing when I became a librarian
Leabharlann UCD
An Coláiste Ollscoile, Baile
Átha Cliath,
Belfield, Baile Átha Cliath 4,
Eire
UCD Library
University College Dublin,
Belfield, Dublin 4, Ireland
Robot hunter
Or, precisely what I thought I wouldn’t
be doing when I became a librarian
Joseph Greene
Research Repository Librarian
http://researchrepository.ucd.ie
Counting downloads
• Open Access repositories make science and
scholarship accessible, and we need to
demonstrate our value
• Simple question: how often are these papers
used? How many times have they been
downloaded?
Enter the Robot
• At least 18% of web requests are from robots
• Less than half can be accounted for by the five
main search engines
• At Research Repository UCD, 2/3rds of our
repository’s downloads are marked as web robots
What are you talking about?
Internet robot, Web robot, automated agent,
crawler, spider, bot: any programme that visits
websites and systematically retrieves information
from them
Good and bad
• Search engines, link verifiers, computer science
experiments
• Gathering content for spam, phishing and copycat
sites, artificially improving a website’s ranking
(spamdexing), looking for security holes, DDoS
attacks…………
‘And the noisy, nasty nuisance grew, ‘til
the villagers cried, “What can we do?”’
Detection methods:
• Blocking robots in real-time:
Turing tests
• Detecting later and removing
from statistics
Appropriate, but problematic methods
for repositories
• Excluding known robots by user-agent name
– Easily faked or omitted
• Excluding by IP address
– DHCP, and list is growing exponentially
• Usage pattern analysis: query rate and resources
requested
– Expensive to automate
• Machine learning: training decision trees, neural
nets and/or statistical systems
– Did you say expensive???
• Combined approaches
Effectiveness, and repository out-of-the-
box repository strategies
Strength
Robots detected by Recall (%) Precision (%)
No images requested 98.34 75.48
No referring site 96.27 52.25
List of IP addresses 69.29 99.40
HEAD method to access site 32.37 100.00
Agent name declared 26.56 100.00
Access only at night 24.48 50.43
Robots.txt file accessed 17.01 100.00
Time, σ (3s) 2.49 100.00
Time, average (1s) 2.49 75.00
DSpace uses IP addresses of known agents – much weaker than
in the benchmarking study
Effectiveness, and repository out-of-the-
box repository strategies
Strength
Robots detected by Recall (%) Precision (%)
No images requested 98.34 75.48
No referring site 96.27 52.25
List of IP addresses 69.29 99.40
HEAD method to access site 32.37 100.00
Agent name declared 26.56 100.00
Access only at night 24.48 50.43
Robots.txt file accessed 17.01 100.00
Time, σ (3s) 2.49 100.00
Time, average (1s) 2.49 75.00
Eprints filters based on number of hits from an IP address per day – similar to time based strategies in
the benchmarking study
Effectiveness, and repository out-of-the-
box repository strategies
Strength
Robots detected by Recall (%) Precision (%)
No images requested 98.34 75.48
No referring site 96.27 52.25
List of IP addresses 69.29 99.40
HEAD method to access site 32.37 100.00
Agent name declared 26.56 100.00
Access only at night 24.48 50.43
Robots.txt file accessed 17.01 100.00
Time, σ (3s) 2.49 100.00
Time, average (1s) 2.49 75.00
Centralised strategy: IRUS-UK
• Collects and filters statistics from 84 DSpace and
Eprints repositories
• COUNTER compliant usage statistics
• Robot exclusion:
– The COUNTER list of agent names
– All downloads from IP addresses where there are more than 200 downloads in a day from a repository
– Most downloads from IP addresses where there are more than 100 downloads in a day from a repository
• Work commissioned to investigate feasibility and
approach to adaptive filtering based on usage
behaviour
Sources by slide 1 Bill Gosper's Glider Gun in action—a variation of Conway's Game of Life. Johan G.
Bontes. <https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life#/media/File:Gospers_glider_gun.gif>
3, 6, 7 Doran, D.; Gokhale, S.S. Web robot detection techniques: overview and limitations. Data Mining and Knowledge Discovery (2011) 22:183-210. DOI:10.1007/s10618-010-0180-z
4 http://pixabay.com/static/uploads/photo/2015/05/31/12/09/wooden-791421_640.jpg
5 Bad Robot Productions logo. 2001-2008. <https://en.wikipedia.org/wiki/Bad_Robot_Productions#/media/File:Bad_Robot_Productions_logo.jpg>
6 Burroway, J., Loard, J. V. The Giant Jam Sandwich. 1972, Houghton Mifflin Harcourt.
8, 9, 10 Nick Geens, Johan Huysmans, Jan Vanthienen. Evaluation of Web Robot Discovery Techniques: A Benchmarking Study. Advances in Data Mining. Applications in Medicine, Web Mining, Marketing, Image and Signal Mining. Lecture Notes in Computer Science 4065, pp 121-130, 2006. DOI:10.1007/11790853_10
8 Diggory, Mark. SOLR Statistics. DSpace Wiki. <https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics>
9 Joint, Nicholas. [EP-tech] Re: Please change the way IRstats works. Eprints_tech mailing list 2011-10-13 <http://www.eprints.org/tech.php/15695.html>
11 IRUS-UK. <http://www.irus.mimas.ac.uk/participants/>