DSpace

mirror of https://github.com/DSpace/DSpace.git synced 2025-10-09 19:13:18 +00:00

Files

Alan Orth abad6116cb DS-4587: Update the spider user agent file

This updates the included spider user agent file to the latest from
the COUNTER-Robots project. DSpace's own copy is over five years old
and is missing a bunch of new patterns, which greatly decreases the
accuracy of the Solr usage statistics.

Port of #3333 to main for DSpace 7.x.

See: https://jira.lyrasis.org/browse/DS-4587
See: https://github.com/atmire/COUNTER-Robots/releases/tag/2021-07-05

2021-09-04 21:24:16 +03:00

agents

DS-4587: Update the spider user agent file

2021-09-04 21:24:16 +03:00

domains

DS-1841: adding example files for agent and domain-based spider filtering, borrowed from OSU Libraries, with much thanks

2013-12-12 23:17:52 +00:00

iplists.com-altavista.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-excite.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}