Alan Orth 7566a79d90 dspace/config: update spider agent list ...

Update list of spider user agents from the COUNTER-Robots project.

See: https://github.com/atmire/COUNTER-Robots

2023-10-30 09:05:36 +03:00

..

agents

dspace/config: update spider agent list

2023-10-30 09:05:36 +03:00

domains

DS-1841: adding example files for agent and domain-based spider filtering, borrowed from OSU Libraries, with much thanks

2013-12-12 23:17:52 +00:00

iplists.com-altavista.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-excite.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-google.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-infoseek.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-inktomi.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-lycos.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00

iplists.com-misc.txt

[DS-440] Adjust SpiderDownloader to download multiple files in a "config/spiders" directory relative ${dspace.dir}

2010-02-07 16:42:56 +00:00