mirror of
https://github.com/DSpace/DSpace.git
synced 2025-10-10 11:33:11 +00:00
13 lines
412 B
Plaintext
13 lines
412 B
Plaintext
# example spider filder by domain regular expressions courtesy of OSU Libraries
|
|
# https://raw.github.com/osulibraries/DSpace/osukb/dspace/config/Spiders-DomainName.txt
|
|
(.*)\.fastsearch\.net\.
|
|
(.*)\.scoutjet\.com\.
|
|
(.*)\.yahoo\.com\.
|
|
crawl(.*)\.exabot\.com\.
|
|
crawl-(.*)-googlebot\.com\.
|
|
crawler(.*)\.ask\.com\.
|
|
discobot-(.*)\.discoveryengine\.com\.
|
|
localhost\.
|
|
spider(.*)\.yandex\.ru\.
|
|
spider-(.*)\.yandex\.com\.
|