Log In Sign Up

base-search.net
robots.txt

Robots Exclusion Standard data for base-search.net

Resource Scan

Scan Details

Site Domain base-search.net
Base Domain base-search.net
Scan Status Ok
Last Scan2024-06-07T12:52:54+00:00
Next Scan 2024-07-07T12:52:54+00:00

Last Scan

Scanned2024-06-07T12:52:54+00:00
URL https://base-search.net/robots.txt
Domain IPs 129.70.12.222
Response IP 129.70.12.222
Found Yes
Hash c5daa41ee2aa965056fda4cb55e41ffea4c3b7f96c4c1edcdabacf1c8d02116e
SimHash 893484404796

Groups

*

Rule Path
Disallow /about/
Disallow /about/backup
Disallow /about/classes
Disallow /about/css
Disallow /about/download
Disallow /about/images
Disallow /about/js
Disallow /about/de/about_sources_date.php
Disallow /about/en/about_sources_date.php
Disallow /blog/base/date/
Disallow /blog/base/category/
Disallow /blog/baseoai/date/
Disallow /blog/baseoai/category/
Disallow /blog/roller-ui/
Disallow /conf
Disallow /Crypt
Disallow /Drivers
Disallow /interface
Disallow /js
Disallow /lang
Disallow /RecordDrivers
Disallow /sys
Disallow /xsl
Disallow /services/Ajax
Disallow /services/Browse
Disallow /services/Intern
Disallow /services/MyResearch
Disallow /services/Record
Disallow /services/Records
Disallow /Ajax
Disallow /Browse/Dewey
Disallow /Browse/Tags
Disallow /MyResearch
Disallow /Record
Disallow /services/Search/Ajax
Disallow /services/Search/Email
Disallow /services/Search/History
Disallow /services/Search/OpenSearch
Disallow /services/Search/Suggest
Disallow /services/Search/cache
Disallow /services/Search/xsl
Disallow /Search/Ajax
Disallow /Search/Email
Disallow /Search/History
Disallow /Search/OpenSearch
Disallow /Search/Results
Disallow /Search/Suggest
Disallow /Search/cache
Disallow /Search/xsl
Disallow /verboten4bots

Other Records

Field Value
crawl-delay 2

acoonbot
ahrefsbot
baiduspider
gptbot
dotbot
easouspider
exabot
ezooms
femtosearchbot
gptbot
litefinder
ltx71
megaindex.ru
mj12bot
nachobot
parsijoo
scrapy
sitebot
sogou
unwindfetchor
wbsearchbot
webcopier
webspider
yandexbot
yisouspider

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for www.base-search.net
  • last update: 2015-06-03
  • Es wird nur die Startseite und die Erweiterte Suche erlaubt!
  • Wie oft ein Crowler eine Abfrage senden darf in Sekunden (Angabe 1.5 auch möglich)
  • Disallow: /Browse
  • Bei der Notation: /index , werden sowohl die Datei /index.html als auch das Unterverzeichnis /index/. betroffen.
  • Disallow: /services/Search/Results
  • wegen der Detailseite ist der Zugriff auf die Trefferliste erlaubt (z.Z. Zugriff verboten)
  • Bots, die auf dieses fiktives Verzeichnis unerlaubt zugreifen, sollen mithilfe der Firewall gesperrt werden
  • unerwuenschte Bots ganz aussperren
Back to top

Warnings

  • 1 invalid line.
Back to top