Log In Sign Up

netdania.com
robots.txt

Robots Exclusion Standard data for netdania.com

Resource Scan

Scan Details

Site Domain netdania.com
Base Domain netdania.com
Scan Status Ok
Last Scan2024-05-23T10:58:12+00:00
Next Scan 2024-05-30T10:58:12+00:00

Last Scan

Scanned2024-05-23T10:58:12+00:00
URL https://www.netdania.com/robots.txt
Domain IPs 81.18.71.228
Response IP 81.18.71.228
Found Yes
Hash 8eeb409d3d13cc76b77fe677aa99706a4a687e3c369c70777b33413d48844fe2
SimHash 615ed031e7d0

Groups

bubing

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

*

Rule Path
Disallow /QuoteList.asp
Disallow /UI/Controls/MasterPage/

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Stop BUbiNG
  • Stop BLEXBot Crawler - (http://webmeup-crawler.com/)
  • Stop MauiBot - (crawler.feedback+wc@gmail.com)
  • Stop PageFreezer - (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)
  • Stop MJ12Bot
  • Stop ADmantX
Back to top