Log In Sign Up

netdania.com
robots.txt

Robots Exclusion Standard data for netdania.com

Resource Scan

Scan Details

Site Domain netdania.com
Base Domain netdania.com
Scan Status Ok
Last Scan2024-11-15T01:05:51+00:00
Next Scan 2024-11-22T01:05:51+00:00

Last Scan

Scanned2024-11-15T01:05:51+00:00
URL https://www.netdania.com/robots.txt
Domain IPs 81.18.71.228
Response IP 81.18.71.228
Found Yes
Hash 6dd7152e79e3c658ac4c9d60cea5a88ca3f956df7d351a7613f92082fce57ebd
SimHash 6756d030c680

Groups

bubing

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /
Back to top

Comments

  • Stop BUbiNG
  • Stop BLEXBot Crawler - (http://webmeup-crawler.com/)
  • Stop MauiBot - (crawler.feedback+wc@gmail.com)
  • Stop PageFreezer - (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)
  • Stop MJ12Bot
  • Stop ADmantX
Back to top