Log In Sign Up

bigdautu.com
robots.txt

Robots Exclusion Standard data for bigdautu.com

Resource Scan

Scan Details

Site Domain bigdautu.com
Base Domain bigdautu.com
Scan Status Ok
Last Scan2025-04-26T17:32:03+00:00
Next Scan 2025-05-26T17:32:03+00:00

Last Scan

Scanned2025-04-26T17:32:03+00:00
URL https://bigdautu.com/robots.txt
Domain IPs 104.21.2.184, 172.67.129.140, 2606:4700:3031::6815:2b8, 2606:4700:3033::ac43:818c
Response IP 172.67.129.140
Found Yes
Hash 700706fcc4ce6d43c022e71ddb8cf247cfd8944ffaf8d475cc7065da166259f4
SimHash f74fd8424730

Groups

easouspider
ezooms
mj12bot
sitesucker
httrack
httrack website copier
teleport
teleportpro
emailcollector
emailsiphon
webbandit
webzip
webreaper
webstripper
web downloader
webcopier
offline explorer pro
offline commander
leech
websnake
blackwidow
http weazel

Rule Path
Disallow /

nutch

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • protect my site from HTTrack or other software's ripping?
Back to top

Warnings

  • 5 invalid lines.
Back to top