Log In Sign Up

goohardasht.com
robots.txt

Robots Exclusion Standard data for goohardasht.com

Resource Scan

Scan Details

Site Domain goohardasht.com
Base Domain goohardasht.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2026-01-21T09:22:40+00:00
Next Scan 2026-02-04T09:22:40+00:00

Last Successful Scan

Scanned2026-01-06T08:03:25+00:00
URL https://goohardasht.com/robots.txt
Redirect https://www.gohardasht.ir/robots.txt
Redirect Domain www.gohardasht.ir
Redirect Base gohardasht.ir
Domain IPs 104.21.4.9, 172.67.131.113, 2606:4700:3034::ac43:8371, 2606:4700:3036::6815:409
Redirect IPs 46.100.61.30
Response IP 46.100.61.30
Found Yes
Hash de95a0bd5ddc28151d6869a055ba9718fd7dc3e3f11d9d486fb33a22b940ff92
SimHash 7d1ccc42e7f3

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

dotbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

imagesiftbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

sqlmap

Rule Path
Disallow /

masscan

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

httrack

Rule Path
Disallow /

download demon

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

nutch

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Disallow /config/
Disallow /data/
Disallow /logs/
Disallow /private/
Disallow /temp/
Disallow /cgi-bin/
Disallow /xmlrpc.php
Back to top

Comments

  • ============================
  • MAIN SEARCH ENGINES (ALLOWED)
  • ============================
  • ============================
  • SEO CRAWLERS (ALLOWED)
  • ============================
  • ============================
  • AI CRAWLERS (ALLOWED)
  • ============================
  • ============================
  • SOCIAL PREVIEW BOTS (ALLOWED)
  • ============================
  • ============================
  • BAD / SCANNER BOTS (BLOCKED)
  • (They will likely ignore this, but it's still good practice)
  • ============================
  • ============================
  • DEFAULT RULES FOR EVERYONE
  • Only protect sensitive areas
  • ============================
Back to top