hashemian.com
robots.txt

Robots Exclusion Standard data for hashemian.com

Resource Scan

Scan Details

Site Domain hashemian.com
Base Domain hashemian.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-20T23:27:35+00:00
Next Scan 2025-10-27T23:27:35+00:00

Last Successful Scan

Scanned2024-03-12T18:13:24+00:00
URL https://hashemian.com/robots.txt
Redirect https://www.hashemian.com/robots.txt
Redirect Domain www.hashemian.com
Redirect Base hashemian.com
Domain IPs 104.21.1.184, 172.67.129.188, 2606:4700:3033::6815:1b8, 2606:4700:3036::ac43:81bc
Redirect IPs 104.21.1.184, 172.67.129.188, 2606:4700:3033::6815:1b8, 2606:4700:3036::ac43:81bc
Response IP 104.21.1.184
Found Yes
Hash b49b2c0715487a7a3c0356e7dc04c4bef27b303c528e381ca5aa9b386a8c2e6b
SimHash 823cd0f2eb8b

Groups

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

*

Rule Path
Disallow /market/
Disallow /ebay/

Other Records

Field Value
sitemap https://www.hashemian.com/sitemap.txt
sitemap https://www.hashemian.com/blog/wp-sitemap.xml

Comments

  • blocking bad bots
  • Disallow: /articles/2005/
  • Disallow: /articles/2006/
  • Disallow: /articles/2007/
  • Crawl-delay: 20