stlrealtors.com
robots.txt

Robots Exclusion Standard data for stlrealtors.com

Resource Scan

Scan Details

Site Domain stlrealtors.com
Base Domain stlrealtors.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-24T06:51:10+00:00
Next Scan 2024-12-23T06:51:10+00:00

Last Successful Scan

Scanned2023-02-09T20:14:03+00:00
URL https://stlrealtors.com/robots.txt
Redirect https://www.stlrealtors.com/robots.txt
Redirect Domain www.stlrealtors.com
Redirect Base stlrealtors.com
Domain IPs 104.26.0.31, 104.26.1.31, 172.67.75.107, 2606:4700:20::681a:11f, 2606:4700:20::681a:1f, 2606:4700:20::ac43:4b6b
Redirect IPs 104.26.0.31, 104.26.1.31, 172.67.75.107, 2606:4700:20::681a:11f, 2606:4700:20::681a:1f, 2606:4700:20::ac43:4b6b
Response IP 104.26.1.31
Found Yes
Hash cd8be2298347b7c196dc48b2f96fa31d96780e9768a0143dd4e2e9a862acc78d
SimHash ade89ac0e5b6

Groups

*

Rule Path
Disallow /*print%3Dpdf*

Other Records

Field Value
crawl-delay 5

Comments

  • ROBOTS.TXT
  • www.stlrealtors.com
  • Google
  • User-agent: Googlebot
  • Disallow:
  • Yahoo
  • User-agent: Slurp
  • Disallow:
  • Alta-Vista
  • User-agent: Scooter
  • Disallow:
  • Excite
  • User-agent: ArchitextSpider
  • Disallow:
  • InfoSeek
  • User-agent: UltraSeek
  • Disallow:
  • Lycos
  • User-agent: Lycos_Spider_(T-Rex)
  • Disallow:
  • LookSmart
  • User-agent: MantraAgent
  • Disallow:
  • Alltheweb
  • User-agent: FAST-WebCrawler
  • Disallow: