Log In Sign Up

mthree.com
robots.txt

Robots Exclusion Standard data for mthree.com

Resource Scan

Scan Details

Site Domain mthree.com
Base Domain mthree.com
Scan Status Ok
Last Scan2025-09-18T22:07:13+00:00
Next Scan 2025-10-18T22:07:13+00:00

Last Scan

Scanned2025-09-18T22:07:13+00:00
URL https://mthree.com/robots.txt
Domain IPs 104.26.6.103, 104.26.7.103, 172.67.68.247, 2606:4700:20::681a:667, 2606:4700:20::681a:767, 2606:4700:20::ac43:44f7
Response IP 104.26.6.103
Found Yes
Hash 7fddb971a12c768445252a881e4a2307487d7bfe3f26aa91541aea69e90de5d1
SimHash e85c9a50c114

Groups

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

msnbot/2.0b

Rule Path
Allow /

Other Records

Field Value
crawl-delay 100

googlebot

Rule Path
Allow /

sensis web crawler

Rule Path
Allow /

sensis.com.au web crawler

Rule Path
Allow /

*

Rule Path
Allow /

sitebot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yeti

Rule Path
Allow /

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://mthree.com/sitemap.xml
Back to top

Comments

  • Yahoo!
  • Bing
  • Google
  • Sensis
  • Sensis.com.au
  • All other agents will not spider
Back to top