mattrobinson.net
robots.txt

Robots Exclusion Standard data for mattrobinson.net

Resource Scan

Scan Details

Site Domain mattrobinson.net
Base Domain mattrobinson.net
Scan Status Ok
Last Scan2025-10-31T14:52:19+00:00
Next Scan 2025-11-30T14:52:19+00:00

Last Scan

Scanned2025-10-31T14:52:19+00:00
URL https://mattrobinson.net/robots.txt
Redirect https://www.mattrobinson.net/robots.txt
Redirect Domain www.mattrobinson.net
Redirect Base mattrobinson.net
Domain IPs 2600:1901:0:7b8a::, 35.185.44.232
Redirect IPs 2600:1901:0:7b8a::, 35.185.44.232
Response IP 35.185.44.232
Found Yes
Hash 4e50f2580ce8e8bc846ad6a6ca2c26460ec2328f87aeb9148779dad7eec690f9
SimHash 69a45b40a593

Groups

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.mattrobinson.net/sitemap.xml