refetter.com
robots.txt

Robots Exclusion Standard data for refetter.com

Resource Scan

Scan Details

Site Domain refetter.com
Base Domain refetter.com
Scan Status Ok
Last Scan2024-11-02T18:48:19+00:00
Next Scan 2024-11-09T18:48:19+00:00

Last Scan

Scanned2024-11-02T18:48:19+00:00
URL https://refetter.com/robots.txt
Redirect https://insta.refetter.com/robots.txt
Redirect Domain insta.refetter.com
Redirect Base refetter.com
Domain IPs 104.21.91.42, 172.67.209.208, 2606:4700:3030::ac43:d1d0, 2606:4700:3035::6815:5b2a
Redirect IPs 104.21.91.42, 172.67.209.208, 2606:4700:3030::ac43:d1d0, 2606:4700:3035::6815:5b2a
Response IP 104.21.91.42
Found Yes
Hash 86cdf82483c35a15ef18e778ccfa07bcc9ee5051cd9bbe8144e5fe5beb3fdba9
SimHash 582c8172eb30

Groups

blexbot

Rule Path
Disallow /

steeler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /media/

gptbot

Rule Path
Disallow /

*

Rule Path
Allow /ads.txt
Disallow