cristalsex.com
robots.txt

Robots Exclusion Standard data for cristalsex.com

Resource Scan

Scan Details

Site Domain cristalsex.com
Base Domain cristalsex.com
Scan Status Ok
Last Scan5/23/2025, 12:57:16 AM
Next Scan 6/22/2025, 12:57:16 AM

Last Scan

Scanned5/23/2025, 12:57:16 AM
URL https://cristalsex.com/robots.txt
Redirect https://www.cristalsex.pro/robots.txt
Redirect Domain www.cristalsex.pro
Redirect Base cristalsex.pro
Domain IPs 104.21.94.159, 172.67.137.206, 2606:4700:3034::6815:5e9f, 2606:4700:3034::ac43:89ce
Redirect IPs 104.21.67.165, 172.67.178.124, 2606:4700:3036::ac43:b27c, 2606:4700:3037::6815:43a5
Response IP 104.21.67.165
Found Yes
Hash bccb97698bddf4c8a063804daf56a15dfc1695959b21dc60a087d9987a33f998
SimHash 500c0842c8b2

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

ia_archiver

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.cristalsex.pro/sitemap.xml