curlytales.com
robots.txt

Robots Exclusion Standard data for curlytales.com

Resource Scan

Scan Details

Site Domain curlytales.com
Base Domain curlytales.com
Scan Status Ok
Last Scan2024-09-22T01:08:51+00:00
Next Scan 2024-09-29T01:08:51+00:00

Last Scan

Scanned2024-09-22T01:08:51+00:00
URL https://curlytales.com/robots.txt
Domain IPs 13.35.210.111, 13.35.210.36, 13.35.210.51, 13.35.210.64
Response IP 13.35.210.51
Found Yes
Hash 5997269d98c127b0cf7a78a98428561046f070df38db1755ff2f14a632414a91
SimHash 69171946e0a1

Groups

*

Rule Path
Allow /
Disallow /wp-admin/
Disallow /events/
Disallow /wp-content/sitemaps/pagination-sitemap.xml
Disallow /tag/*
Disallow */page/*
Disallow */?fbclid=*
Disallow */?_gl*
Disallow */?fb_*
Disallow *mode%3Dlist*
Disallow *mode%3Dgrid*
Disallow */search/*
Disallow */attachment/*
Disallow /?attachment*
Disallow */shutterstock_*
Disallow */whatsapp*
Disallow *untitled*
Disallow */?q=%2F*
Disallow */wp-includes/*
Disallow /?p=*
Disallow */wp-content/uploads/betting/*
Disallow */wp-content/uploads/wikipedia/*
Disallow *.php?id=*
Disallow *?*
Disallow *//*
Disallow */sponsored/*
Disallow */podcast/*

mazbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

Other Records

Field Value
sitemap https://curlytales.com/sitemap_index.xml