consagra.com.br
robots.txt

Robots Exclusion Standard data for consagra.com.br

Resource Scan

Scan Details

Site Domain consagra.com.br
Base Domain consagra.com.br
Scan Status Ok
Last Scan2024-11-16T01:13:43+00:00
Next Scan 2024-11-23T01:13:43+00:00

Last Scan

Scanned2024-11-16T01:13:43+00:00
URL https://www.consagra.com.br/robots.txt
Domain IPs 108.157.254.104, 108.157.254.120, 108.157.254.21, 108.157.254.44
Response IP 108.157.254.44
Found Yes
Hash b40ec44665ffab598ff2710f58d62122df7cd1d93b534bf46740df6f6d02af00
SimHash 6254da65f421

Groups

*
googlebot
googlebot-image

Rule Path
Allow /
Disallow /conta/*
Disallow /carrinho/*
Disallow /buscar
Disallow /documentacao
Disallow /api/*
Disallow /*fq%3D*
Disallow /*q%3D*
Disallow /*sort%3D*
Disallow /compre_junto/*
Disallow /_events/*
Disallow /tracking/convertion
Disallow /static/*
Disallow /store/*

Other Records

Field Value
crawl-delay 10

dirbuster-0.12
baiduspider
baiduspider-video
baiduspider-image
baiduspider+
twengabot-discover
twengabot
twengabot-2.0
mj12bot
bdcbot
spbot
linkpadbot
wbsearchbot
addthis.com robot tech.support@clearspring.com
addthis.com
exabot
yandeximages
yandex
slurp
spbot
ahrefsbot
everyonesocialbot
blexbot
seekport crawler
geedobot
gptbot
chatgpt-user
claude-web
claudebot
anthropic-ai
cohere-ai
bytespider
perplexitybot
diffbot
scrapy
magpie-crawler
ccbot
omgili
omgilibot
node/simplecrawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.consagra.com.br/sitemap.xml