Log In Sign Up

scrape.com.br
robots.txt

Robots Exclusion Standard data for scrape.com.br

Resource Scan

Scan Details

Site Domain scrape.com.br
Base Domain scrape.com.br
Scan Status Ok
Last Scan2026-01-20T15:25:50+00:00
Next Scan 2026-01-27T15:25:50+00:00

Last Scan

Scanned2026-01-20T15:25:50+00:00
URL https://scrape.com.br/robots.txt
Redirect https://www.scrape.com.br/robots.txt
Redirect Domain www.scrape.com.br
Redirect Base scrape.com.br
Response IP 192.185.215.254
Found Yes
Hash 97f1192e3465657b23aaa05c0c45cdc38da6405ba3efb6e09103e6c1e85b24ef
SimHash b83f2c333ef2

Groups

uptimebot

Rule Path
Disallow /

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

wbsrch/1.1

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-httplib2

Rule Path
Disallow /

cipacrawler/3.0

Rule Path
Disallow /
Back to top

Comments

  • /robots.txt arquivo do site http://www.scrape.com.br/
Back to top