Log In Sign Up

respostasaqui.com.br
robots.txt

Robots Exclusion Standard data for respostasaqui.com.br

Resource Scan

Scan Details

Site Domain respostasaqui.com.br
Base Domain respostasaqui.com.br
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-07T13:38:26+00:00
Next Scan 2024-12-06T13:38:26+00:00

Last Successful Scan

Scanned2024-05-11T06:00:37+00:00
URL https://respostasaqui.com.br/robots.txt
Domain IPs 2a02:4780:38:e8d6:1417:cf58:9ced:ca91, 77.37.75.114
Response IP 191.101.228.21
Found Yes
Hash eeec7168cdfe0f80d418c30fbfcbba0b0e3d7ef9689c960327c98bd4e80b55ed
SimHash ec3098522cd3

Groups

*

Rule Path
Disallow /user/*
Disallow /busca-resposta*
Disallow /megamozg*
Disallow /*megamozg
Disallow /cdn-cgi/l/email-protection
Allow /

semrushbot-sa

Rule Path Comment
Disallow / Semrush
Allow /ads.txt -

semrushbot

Rule Path Comment
Disallow / Semrush
Allow /ads.txt -

rogerbot

Rule Path Comment
Disallow / MOZ
Allow /ads.txt -

dotbot

Rule Path Comment
Disallow / MOZ
Allow /ads.txt -

blexbot

Rule Path Comment
Disallow / Webmeup.com
Allow /ads.txt -

spbot

Rule Path Comment
Disallow / Openlinkprofiler
Allow /ads.txt -

seodiver

Rule Path Comment
Disallow / SEOdiver
Allow /ads.txt -

dataprovider

Rule Path Comment
Disallow / DataProvider.com
Allow /ads.txt -

magpie-crawler

Rule Path Comment
Disallow / BrandWatch.com
Allow /ads.txt -

getintent crawler

Rule Path
Disallow /
Allow /ads.txt

grapeshot

Rule Path
Disallow

doubleverify

Rule Path
Disallow

white ops

Rule Path
Disallow

moatbot

Rule Path
Disallow

ias_crawler

Rule Path
Disallow

forensiq

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

leikibot

Rule Path
Disallow

baidu-yunguance-scanbot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-slabot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-perfbot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-vsbot(ce.baidu.com)

Rule Path
Disallow

seznambot

Rule Path
Disallow /
Allow /ads.txt

sogou web spider

Rule Path
Disallow /
Allow /ads.txt

baiduspider

Rule Path
Disallow /
Allow /ads.txt

naverbot

Rule Path
Disallow /
Allow /ads.txt

yeti

Rule Path
Disallow /
Allow /ads.txt

coccocbot-web

Rule Path
Disallow /
Allow /ads.txt

qwantify

Rule Path
Disallow /
Allow /ads.txt

exabot

Rule Path
Disallow /
Allow /ads.txt

linguee

Rule Path Comment
Disallow / Language tool
Allow /ads.txt -

surdotlybot

Rule Path Comment
Disallow / Sur.ly
Allow /ads.txt -

bubing

Rule Path Comment
Disallow / Bubing academic crawler
Allow /ads.txt -

twitterbot

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

facebot

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://respostasaqui.com.br/sitemap.xml
Back to top

Comments

  • Disallow Marketing bots
  • Disallow exotic search engine crawlers
  • Disallow other crawlers
  • Good bots whitelisting:
  • Other bots
  • Neticle Crawler v1.0 ( http://bot.neticle.hu/ ) https://bot.neticle.hu/ - brand monitoring
  • Mega https://megaindex.com/crawler - link indexer tool (supports directives in user-agent:*)
  • Obot - IBM X-Force service
  • SafeDNSBot (https://www.safedns.com/searchbot)
Back to top

Warnings

  • 3 invalid lines.
Back to top