hempmedsbr.com
robots.txt

Robots Exclusion Standard data for hempmedsbr.com

Resource Scan

Scan Details

Site Domain hempmedsbr.com
Base Domain hempmedsbr.com
Scan Status Ok
Last Scan2024-11-07T09:14:08+00:00
Next Scan 2024-12-07T09:14:08+00:00

Last Scan

Scanned2024-11-07T09:14:08+00:00
URL https://hempmedsbr.com/robots.txt
Domain IPs 162.159.136.54, 162.159.137.54, 2606:4700:7::a29f:8836, 2606:4700:7::a29f:8936
Response IP 162.159.136.54
Found Yes
Hash a0296f4e2f941a12cd2ce5181d29df6dd9c3eddc160647c8a2eb03eb46252d23
SimHash 68595deafa08

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/load-scripts.php
Allow /wp-admin/load-styles.php
Disallow /wp-admin/
Disallow /wp-signup.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /archive/
Disallow /activate/
Disallow /cgi-bin/
Disallow /dropbox
Disallow /extend/plugins/search.php
Disallow /extend/themes/search.php
Disallow /feed/
Disallow /mshots/v1/
Disallow /next/
Disallow /plugins/search.php
Disallow /press-this.php
Disallow /public.api/
Disallow /remote-login.php
Disallow /search
Disallow /support/rss
Disallow /support/search.php
Disallow /themes/search.php
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /wp
Disallow

slurp

Rule Path
Disallow

msnbot

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

psbot
baiduspider

Rule Path
Disallow /

baiduspider-imagem

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baidu

Rule Path
Disallow /
Disallow /

yandeximages

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://hempmedsbr.com/sitemap_index.xml

Comments

  • Não rastreie páginas de subcategorias que são classificadas ou filtradas.
  • Baiduspider
  • Parâmetro de atraso de rastreamento: número de segundos para esperar entre solicitações sucessivas para o mesmo servidor.
  • Defina uma taxa de rastreamento personalizada se estiver tendo problemas de tráfego com seu servidor.
  • Crawl-delay: 30

Warnings

  • 1 invalid line.
  • `agente do usuã¡rio` is not a known field.