hablamosdegamers.com
robots.txt

Robots Exclusion Standard data for hablamosdegamers.com

Resource Scan

Scan Details

Site Domain hablamosdegamers.com
Base Domain hablamosdegamers.com
Scan Status Ok
Last Scan2024-11-09T09:30:16+00:00
Next Scan 2024-11-16T09:30:16+00:00

Last Scan

Scanned2024-11-09T09:30:16+00:00
URL https://hablamosdegamers.com/robots.txt
Domain IPs 156.67.64.186
Response IP 156.67.64.186
Found Yes
Hash bf0f98d5dd9a4e437e2f791ba5c31e64939cea2f5f2bf481c3ae677ab82b5c95
SimHash 64aec8522abb

Groups

openai-user-agent

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /?*
Disallow /*?s=
Disallow /*%26s%3D
Disallow /search/
Disallow /*?attachment_id=
Disallow /*/feed/
Disallow /*/rss/
Disallow /*/embed/
Disallow /comments/
Disallow /trackback/
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /*/*.js
Allow /*/*.css
Allow /wp-*.png
Allow /wp-*.jpg
Allow /wp-*.jpeg
Allow /wp-*.gif
Allow /wp-*.svg
Allow /wp-*.pdf

Other Records

Field Value
sitemap https://hablamosdegamers.com/sitemap_index.xml

Comments

  • Allow only OpenAI-User-Agent
  • Optional: Set a crawl delay to reduce server load
  • Crawl-delay: 10