Log In Sign Up

jogarquiz.com
robots.txt

Robots Exclusion Standard data for jogarquiz.com

Resource Scan

Scan Details

Site Domain jogarquiz.com
Base Domain jogarquiz.com
Scan Status Ok
Last Scan2026-01-29T06:50:56+00:00
Next Scan 2026-02-05T06:50:56+00:00

Last Scan

Scanned2026-01-29T06:50:56+00:00
URL https://jogarquiz.com/robots.txt
Domain IPs 104.26.2.142, 104.26.3.142, 172.67.74.199, 2606:4700:20::681a:28e, 2606:4700:20::681a:38e, 2606:4700:20::ac43:4ac7
Response IP 104.26.3.142
Found Yes
Hash 406c16dfcdfc39ac9bf9a201dfc09949a7c39c2926260f38dc3cbf4165b59324
SimHash 4f09c871c200

Groups

*

Rule Path
Disallow /url?q=
Disallow /*?*ajax=1*
Disallow /index.php/
Disallow /*.js$
Disallow /*.css$

mediapartners-google

Rule Path
Allow /

gptbot

Rule Path
Allow /llms.txt
Allow /llms-full.txt
Disallow /

anthropic-ai

Rule Path
Allow /llms.txt
Allow /llms-full.txt
Disallow /

amazonbot

Rule Path
Allow /llms.txt
Allow /llms-full.txt
Disallow /

googlebot

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

exabot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /en/tags/
Allow /
Allow /articles/
Allow /products/
Disallow /admin/
Disallow /api/
Disallow /private/

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://buzzfun.me/sitemap-buzzfun/sitemap.xml
Back to top

Comments

  • 针对 Meta 爬虫的专属规则（优先级高于全局规则）
  • 设置抓取间隔：每 5 秒抓取 1 次，可根据服务器性能调整（单位：秒）
  • 允许抓取标签页面（对应你收到请求的 /pl/tags 路径）
  • 允许抓取网站首页及公开内容页（按需扩展）
  • 禁止 Meta 爬虫抓取敏感路径（与全局规则保持一致，双重保障）
Back to top

Warnings

  • `llm-content` is not a known field.
  • `llm-full-content` is not a known field.
Back to top