Log In Sign Up

chunk.gg
robots.txt

Robots Exclusion Standard data for chunk.gg

Resource Scan

Scan Details

Site Domain chunk.gg
Base Domain chunk.gg
Scan Status Ok
Last Scan2025-12-16T22:20:20+00:00
Next Scan 2025-12-30T22:20:20+00:00

Last Scan

Scanned2025-12-16T22:20:20+00:00
URL https://chunk.gg/robots.txt
Domain IPs 2a10:9840:1:7200::, 2a10:9840:2:7300::, 31.43.172.0, 31.43.173.0
Response IP 31.43.172.0
Found Yes
Hash 8919f5db2ecd4c83bdc771f8b76c838420664c705e99fd4f0a5420fb4bb82425
SimHash 381fd112cf7b

Groups

archivebot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

applebot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

applebot-extended

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

oai-searchbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

chatgpt-user

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

qwantbot

Rule Path
Allow *

facebookexternalhit
facebookcatalog

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

meta-externalfetcher

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

meta-externalagent

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

meta-externalads

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

seznambot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

panascais-search-bot

Rule Path
Allow *

googlebot
googlebot-image
googlebot-news
googlebot-video
googlebot-mobile
google-inspectiontool
storebot-google

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

yandexbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

yeti

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

qwantify

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

ecosiabot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

startpagebot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

searx

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

kagibot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot
siteauditbot
splitsignalbot
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ocob
semrushbot-ft

Rule Path
Disallow /

dotbot
rogerbot

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

penthesilea

Rule Path
Disallow /

*

Rule Path
Allow *
Back to top

Other Records

Field Value
sitemap https://chunk.gg/sitemap-index.xml
Back to top

Comments

  • ===================================
  • Website: https://chunk.gg/
  • ===================================
  • ===================================
  • Allowed Crawlers
  • ===================================
  • Archive.org Configuration
  • Apple Crawler Configuration
  • Apple AI Crawler Configuration
  • OpenAI Search Configuration
  • OpenAI ChatGPT Configuration
  • Qwant Configuration
  • Facebook External Hit Configuration
  • Meta External Fetcher Configuration
  • Meta External Agent Configuration
  • Meta External Ads Configuration
  • Seznam Crawler Configuration
  • Panascais Search Configuration
  • Google Configuration
  • Bing Configuration
  • Yahoo! Configuration
  • DuckDuckGo Configuration
  • Yandex Configuration
  • Baidu Configuration
  • Naver Configuration
  • Qwant Configuration
  • Ecosia Configuration
  • Startpage Configuration
  • Searx Configuration
  • Kagi Configuration
  • ===================================
  • Disallowed Crawlers
  • ===================================
  • Google Extended Configuration
  • Common Crawl Configuration
  • OpenAI Configuration
  • ImageSift Configuration
  • Perplexity AI Configuration
  • Majestic 12 SEO Configuration
  • Ahrefs Configuration
  • SEMrush Configuration
  • SEOmoz Configuration
  • netEstate NE Configuration
  • Geedo Product Search Configuration
  • DataForSEO Configuration
  • Babbar Crawler Configuration
  • Petal Search Crawler Configuration
  • ===================================
  • Generic Bad Crawlers
  • ===================================
  • ===================================
  • Global Configuration
  • ===================================
Back to top