Log In Sign Up

cryptosnewss.com
robots.txt

Robots Exclusion Standard data for cryptosnewss.com

Resource Scan

Scan Details

Site Domain cryptosnewss.com
Base Domain cryptosnewss.com
Scan Status Ok
Last Scan2026-03-30T02:52:59+00:00
Next Scan 2026-04-06T02:52:59+00:00

Last Scan

Scanned2026-03-30T02:52:59+00:00
URL https://cryptosnewss.com/robots.txt
Domain IPs 65.108.108.217
Response IP 65.108.108.217
Found Yes
Hash c08d2d4205b4747d3ecd94d8e3b5bbf18cf2124b4fea0be611dd9449d6c2828f
SimHash 2974da626536

Groups

googlebot

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Product Comment
slurp Yahoo
Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandex

Rule Path
Allow /

sogou

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

chatgpt

Rule Path
Allow /

gptbot

Product Comment
gptbot OpenAI
Rule Path
Allow /

ccbot

Product Comment
ccbot Common Crawl
Rule Path
Allow /

perplexitybot

Rule Path
Allow /

claudebot

Product Comment
claudebot Anthropic
Rule Path
Allow /

facebookbot

Rule Path
Allow /

applebot

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /tag/
Disallow /author/
Disallow /archive/
Disallow /trackback/
Disallow /feed/
Disallow /search?
Disallow /*?s=
Disallow /*?replytocom=
Disallow /page/*/?s=*
Disallow /*?amp
Disallow /*?utm=*
Disallow /*%26amp%3D*
Allow /wp-admin/admin-ajax.php
Disallow /*.php$
Disallow /readme.html
Disallow /license.txt
Back to top

Other Records

Field Value
sitemap https://cryptosnewss.com/sitemap_index.xml
sitemap https://cryptosnewss.com/post-sitemap.xml
sitemap https://cryptosnewss.com/page-sitemap.xml
sitemap https://cryptosnewss.com/category-sitemap.xml
sitemap https://cryptosnewss.com/sitemap_index.xml
Back to top

Comments

  • ----------------
  • Google Crawlers
  • ----------------
  • ----------------
  • Bing & Other Search Engines
  • ----------------
  • ----------------
  • AI Crawlers (ChatGPT, Perplexity, etc.)
  • ----------------
  • Block irrelevant/thin/duplicate pages
  • Allow essential admin-ajax
  • Block direct crawling of script & system files
  • Crawl-delay (optional, uncomment if server load issues)
  • Crawl-delay: 10
  • Google News & Search sitemaps
Back to top

Warnings

  • 1 invalid line.
  • `https` is not a known field.
Back to top