cosmoschaos.net
robots.txt

Robots Exclusion Standard data for cosmoschaos.net

Resource Scan

Scan Details

Site Domain cosmoschaos.net
Base Domain cosmoschaos.net
Scan Status Ok
Last Scan2024-09-28T02:04:03+00:00
Next Scan 2024-10-05T02:04:03+00:00

Last Scan

Scanned2024-09-28T02:04:03+00:00
URL https://cosmoschaos.net/robots.txt
Domain IPs 104.21.67.119, 172.67.221.227, 2606:4700:3030::6815:4377, 2606:4700:3031::ac43:dde3
Response IP 104.21.67.119
Found Yes
Hash bbac13cc5201caf95064d8469efd0bb80dce3cbe7fe34d5775b5d36f07003be6
SimHash 6b3cc253c321

Groups

*

Rule Path
Disallow /wp-
Disallow /lib/
Disallow /*.php
Allow /apps/*.php
Disallow /*.txt
Allow /ads.txt
Disallow /*.pdf
Disallow /*.zip
Disallow /*?action=
Disallow /*?*&action=
Disallow /*?doing_wp_cron=
Disallow /*?*&doing_wp_cron=
Disallow /*?s=
Disallow /*?*&s=
Disallow /*?orderby=
Disallow /*?*&orderby=
Disallow /*?order=
Disallow /*?*&order=
Disallow /*?lang=
Disallow /*?*&lang=
Allow /*?lang=en
Allow /*?*&lang=en
Allow /feed/?taxonomy=blogcat&term=on-sale&orderby=modified&order=DESC

adsbot-google

Rule Path
Disallow /wp-
Disallow /lib/

mediapartners-google

Rule Path
Disallow /wp-
Disallow /lib/

amazonbot

Rule Path
Disallow /wp-
Disallow /lib/

ia_archiver

Rule Path
Disallow /wp-
Disallow /lib/
Disallow /blog/
Disallow /*?s=
Disallow /*?*&s=
Disallow /*.php
Disallow /*.txt
Disallow /*.pdf
Disallow /*.zip

megalodon

Rule Path
Disallow /wp-
Disallow /lib/
Disallow /blog/
Disallow /*?s=
Disallow /*?*&s=
Disallow /*.php
Disallow /*.txt
Disallow /*.pdf
Disallow /*.zip

applebot-extended

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /novels/

oai-searchbot

Rule Path
Disallow /novels/

perplexitybot

Rule Path
Disallow /novels/

timpibot

Rule Path
Disallow /novels/

Other Records

Field Value
crawl-delay 5

smtbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://cosmoschaos.net/sitemap.xml
sitemap https://invest.cosmoschaos.net/sitemap-invest.xml
sitemap https://rose.cosmoschaos.net/sitemap-rose.xml

Comments

  • 広告
  • Alexa
  • Internet Archive
  • ML 学習
  • ML アクセス
  • 技術調査