asiatechdaily.com
robots.txt

Robots Exclusion Standard data for asiatechdaily.com

Resource Scan

Scan Details

Site Domain asiatechdaily.com
Base Domain asiatechdaily.com
Scan Status Ok
Last Scan2026-03-02T16:30:53+00:00
Next Scan 2026-03-09T16:30:53+00:00

Last Scan

Scanned2026-03-02T16:30:53+00:00
URL https://asiatechdaily.com/robots.txt
Domain IPs 104.21.3.128, 172.67.130.182, 2606:4700:3031::ac43:82b6, 2606:4700:3033::6815:380
Response IP 104.21.3.128
Found Yes
Hash 1118cfa95fd1702b2bbf02ab5a606342217ad54da41292057bb16c2948755f6c
SimHash 451e5e42e096

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

googlebot

Rule Path
Allow /

googleother

Rule Path
Allow /

google-extended

Rule Path
Allow /

gptbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

ccbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

claudebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

claude-web

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

bytespider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

applebot

Rule Path
Allow /
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-login.php
Disallow /?s=
Disallow /search/

Other Records

Field Value
crawl-delay 60

Other Records

Field Value
sitemap https://asiatechdaily.com/sitemap.xml
sitemap https://asiatechdaily.com/sitemap_index.xml

Comments

  • NOTE: Google ignores Crawl-delay; use Search Console/server rate limit
  • NOTE: Google ignores Crawl-delay
  • NOTE: Google ignores Crawl-delay