Log In Sign Up

drstth.com
robots.txt

Robots Exclusion Standard data for drstth.com

Resource Scan

Scan Details

Site Domain drstth.com
Base Domain drstth.com
Scan Status Ok
Last Scan2026-02-10T09:21:04+00:00
Next Scan 2026-03-12T09:21:04+00:00

Last Scan

Scanned2026-02-10T09:21:04+00:00
URL https://drstth.com/robots.txt
Domain IPs 104.21.4.192, 172.67.154.44, 2606:4700:3031::6815:4c0, 2606:4700:3036::ac43:9a2c
Response IP 104.21.4.192
Found Yes
Hash 135c05d94a36f4d9eb045c11f1d407e7783b633ed774845474a15c2ad423cebb
SimHash 650ed8c26502

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /_next/
Disallow /admin/
Allow /about
Allow /service
Allow /news
Allow /jobs

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

sogou spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.drstth.com/sitemap.xml
Back to top

Comments

  • Robots.txt for 星渊科技
  • https://www.drstth.com
  • Allow all crawlers
  • Sitemap location
  • Crawl-delay for polite crawling
  • Disallow admin or private areas (if any)
  • Allow specific important paths
  • Special rules for major search engines
Back to top

Warnings

  • 3 invalid lines.
Back to top