Log In Sign Up

applebyme.cn
robots.txt

Robots Exclusion Standard data for applebyme.cn

Resource Scan

Scan Details

Site Domain applebyme.cn
Base Domain applebyme.cn
Scan Status Ok
Last Scan2024-09-26T15:00:15+00:00
Next Scan 2024-10-26T15:00:15+00:00

Last Scan

Scanned2024-09-26T15:00:15+00:00
URL https://www.applebyme.cn/robots.txt
Domain IPs 180.163.146.99
Response IP 180.163.146.99
Found Yes
Hash cb344a383e8d9b08945600e6d7861379a04d5a2e744458160ca3b617e171f5d2
SimHash 08d41ed56454

Groups

*

Rule Path
Disallow /aspx/
Disallow /admin/
Disallow /content/*
Disallow /assets/*
Disallow /account/

trendkite-akashic-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

yisouspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

cliqzbot

Rule Path
Disallow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Allow /

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top

Warnings

  • `request-rate` is not a known field.
Back to top