daily-sun.com
robots.txt

Robots Exclusion Standard data for daily-sun.com

Resource Scan

Scan Details

Site Domain daily-sun.com
Base Domain daily-sun.com
Scan Status Ok
Last Scan2024-06-29T22:20:07+00:00
Next Scan 2024-07-06T22:20:07+00:00

Last Scan

Scanned2024-06-29T22:20:07+00:00
URL https://daily-sun.com/robots.txt
Redirect https://www.daily-sun.com/robots.txt
Redirect Domain www.daily-sun.com
Redirect Base daily-sun.com
Domain IPs 104.17.214.2, 104.17.215.2, 2606:4700::6811:d602, 2606:4700::6811:d702
Redirect IPs 104.17.214.2, 104.17.215.2, 2606:4700::6811:d602, 2606:4700::6811:d702
Response IP 104.17.214.2
Found Yes
Hash b197a520c74db9e627d1712afbdbb11d8361d913d7395b890deb7fef93816545
SimHash 66457783c49b

Groups

*

Rule Path
Allow /
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /ads.txt
Disallow /cgi-bin/
Disallow /cgi-bin/*
Disallow /home
Disallow /index.php

googlebot-image

Rule Path
Allow /assets/news_images/

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /assets/news_images/

applebot

Rule Path
Allow /

yandex

Rule Path
Allow /

yandeximages

Rule Path
Allow /assets/news_images/

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

qwantify

Rule Path
Allow /

baiduspider

Rule Path
Allow /

baiduspider/2.0

Rule Path
Allow /

baiduspider-video

Rule Path
Allow /

baiduspider-image

Rule Path
Allow /

sogou spider

Rule Path
Allow /

sogou web spider

Rule Path
Allow /

sosospider

Rule Path
Allow /

sosospider+

Rule Path
Allow /

sosospider/2.0

Rule Path
Allow /

yodao

Rule Path
Allow /

youdao

Rule Path
Allow /

youdaobot

Rule Path
Allow /

youdaobot/1.0

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.daily-sun.com/sitemap.xml

Comments

  • Crawl daily-sun.com,
  • Popular chinese search engines