mctw.com.tw
robots.txt

Robots Exclusion Standard data for mctw.com.tw

Resource Scan

Scan Details

Site Domain mctw.com.tw
Base Domain mctw.com.tw
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-11-13T05:38:45+00:00
Next Scan 2024-11-20T05:38:45+00:00

Last Successful Scan

Scanned2024-10-22T05:30:04+00:00
URL https://www.mctw.com.tw/robots.txt
Redirect https://www.marieclaire.com.tw/robots.txt
Redirect Domain www.marieclaire.com.tw
Redirect Base marieclaire.com.tw
Domain IPs 34.96.112.91
Redirect IPs 35.241.47.28
Response IP 35.241.47.28
Found Yes
Hash c9b51bda55f1c7c238449606a77ae8514fefa176d1d6978b2fce19724845b964
SimHash 2a96cc30c793

Groups

*

Rule Path
Disallow /preview/
Disallow /admin/
Disallow /ap/
Disallow /etc/
Disallow /tmp/
Disallow /ADdemo/
Disallow /channel/demo_view/
Disallow /mobile/
Disallow /talk/view/
Disallow /slide_content/
Disallow /slide/
Disallow /slice/
Disallow /share/
Disallow /share/fb/
Disallow /insight/
Disallow /ajax/
Disallow /api/

gptbot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.marieclaire.com.tw/sitemap.xml
sitemap https://www.marieclaire.com.tw/google-news.xml