dlptips.com
robots.txt

Robots Exclusion Standard data for dlptips.com

Resource Scan

Scan Details

Site Domain dlptips.com
Base Domain dlptips.com
Scan Status Ok
Last Scan2025-07-11T21:16:52+00:00
Next Scan 2025-07-18T21:16:52+00:00

Last Scan

Scanned2025-07-11T21:16:52+00:00
URL https://dlptips.com/robots.txt
Domain IPs 2a05:f480:3400:208d:5400:5ff:fe50:c037, 66.245.195.49
Response IP 66.245.195.49
Found Yes
Hash c76a4d8146ec63adcaab05bf8c5fbccc28945a41a792fb7f69a1ec879215bcc5
SimHash 08a0ebaa42b6

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow */trackback/
Disallow */feed/
Disallow /wp-admin/remote
Allow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /
Disallow *?replytocom

*

Rule Path Comment
Disallow *?noamp=available -
Disallow *?amp -
Disallow /20*/ Blocks all year/month archive pages like /2025/04/

Comments

  • General directives for all crawlers
  • Block Baidu bots
  • Block magpie crawler