auto.epochtimes.com
robots.txt

Robots Exclusion Standard data for auto.epochtimes.com

Resource Scan

Scan Details

Site Domain auto.epochtimes.com
Base Domain epochtimes.com
Scan Status Ok
Last Scan2024-10-26T17:59:22+00:00
Next Scan 2024-11-25T17:59:22+00:00

Last Scan

Scanned2024-10-26T17:59:22+00:00
URL https://auto.epochtimes.com/robots.txt
Domain IPs 45.79.178.193
Response IP 45.79.178.193
Found Yes
Hash d63e92e1a93f1f6ba8afd7daa5ce9d3841a7ee82b655727d95ff9d0879a487c3
SimHash b68d28ad60f0

Groups

*

Rule Path
Disallow /to_my_saved_search*
Disallow /to_my_list*
Disallow /usedcar/to_my_saved_search*
Disallow /usedcar/to_my_list*
Disallow /usedcar1*

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Crawl-delay: 20