howtobuybitcoin101.com
robots.txt

Robots Exclusion Standard data for howtobuybitcoin101.com

Resource Scan

Scan Details

Site Domain howtobuybitcoin101.com
Base Domain howtobuybitcoin101.com
Scan Status Ok
Last Scan2024-09-30T02:23:15+00:00
Next Scan 2024-10-07T02:23:15+00:00

Last Scan

Scanned2024-09-30T02:23:15+00:00
URL https://howtobuybitcoin101.com/robots.txt
Redirect https://www.howtobuybitcoin101.com/robots.txt
Redirect Domain www.howtobuybitcoin101.com
Redirect Base howtobuybitcoin101.com
Domain IPs 2406:da18:880:3801::c8, 2406:da18:880:3802::c8, 46.137.195.11, 52.74.166.77
Redirect IPs 13.228.199.255, 13.251.96.10, 2406:da18:880:3800::c8, 2406:da18:b3d:e202::64
Response IP 52.74.166.77
Found Yes
Hash 72b54d9d71a5d6f3172701dcea4011a0c9f05f2d77c2dc33ec5d794ce7917004
SimHash 23174b71c601

Groups

*

Rule Path Comment
Disallow /private/ Exclude any private directories
Disallow /tmp/ Exclude temporary directories
Disallow /cgi-bin/ Exclude CGI scripts directory
Disallow /images/ Exclude images directory if you want to
Allow / -

Other Records

Field Value
sitemap https://www.howtobuybitcoin101.com/sitemap.xml

Comments

  • Allow crawling of the entire website
  • Sitemap location