ehubt.io
robots.txt

Robots Exclusion Standard data for ehubt.io

Resource Scan

Scan Details

Site Domain ehubt.io
Base Domain ehubt.io
Scan Status Ok
Last Scan4/24/2025, 9:08:40 AM
Next Scan 5/24/2025, 9:08:40 AM

Last Scan

Scanned4/24/2025, 9:08:40 AM
URL https://ehubt.io/robots.txt
Domain IPs 104.21.10.19, 172.67.162.31, 2606:4700:3032::ac43:a21f, 2606:4700:3036::6815:a13
Response IP 172.67.162.31
Found Yes
Hash 2327569c5d27fca5b6defc6cbb79ace2622a5b36194e98b4fbf03d4d8bf2e1ef
SimHash 6db444fa55a9

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /xmlrpc.php
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Allow /sitemap.xml
Allow /feed/
Disallow /*.php
Disallow /*.pdf
Disallow /*.doc
Disallow /*.xls
Disallow /*.ppt
Disallow /*.jpg
Disallow /*.gif
Disallow /*.png
Allow /category/
Allow /tag/
Allow /search/
Disallow /?

Comments

  • Allow search engines to crawl specific pages
  • Disallow certain types of files
  • Allow search engines to crawl specific directories or pages
  • Disallow crawling of dynamic content