Log In Sign Up

markitors.com
robots.txt

Robots Exclusion Standard data for markitors.com

Resource Scan

Scan Details

Site Domain markitors.com
Base Domain markitors.com
Scan Status Ok
Last Scan2025-10-14T01:32:11+00:00
Next Scan 2025-11-13T01:32:11+00:00

Last Scan

Scanned2025-10-14T01:32:11+00:00
URL https://markitors.com/robots.txt
Domain IPs 104.21.5.51, 172.67.132.252, 2606:4700:3032::ac43:84fc, 2606:4700:3037::6815:533
Response IP 104.21.5.51
Found Yes
Hash c59c8c425581ce78d700b4f3573d08371ffc11595adf6807a7fee9b86a484f8b
SimHash 22c65a1264a1

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-json/
Disallow /*.js$
Disallow /*.css$
Disallow /*.json$
Disallow /*.pdf$
Disallow /tag/
Disallow /category/
Disallow /?s=
Back to top

Other Records

Field Value
sitemap https://markitors.com/sitemap_index.xml
Back to top

Comments

  • Block access to core WordPress directories
  • Block unnecessary file types to optimize crawl budget
  • Disallow duplicate or low-value pages
  • Sitemap to guide search engines for efficient crawling
Back to top