Log In Sign Up

autofix.co
robots.txt

Robots Exclusion Standard data for autofix.co

Resource Scan

Scan Details

Site Domain autofix.co
Base Domain autofix.co
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-19T03:55:26+00:00
Next Scan 2024-10-19T03:55:26+00:00

Last Successful Scan

Scanned2024-08-21T03:54:48+00:00
URL https://autofix.co/robots.txt
Domain IPs 149.28.223.215
Response IP 149.28.223.215
Found Yes
Hash 35da06cd84ed852d299535a2c399b63d89913fa2a7332150c70a260f83381f72
SimHash 245c59e30512

Groups

googlebot

Rule Path
Allow *.js
Allow *.css
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /search/
Disallow *.pdf
Disallow /page/

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /
Disallow /wp-admin
Disallow /*/feed/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /trackback/

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin
Disallow /wp-login.php
Disallow /trackback/
Disallow /wp-register.php
Back to top

Comments

  • Allow files critical for rendering
  • Allow AJAX - Do Not Remove
  • Prevent crawl-budget waste on search pages
  • Prevent private admin areas from being crawled
  • Prevent duplicate /feed/ pages from being crawled
  • Prevent login page crawls etc
  • Prevent register page crawls etc
  • Prevent Trackback Neg SEO
  • Disallow: *trafficonline.xyz*
  • Allow AJAX - Do Not Remove
  • Add all sitemaps
Back to top