Log In Sign Up

zcrawler.com
robots.txt

Robots Exclusion Standard data for zcrawler.com

Resource Scan

Scan Details

Site Domain zcrawler.com
Base Domain zcrawler.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-06-18T04:12:14+00:00
Next Scan 2025-07-18T04:12:14+00:00

Last Successful Scan

Scanned2025-05-20T03:53:57+00:00
URL https://zcrawler.com/robots.txt
Domain IPs 109.234.167.14
Response IP 109.234.167.14
Found Yes
Hash 2098e68078237633c0a3bc51e034d7e86fd54158c6f5f6a01d3785b799aa35df
SimHash 4146d8d2f245

Groups

*

Rule Path
Disallow /wp-admin
Disallow /wp-includes
Disallow /wp-content/plugins
Disallow /wp-content/cache
Disallow /trackback
Disallow /feed
Disallow /comments
Disallow /category/*/*
Disallow */trackback
Disallow */feed
Disallow */comments
Disallow /*.pdf$
Disallow /*?*
Disallow /*?
Disallow /wp-login.php
Allow /wp-content/uploads

googlebot

Rule Path
Disallow /*.php$
Disallow /*.inc$
Disallow /*.gz$
Disallow /*.pdf$

googlebot-image

Rule Path
Disallow
Allow /*

mediapartners-google*

Rule Path
Disallow
Allow /*

ahrefssiteaudit

Rule Path
Allow /*

ahrefsbot

Rule Path
Allow /*
Disallow /wp-content/sabai/
Allow /wp-content/sabai/File/thumbnails/
Disallow /wp-content/plugins/sabai/
Disallow /wp-content/plugins/sabai-directory/
Back to top

Comments

  • On empeche l'indexation des dossiers sensibles
  • On desindexe toutes les URL ayant des parametres (duplication de contenu)
  • On desindexe la page de connexion (contenu inutile)
  • On autorise l'indexation des images
  • On empeche l'indexation des fichiers sensibles
  • Autoriser Google Image
  • Autoriser Google AdSense
  • Autoriser Ahrefs
Back to top