Log In Sign Up

preferente.com
robots.txt

Robots Exclusion Standard data for preferente.com

Resource Scan

Scan Details

Site Domain preferente.com
Base Domain preferente.com
Scan Status Ok
Last Scan2024-09-19T17:32:03+00:00
Next Scan 2024-10-19T17:32:03+00:00

Last Scan

Scanned2024-09-19T17:32:03+00:00
URL https://preferente.com/robots.txt
Redirect https://www.preferente.com/robots.txt
Redirect Domain www.preferente.com
Redirect Base preferente.com
Domain IPs 46.105.198.23
Redirect IPs 46.105.198.23
Response IP 46.105.198.23
Found Yes
Hash e5b171c2527f3a2a6de996e00fe2964a4cf8da4de596fab3f532eb1faabf22f4
SimHash 2a1a9d1a40f0

Groups

magpie-crawler

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Disallow /search/
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.gz$
Disallow /*.wmv$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*rurl%3D*
Allow /sitemap.xml.gz$
Allow /index.php$

Other Records

Field Value
crawl-delay 180

duggmirror

Rule Path
Disallow /

mediapartners-google

Rule Path
Allow /
Back to top

Comments

  • iRobots.txt SEO
  • All Bots
  • Disallow: /wp-content/
  • Disallow: /*/feed*
  • Disallow: /*?
  • Disallow: /*.css$
  • Allow: /wp-content/uploads/
  • Dugg Mirror
  • Google AdSense
  • Sitemap
  • YOUR WEBSITE DOES NOT HAVE A SITEMAP! Please consider
  • installing an automated sitemap generator such as
  • Google XML Sitemaps -
  • http://www.arnebrachhold.de/redir/sitemap-home/
  • Robots.txt file generated by iRobots.txt SEO v1.1.2
  • by Mark Beljaars
  • _ _ _ _ | |_ _ |. _ _ _ _ _ _ _ _
  • | | |(_|| |< |_)(/_||(_|(_|| _\.(_(_)| | |
  • _|
  • http://markbeljaars.com/plugins/irobotstxt-seo
  • Note:
  • The Allow directive and wildcards (*) in filenames are
  • not standard robots.txt syntax, however they are
  • supported by most search engines.
Back to top

Warnings

  • 2 invalid lines.
Back to top