Log In Sign Up

proweblogs.com
robots.txt

Robots Exclusion Standard data for proweblogs.com

Resource Scan

Scan Details

Site Domain proweblogs.com
Base Domain proweblogs.com
Scan Status Ok
Last Scan2025-04-07T02:34:48+00:00
Next Scan 2025-04-14T02:34:48+00:00

Last Scan

Scanned2025-04-07T02:34:48+00:00
URL https://proweblogs.com/robots.txt
Redirect http://www.proweblogs.com/robots.txt
Redirect Domain www.proweblogs.com
Redirect Base proweblogs.com
Domain IPs 104.21.83.107, 172.67.223.118, 2606:4700:3030::6815:536b, 2606:4700:3036::ac43:df76
Redirect IPs 217.79.252.162
Response IP 217.79.252.162
Found Yes
Hash 1b36937fc64b04235ec92e508d057640ab54aa86fae1aa79f3fe09bf25623558
SimHash e03c674b53e5

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /stats/
Disallow /dh_
Disallow /about/legal-notice/
Disallow /about/copyright-policy/
Disallow /about/terms-and-conditions/
Disallow /contact/
Disallow /tag/
Disallow /wp-admin/

googlebot

Rule Path
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.css$
Disallow /*.gz$
Disallow /*.wmv$
Disallow /*.tar$
Disallow /*.tgz$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /2007/1*

googlebot-image

Rule Path
Allow /*

mediapartners-google*

Rule Path
Allow /*
Back to top

Comments

  • This rule means it applies to all user-agents
  • Disallow all directories and files within
  • Disallow: /wp-includes/
  • Disallow: /page/
  • Disallow all monthly archive pages
  • The Googlebot is the main search bot for google
  • Disallow all files ending with these extensions
  • Disallow Google from parsing indididual post feeds and trackbacks..
  • Disallow: */feed/
  • Disallow: */trackback/
  • Disallow all files with ? in url
  • Disallow: /*?*
  • Disallow: /*?
  • Disallow all archived monthlies
  • Disallow: /2006/0*
  • Disallow: /2007/0*
  • Disallow: /2006/1*
  • The Googlebot-Image is the image bot for google
  • Allow Everything
  • This is the ad bot for google
  • Allow Everything
Back to top