Log In Sign Up

bigbrothernetwork.com
robots.txt

Robots Exclusion Standard data for bigbrothernetwork.com

Resource Scan

Scan Details

Site Domain bigbrothernetwork.com
Base Domain bigbrothernetwork.com
Scan Status Ok
Last Scan2024-11-16T15:05:03+00:00
Next Scan 2024-11-23T15:05:03+00:00

Last Scan

Scanned2024-11-16T15:05:03+00:00
URL https://bigbrothernetwork.com/robots.txt
Domain IPs 162.159.134.42
Response IP 162.159.134.42
Found Yes
Hash 2f3225dbfb14ef64db5302d110cbf6429abe95f521b619eb6eca5b49ce08487f
SimHash 2c7b491044d3

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /*.inc$
Disallow /*.txt$
Disallow /wp-admin/
Disallow /wp-content/plugins.backup/
Disallow /wp-content/themes/
Disallow /wp-content/upgrade/
Disallow /wp-content/w3tc/
Disallow /go/
Disallow /ads/
Disallow /aff/
Disallow /app/
Disallow /docs/
Disallow /doubleclick/
Disallow /live-feeds/
Disallow /misc/

googlebot

Rule Path
Disallow /stats*

googlebot-image

Rule Path
Disallow
Allow /*
Back to top

Other Records

Field Value
sitemap http://bigbrothernetwork.com/sitemap.xml.gz
Back to top

Comments

  • disallow files in the following folders
  • disallow all files ending in .php
  • Disallow: /*.js$
  • Disallow: /*.css$
  • disallow all files in /wp- directories
  • Disallow: /wp-includes/
  • Disallow: /wp-content/plugins/
  • Disallow: /wp-content/wptouch-data/
  • disallow all files with ? in url
  • Disallow: /*?
  • disallow any files that are stats related
  • Sitemap location for auto-discovery
Back to top