Log In Sign Up

brewersfriend.com
robots.txt

Robots Exclusion Standard data for brewersfriend.com

Resource Scan

Scan Details

Site Domain brewersfriend.com
Base Domain brewersfriend.com
Scan Status Ok
Last Scan2024-06-26T23:53:31+00:00
Next Scan 2024-07-03T23:53:31+00:00

Last Scan

Scanned2024-06-26T23:53:31+00:00
URL https://brewersfriend.com/robots.txt
Redirect https://www.brewersfriend.com/robots.txt
Redirect Domain www.brewersfriend.com
Redirect Base brewersfriend.com
Domain IPs 104.26.14.147, 104.26.15.147, 172.67.69.136, 2606:4700:20::681a:e93, 2606:4700:20::681a:f93, 2606:4700:20::ac43:4588
Redirect IPs 104.26.14.147, 104.26.15.147, 172.67.69.136, 2606:4700:20::681a:e93, 2606:4700:20::681a:f93, 2606:4700:20::ac43:4588
Response IP 104.26.15.147
Found Yes
Hash 261585bfbc4e2a670c8af598549084c64ea43ec8a47c2eea4c85e6b487f186d0
SimHash a1205b5fcdb4

Groups

*

Rule Path
Disallow /1004971/Cultured
Disallow /forum/members/*
Disallow /homebrew/brewer/notfound
Disallow /homebrew/brewsession/savemashentry
Disallow /homebrew/quick/waterrequirements/
Disallow /homebrew/recipe/beerxml*/
Disallow /homebrew/recipe/buy/
Disallow /homebrew/recipe/delete
Disallow /homebrew/recipe/export/
Disallow /homebrew/recipe/grist
Disallow /homebrew/recipe/json/
Disallow /homebrew/recipe/versions/
Disallow /homebrew/search_mybrewing
Disallow /homebrew/search_myrecipes
Disallow /homebrew/watercalc/save
Disallow /homebrew/watercalc/delete
Disallow /random
Disallow /xmlrpc.php
Disallow /z/
Disallow CVS
Disallow .svn
Disallow .idea
Disallow .sql
Disallow .tgz
Disallow .git
Disallow /wp
Allow /wp-content/themes/bfstyle
Allow /wp-content/uploads
Allow /wp-includes/css
Disallow /forum-phpbb
Disallow /*_customerAdmin
Disallow /*htmlpurifier
Disallow /*?SID=
Disallow /*PHPSESSID
Disallow /?p=
Disallow /*/mash-chemistry-and-brewing-water-calculator*?id=
Disallow /*?frombp=
Disallow /*%26frombp%3D
Disallow /forum/misc/quick-navigation-menu?selected=
Disallow /yeast-pitch-rate-and-starter-calculator*?og=
Disallow /yeast-pitch-rate-and-starter-calculator*%26og%3D
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Back to top

Other Records

Field Value
sitemap https://www.brewersfriend.com/homebrew/sitemap.xml
Back to top

Comments

  • Crawl-Delay: 20
  • Don't crawl development files and folders
  • Do not index the general technical directories
  • Do not index php scripts - everything has an SEO friendly path now
  • Disallow: /*.php
  • Do not index the link from the session ID
  • Skip bad/useless params
  • Do not index searches
  • Do not index mash chemistry and water calc records
  • Do not index parameters
  • Disallow: /*?sort=
  • Disallow: /*&sort=
  • Disallow: /*?order=
  • Disallow: /*&order=
  • Disallow: /*?direction=
  • Disallow: /*&direction=
  • Disallow: /*?*a_aid=
  • Disallow: /*?*mc_eid=
  • Disallow: /*?*mc_cid=
  • Disallow: /*?*utm_creative=
  • Disallow: /*?*utm_source=
  • Disallow: /*?*utm_medium=
  • Disallow: /*?*utm_campaign=
  • Disallow: /*?*%2Fa_aid=
  • Disallow: /*?*replytocom=
  • Disallow: /*?*a_bid=
  • Disallow: /*?*___SID=
  • Server Settings
  • Do not index the general technical directories and files on a server
  • Sitemap file
Back to top