Log In Sign Up

herhoopstats.com
robots.txt

Robots Exclusion Standard data for herhoopstats.com

Resource Scan

Scan Details

Site Domain herhoopstats.com
Base Domain herhoopstats.com
Scan Status Ok
Last Scan2024-11-14T07:37:58+00:00
Next Scan 2024-11-21T07:37:58+00:00

Last Scan

Scanned2024-11-14T07:37:58+00:00
URL https://herhoopstats.com/robots.txt
Domain IPs 34.205.239.236, 52.1.110.113
Response IP 52.1.110.113
Found Yes
Hash 48c74d2e20d477a2a7e5c0ef6693d9db34a0d40365ea5e8eb31466ae1e596cc1
SimHash 6600a210c7c5

Groups

*

Rule Path
Disallow /

ltx71

Rule Path
Disallow /
Disallow /*

googlebot
googlebot-news
googlebot-image
bingbot
yandexbot
baiduspider
facebot
twitterbot
slurp
duckduckbot
ia_archiver

Product Comment
googlebot All Google crawlers
googlebot-news Crawler for Google News
googlebot-image Crawler for Google Images
bingbot Bing’s crawler
yandexbot Yandex’s crawler
baiduspider Baidu’s crawler
facebot Facebook’s crawler
twitterbot Twitter’s crawler
slurp Yahoo’s crawler
duckduckbot DuckDuckGo
ia_archiver Amazon’s Alexa internet rankings
Rule Path Comment
Allow /accounts/login/$ Allow login but without any parameters (must end with /)
Allow /accounts/signup_pay/$ Allow signup_pay but without any parameters (must end with /)
Disallow /accounts/ Otherwise don't allow accounts to be crawled/indexed
Allow /stats/lobos_look/ncaa/team/$ -
Disallow /stats/lobos_look/ncaa/team/ -
Allow /stats/lobos_look/wnba/team/$ -
Disallow /stats/lobos_look/wnba/team/ -
Allow /stats/compare/ncaa/team/$ -
Disallow /stats/compare/ncaa/team/ -
Allow /stats/ncaa/research/player_careers/$ -
Disallow /stats/ncaa/research/player_careers/ -
Allow /stats/ncaa/research/player_single_seasons/$ -
Disallow /stats/ncaa/research/player_single_seasons/ -
Allow /stats/ncaa/research/player_total_seasons/$ -
Disallow /stats/ncaa/research/player_total_seasons/ -
Allow /stats/ncaa/research/player_single_games/$ -
Disallow /stats/ncaa/research/player_single_games/ -
Allow /stats/ncaa/research/player_total_games/$ -
Disallow /stats/ncaa/research/player_total_games/ -
Allow /stats/ncaa/research/team_single_seasons/$ -
Disallow /stats/ncaa/research/team_single_seasons/ -
Allow /stats/ncaa/research/team_total_seasons/$ -
Disallow /stats/ncaa/research/team_total_seasons/ -
Allow /stats/ncaa/research/team_single_games/$ -
Disallow /stats/ncaa/research/team_single_games/ -
Allow /stats/ncaa/research/team_total_games/$ -
Disallow /stats/ncaa/research/team_total_games/ -
Allow /stats/wnba/research/player_careers/$ -
Disallow /stats/wnba/research/player_careers/ -
Allow /stats/wnba/research/player_single_seasons/$ -
Disallow /stats/wnba/research/player_single_seasons/ -
Allow /stats/wnba/research/player_total_seasons/$ -
Disallow /stats/wnba/research/player_total_seasons/ -
Allow /stats/wnba/research/player_single_games/$ -
Disallow /stats/wnba/research/player_single_games/ -
Allow /stats/wnba/research/player_total_games/$ -
Disallow /stats/wnba/research/player_total_games/ -
Allow /stats/wnba/research/team_single_seasons/$ -
Disallow /stats/wnba/research/team_single_seasons/ -
Allow /stats/wnba/research/team_total_seasons/$ -
Disallow /stats/wnba/research/team_total_seasons/ -
Allow /stats/wnba/research/team_single_games/$ -
Disallow /stats/wnba/research/team_single_games/ -
Allow /stats/wnba/research/team_total_games/$ -
Disallow /stats/wnba/research/team_total_games/ -
Disallow /stats/leaderboard/ -
Allow / Allow everything else

Other Records

Field Value
crawl-delay 500
Back to top

Other Records

Field Value
sitemap https://herhoopstats.com/sitemap.xml/
Back to top

Comments

  • Disallow most crawlers
  • Allow Google, and other key crawlers
  • Allow Lobo's Look and Compare Team but without any parameters (must end with /)
  • Allow Research but without any parameters (must end with /)
  • Disallow Leaderboards
Back to top