Log In Sign Up

steelerscountry.com
robots.txt

Robots Exclusion Standard data for steelerscountry.com

Resource Scan

Scan Details

Site Domain steelerscountry.com
Base Domain steelerscountry.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-05-03T05:02:46+00:00
Next Scan 2024-08-01T05:02:46+00:00

Last Successful Scan

Scanned2023-09-14T00:13:30+00:00
URL https://steelerscountry.com/robots.txt
Redirect https://www.steelerscountry.com/robots.txt
Redirect Domain www.steelerscountry.com
Redirect Base steelerscountry.com
Domain IPs 13.226.2.100, 13.226.2.49, 13.226.2.69, 13.226.2.89, 2600:9000:218c:2600:a:ff58:b8c0:93a1, 2600:9000:218c:4e00:a:ff58:b8c0:93a1, 2600:9000:218c:6600:a:ff58:b8c0:93a1, 2600:9000:218c:7400:a:ff58:b8c0:93a1, 2600:9000:218c:7a00:a:ff58:b8c0:93a1, 2600:9000:218c:a800:a:ff58:b8c0:93a1, 2600:9000:218c:ac00:a:ff58:b8c0:93a1, 2600:9000:218c:f800:a:ff58:b8c0:93a1
Redirect IPs 2600:9000:21b3:0:a:ff58:b8c0:93a1, 2600:9000:21b3:1800:a:ff58:b8c0:93a1, 2600:9000:21b3:3a00:a:ff58:b8c0:93a1, 2600:9000:21b3:4e00:a:ff58:b8c0:93a1, 2600:9000:21b3:6000:a:ff58:b8c0:93a1, 2600:9000:21b3:800:a:ff58:b8c0:93a1, 2600:9000:21b3:9c00:a:ff58:b8c0:93a1, 2600:9000:21b3:d600:a:ff58:b8c0:93a1, 3.160.212.110, 3.160.212.48, 3.160.212.71, 3.160.212.75
Response IP 52.84.174.44
Found Yes
Hash b0c94be5010b40364b0c92ad00f1ae45d222eee9a93dc58d7ad8edbf86b26019
SimHash b8b04d038c98

Groups

*

Rule Path
Disallow /startTopic/
Disallow /discover/unread/
Disallow /markallread/
Disallow /staff/
Disallow /cookie/
Disallow /online/
Disallow /discover/
Disallow /leaderboard/
Disallow /search/
Disallow /tags/
Disallow /*?advancedSearchForm=
Disallow /register/
Disallow /lostpassword/
Disallow /login/
Disallow /*?sortby=
Disallow /*?filter=
Disallow /*?tab=
Disallow /*?do=
Disallow /*ref%3D
Disallow /*?forumId*
Disallow /profile/
Back to top

Other Records

Field Value
sitemap https://www.steelerscountry.com/sitemap.php
Back to top

Comments

  • Rules for Invision Community (https://invisioncommunity.com)
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
  • Sitemap URL
Back to top