mets.mlb.com
robots.txt
Robots Exclusion Standard data for mets.mlb.com
Resource Scan
Scan Details
| Site Domain | mets.mlb.com |
| Base Domain | mlb.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-09-02T03:44:59+00:00 |
| Next Scan | 2024-12-01T03:44:59+00:00 |
Last Successful Scan
| Scanned | 2024-02-06T01:55:48+00:00 |
| URL | http://mets.mlb.com/robots.txt |
| Redirect | https://www.mlb.com/robots.txt |
| Redirect Domain | www.mlb.com |
| Redirect Base | mlb.com |
| Domain IPs | 34.102.169.19 |
| Redirect IPs | 104.18.38.42, 172.64.149.214, 2606:4700:4400::6812:262a, 2606:4700:4400::ac40:95d6 |
| Response IP | 172.64.149.214 |
| Found | Yes |
| Hash | 5f9939f1947e4b091a779aaa8fbf234da5bca69ad2757fed4f28fc95b436f021 |
| SimHash | 91959b6e841b |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /api/ |
| Disallow | /app/ |
| Disallow | /embed/ |
| Disallow | /en/ |
| Disallow | /mlb/ |
| Disallow | /share/ |
| Disallow | /tokens/ |
| Disallow | /tv/ |
| Disallow | /web/ |
| Disallow | /legacy |
| Disallow | /beta |
| Disallow | /es/legacy |
| Disallow | /es/beta |
| Disallow | /search |
| Disallow | /angels/search |
| Disallow | /astros/search |
| Disallow | /athletics/search |
| Disallow | /bluejays/search |
| Disallow | /braves/search |
| Disallow | /brewers/search |
| Disallow | /cardinals/search |
| Disallow | /cubs/search |
| Disallow | /dbacks/search |
| Disallow | /dodgers/search |
| Disallow | /giants/search |
| Disallow | /guardians/search |
| Disallow | /mariners/search |
| Disallow | /marlins/search |
| Disallow | /mets/search |
| Disallow | /nationals/search |
| Disallow | /orioles/search |
| Disallow | /padres/search |
| Disallow | /phillies/search |
| Disallow | /pirates/search |
| Disallow | /rangers/search |
| Disallow | /rays/search |
| Disallow | /reds/search |
| Disallow | /redsox/search |
| Disallow | /rockies/search |
| Disallow | /royals/search |
| Disallow | /tigers/search |
| Disallow | /twins/search |
| Disallow | /whitesox/search |
| Disallow | /yankees/search |
| Disallow | /es/search |
| Disallow | /es/angels/search |
| Disallow | /es/astros/search |
| Disallow | /es/athletics/search |
| Disallow | /es/bluejays/search |
| Disallow | /es/braves/search |
| Disallow | /es/brewers/search |
| Disallow | /es/cardinals/search |
| Disallow | /es/cubs/search |
| Disallow | /es/dbacks/search |
| Disallow | /es/dodgers/search |
| Disallow | /es/giants/search |
| Disallow | /es/guardians/search |
| Disallow | /es/mariners/search |
| Disallow | /es/marlins/search |
| Disallow | /es/mets/search |
| Disallow | /es/nationals/search |
| Disallow | /es/orioles/search |
| Disallow | /es/padres/search |
| Disallow | /es/phillies/search |
| Disallow | /es/pirates/search |
| Disallow | /es/rangers/search |
| Disallow | /es/rays/search |
| Disallow | /es/reds/search |
| Disallow | /es/redsox/search |
| Disallow | /es/rockies/search |
| Disallow | /es/royals/search |
| Disallow | /es/tigers/search |
| Disallow | /es/twins/search |
| Disallow | /es/whitesox/search |
| Disallow | /es/yankees/search |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.mlb.com/sitemaps/48-hr-news.xml.gz |
| sitemap | https://www.mlb.com/sitemaps/weekly-news/index.xml.gz |
| sitemap | https://www.mlb.com/sitemaps/weekly-video/index.xml.gz |
| sitemap | https://www.mlb.com/sitemaps/pages/en/index.xml.gz |
| sitemap | https://www.mlb.com/sitemaps/pages/es/index.xml.gz |