gawby.com
robots.txt
Robots Exclusion Standard data for gawby.com
Resource Scan
Scan Details
| Site Domain | gawby.com |
| Base Domain | gawby.com |
| Scan Status | Ok |
| Last Scan | 2024-11-16T18:34:03+00:00 |
| Next Scan | 2024-11-23T18:34:03+00:00 |
Last Scan
| Scanned | 2024-11-16T18:34:03+00:00 |
| URL | https://gawby.com/robots.txt |
| Redirect | https://www.gawby.com/robots.txt |
| Redirect Domain | www.gawby.com |
| Redirect Base | gawby.com |
| Domain IPs | 151.101.131.52, 151.101.195.52, 151.101.3.52, 151.101.67.52, 2a04:4e42:200::820, 2a04:4e42:400::820, 2a04:4e42:600::820, 2a04:4e42::820 |
| Redirect IPs | 138.199.46.68, 2400:52e0:1500::868:1 |
| Response IP | 138.199.46.68 |
| Found | Yes |
| Hash | ea1d801aa8001648c769ef22aa0cda51b13ebff1302149f9d74a46a4c6bb3cc0 |
| SimHash | 181a1c4281e2 |
Groups
imagesiftbot
claudebot
bytespider
amazonbot
facebookexternalhit
meta-externalagent
meta-externalfetcher
dataforseobot
gptbot
ccbot
| Rule | Path |
|---|---|
| Disallow | / |
dotbot
ahrefsbot
ahrefssiteaudit
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
splitsignalbot
semrushbot-ocob
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
*
| Rule | Path |
|---|---|
| Disallow | /*?*lang= |
| Disallow | /*?*language= |
| Disallow | /*?*genres= |
| Disallow | /*?*country= |
| Disallow | /*?*certifications= |
| Disallow | /*?*year= |
| Disallow | /*?*streaming= |
| Disallow | /*?*rating= |
| Disallow | /*?*released= |
| Disallow | /*?*name= |
| Disallow | /*?*gender= |
| Disallow | /*?*award_win= |
| Disallow | /*?*award_nominee= |
| Disallow | /*?*role= |
| Disallow | */site/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.gawby.com/sitemap.xml |
Comments