allthebelts.com
robots.txt

Robots Exclusion Standard data for allthebelts.com

Resource Scan

Scan Details

Site Domain allthebelts.com
Base Domain allthebelts.com
Scan Status Ok
Last Scan2024-09-29T06:42:00+00:00
Next Scan 2024-10-06T06:42:00+00:00

Last Scan

Scanned2024-09-29T06:42:00+00:00
URL https://allthebelts.com/robots.txt
Redirect https://www.allthebelts.com/robots.txt
Redirect Domain www.allthebelts.com
Redirect Base allthebelts.com
Domain IPs 199.34.228.159
Redirect IPs 199.34.228.159
Response IP 199.34.228.159
Found Yes
Hash 1820131cb252183330b3c9cb72edb5ac87d17411836c1b9d160bd12fd2105a0d
SimHash a044d6f62793

Groups

nerdybot

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /ajax/
Disallow /apps/
Disallow /https%3A//www.allthebelts.com/files/theme/app-ads.txt
Disallow /privacy.html
Disallow /images-hidden.html
Disallow /https%3A//apps.apple.com/us/app/allthebelts-boxing-mma-news/id1183780960

Other Records

Field Value
sitemap https://www.allthebelts.com/sitemap.xml