tremendous.com
robots.txt

Robots Exclusion Standard data for tremendous.com

Resource Scan

Scan Details

Site Domain tremendous.com
Base Domain tremendous.com
Scan Status Ok
Last Scan2024-09-24T11:14:10+00:00
Next Scan 2024-10-08T11:14:10+00:00

Last Scan

Scanned2024-09-24T11:14:10+00:00
URL https://tremendous.com/robots.txt
Redirect https://www.tremendous.com/robots.txt
Redirect Domain www.tremendous.com
Redirect Base tremendous.com
Domain IPs 104.18.12.217, 104.18.13.217, 2606:4700::6812:cd9, 2606:4700::6812:dd9
Redirect IPs 104.18.12.217, 104.18.13.217, 2606:4700::6812:cd9, 2606:4700::6812:dd9
Response IP 104.18.12.217
Found Yes
Hash 624b5f5e7b39f42dd5ad852649cf7b2b3d2ae1da987d059fdb4aebbdc9263286
SimHash 661bc660c61b

Groups

*

Rule Path
Disallow /short_urls/
Disallow /rewards/
Disallow /api/
Disallow /v1/
Disallow /internal/
Disallow /lp/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ahrefsbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

megaindex.ru 2.0

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.tremendous.com/sitemap.xml