theensemble.me
robots.txt

Robots Exclusion Standard data for theensemble.me

Resource Scan

Scan Details

Site Domain theensemble.me
Base Domain theensemble.me
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-07T05:52:42+00:00
Next Scan 2024-12-07T05:52:42+00:00

Last Successful Scan

Scanned2024-10-02T05:39:47+00:00
URL https://theensemble.me/robots.txt
Redirect https://borrow.theensemble.me/robots.txt
Redirect Domain borrow.theensemble.me
Redirect Base theensemble.me
Domain IPs 13.35.238.116, 13.35.238.24, 13.35.238.58, 13.35.238.92
Redirect IPs 13.35.238.116, 13.35.238.24, 13.35.238.58, 13.35.238.92
Response IP 13.35.238.58
Found Yes
Hash 1eee615257859cd9f442b897e7d06be9b5e4018b3069f89c467b0099fc514e13
SimHash 2d159c607790

Groups

*

Rule Path
Disallow /*preview_theme_id*

adsbot-google

Rule Path
Disallow /*preview_theme_id*

nutch

Rule Path
Disallow /*preview_theme_id*

ahrefsbot

Rule Path
Disallow /*preview_theme_id*
Disallow /search
Disallow /apple-app-site-association

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /*preview_theme_id*
Disallow /search
Disallow /apple-app-site-association

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml

Comments

  • Google adsbot ignores robots.txt unless specifically named!