borrow.theensemble.me
robots.txt

Robots Exclusion Standard data for borrow.theensemble.me

Resource Scan

Scan Details

Site Domain borrow.theensemble.me
Base Domain theensemble.me
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-25T03:45:12+00:00
Next Scan 2024-07-24T03:45:12+00:00

Last Successful Scan

Scanned2024-03-20T03:31:57+00:00
URL https://borrow.theensemble.me/robots.txt
Domain IPs 13.33.33.28, 13.33.33.61, 13.33.33.77, 13.33.33.82
Response IP 13.33.33.28
Found Yes
Hash 1eee615257859cd9f442b897e7d06be9b5e4018b3069f89c467b0099fc514e13
SimHash 2d159c607790

Groups

*

Rule Path
Disallow /*preview_theme_id*

adsbot-google

Rule Path
Disallow /*preview_theme_id*

nutch

Rule Path
Disallow /*preview_theme_id*

ahrefsbot

Rule Path
Disallow /*preview_theme_id*
Disallow /search
Disallow /apple-app-site-association

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /*preview_theme_id*
Disallow /search
Disallow /apple-app-site-association

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml
sitemap https://borrow.theensemble.me/sitemap.xml

Comments

  • Google adsbot ignores robots.txt unless specifically named!