adolo.uk
robots.txt

Robots Exclusion Standard data for adolo.uk

Resource Scan

Scan Details

Site Domain adolo.uk
Base Domain adolo.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-11-21T04:34:44+00:00
Next Scan 2025-12-05T04:34:44+00:00

Last Successful Scan

Scanned2025-11-04T21:19:48+00:00
URL https://adolo.uk/robots.txt
Domain IPs 2a02:2350:5:10c:8091:e55f:3156:3267, 46.30.213.178
Response IP 46.30.213.178
Found Yes
Hash 6df2896aec5ea209b1c244cff6c4c5dff280fb0216b2de4a94756e76bf7d5b9a
SimHash 4019564064f1

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

metabot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-news

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

baidu

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

majestic-12

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Comments

  • Default rule for unknown bots
  • AI and Large Language Model Crawlers - Blocked
  • Google Bots - Allowed with crawl rate control
  • Bing Bots - Allowed with crawl rate control
  • Other Major Search Engines - Allowed with crawl rate control
  • SEO Tools - Moderate Rate Limiting