the-tls.co.uk
robots.txt

Robots Exclusion Standard data for the-tls.co.uk

Resource Scan

Scan Details

Site Domain the-tls.co.uk
Base Domain the-tls.co.uk
Scan Status Ok
Last Scan2024-11-09T06:17:40+00:00
Next Scan 2024-11-16T06:17:40+00:00

Last Scan

Scanned2024-11-09T06:17:40+00:00
URL https://the-tls.co.uk/robots.txt
Redirect https://www.the-tls.co.uk/robots.txt
Redirect Domain www.the-tls.co.uk
Redirect Base the-tls.co.uk
Domain IPs 34.240.28.43, 52.208.17.106, 54.76.240.177
Redirect IPs 13.33.28.20, 13.33.28.36, 13.33.28.47, 13.33.28.71
Response IP 13.33.28.20
Found Yes
Hash 57b2fefc13eb36cb5bd25173eb06efbd96892b52436911d5228054516e08c284
SimHash 1030d8e3ad9f

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /search/
Disallow /?s=

newsnow

Rule Path
Disallow /

omgili

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webzip

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.the-tls.co.uk/sitemap_index.xml

Comments

  • Agent Specific Disallowed Sections