fnlondon.com
robots.txt

Robots Exclusion Standard data for fnlondon.com

Resource Scan

Scan Details

Site Domain fnlondon.com
Base Domain fnlondon.com
Scan Status Ok
Last Scan2024-06-06T09:43:40+00:00
Next Scan 2024-06-13T09:43:40+00:00

Last Scan

Scanned2024-06-06T09:43:40+00:00
URL https://fnlondon.com/robots.txt
Redirect https://www.fnlondon.com/robots.txt
Redirect Domain www.fnlondon.com
Redirect Base fnlondon.com
Domain IPs 18.67.181.120, 18.67.181.27, 18.67.181.52, 18.67.181.82
Redirect IPs 18.155.68.104, 18.155.68.118, 18.155.68.39, 18.155.68.69
Response IP 18.155.68.69
Found Yes
Hash 1ee182fd5dec3d46939538dc8022da9039b4631fb9645bf1adfea7c02ae18c5c
SimHash 785dc856cd90

Groups

*

Rule Path
Disallow /profile/*
Disallow /search/*
Disallow /client*
Disallow /auth/*
Disallow /user/*
Disallow /newsletters/svc/*
Disallow /_next/*
Disallow /asset/*
Disallow /forms/*
Disallow /follow/*
Disallow /api/*
Disallow /email_most-read/*
Disallow /cookies*

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.fnlondon.com/sitemap.xml
sitemap https://www.fnlondon.com/fn_google_news.xml
sitemap https://www.fnlondon.com/sitemap_lists.xml
sitemap https://www.fnlondon.com/sitemaps/web/fn/en/sitemap_fn_en_index.xml