Log In Sign Up

amp.theguardian.com
robots.txt

Robots Exclusion Standard data for amp.theguardian.com

Resource Scan

Scan Details

Site Domain amp.theguardian.com
Base Domain theguardian.com
Scan Status Ok
Last Scan2024-11-12T03:23:22+00:00
Next Scan 2024-11-19T03:23:22+00:00

Last Scan

Scanned2024-11-12T03:23:22+00:00
URL https://amp.theguardian.com/robots.txt
Domain IPs 151.101.1.111, 151.101.129.111, 151.101.193.111, 151.101.65.111, 2a04:4e42:200::367, 2a04:4e42:400::367, 2a04:4e42:600::367, 2a04:4e42::367
Response IP 199.232.45.111
Found Yes
Hash 41aa1fd8d85702c48e287b661ca2a8cc9337ab1d5c355c040e7922261a6900f7
SimHash cf11515bc3f6

Groups

*

Rule Path
Disallow

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

https://hada.news

Rule Path
Disallow /

https://www.imediaethics.org

Rule Path
Disallow /

mojeek

Rule Path
Disallow /

jenkersbot

Rule Path
Disallow /

seekr

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

youbot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

yacy

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

netvibes

Rule Path
Disallow /

sentione

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

imagesift

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

yandexadditional

Rule Path
Disallow /

yandexadditionalbot

Rule Path
Disallow /

buck/2.4.2

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /
Back to top

Comments

  • This is the robots.txt file for amp.theguardian.com
  • Guardian content is made available under our terms and conditions of use.
  • Any other uses are not permitted, incl. but not limited to: for large language
  • models (LLMs), machine learning and/or artificial intelligence-related
  • purposes; with any of the aforementioned technologies; and/or for any
  • commercial purposes. Contact licensing@theguardian.com for assistance
Back to top

Warnings

  • 2 invalid lines.
Back to top