Log In Sign Up

edinburghfestival.datathistle.com
robots.txt

Robots Exclusion Standard data for edinburghfestival.datathistle.com

Resource Scan

Scan Details

Site Domain edinburghfestival.datathistle.com
Base Domain datathistle.com
Scan Status Ok
Last Scan2024-09-09T12:47:20+00:00
Next Scan 2024-10-09T12:47:20+00:00

Last Scan

Scanned2024-09-09T12:47:20+00:00
URL https://edinburghfestival.datathistle.com/robots.txt
Domain IPs 78.129.221.39
Response IP 78.129.221.39
Found Yes
Hash 1b3ef82ae130f77777b380ecc2718c766cc9f0502d3118d5ee76e7cbee7290ec
SimHash 535e53558520

Groups

*

Rule Path
Disallow
Disallow /details/
Disallow /events/*.xml$
Disallow /places/*.xml$
Disallow /*/show%3A*/
Disallow /*/sort%3A*/
Disallow /articles/*/what%3A*/
Disallow /articles/what%3A*/page%3A*/
Disallow /events/*/what%3A*/
Disallow /events/what%3A*/page%3A*/
Disallow /listings/*/what%3A*/
Disallow /listings/what%3A*/page%3A*/
Disallow /member/
Disallow /places/what%3A*/page%3A*/
Disallow /sign-in/
Disallow /update/*/*/
Disallow /js/
Disallow /*/*/*/*/*/*/

applebot

Rule Path
Disallow
Disallow /details/
Disallow /events/*.xml$
Disallow /places/*.xml$
Disallow /*/what%3A*/
Disallow /*/show%3A*/
Disallow /*/sort%3A*/
Disallow /*/page%3A*/
Disallow /*/distance%3Aany/
Disallow /member/
Disallow /sign-in/
Disallow /update/
Disallow /js/
Disallow /*/*/*/*/*/

googlebot-news

Rule Path
Disallow /event/
Disallow /listing/
Disallow /place/
Disallow /cinema/

adsbot
abonti
ahrefsbot
amazonbot
anthropic-ai
awariobot
awariorssbot
awariosmartbot
baiduspider
barkrowler
berlin-fu-cow
blexbot
buck
ccbot
chatgpt-user
claudebot
coccocbot-web
connexunbot
criteobot/0.1
crystalsemanticsbot
dataforseobot
domainappender
domains-crawler
dotbot
eventseekerbot
exabot
ezooms
flamingo_searchengine
geedobot
geedoproductsearch
genai
genieo
gptbot
grapeshot
hawaiibot
httrack
imagesiftbot
infotigerbot
kalooga
kraken
lcc
magpie-crawler
mail.ru
megaindex.ru
mj12bot
moatbot
mojeekbot
nestreader
netestate ne crawler
netseer
newsnow
node/simplecrawler
nutch
owler
panscient.com
paperlibot
pcore-http
petalbot
piplbot
proximic
psbot
punkspider
qwantify
ravencrawler
riddler
r6_commentreader
scooperbot
scrapy
screaming frog seo spider
searchmetricsbot
seekportbot
semrushbot-ba
semrushbot-bm
semrushbot-coub
semrushbot-ct
semrushbot-si
semrushbot-swa
seokicks-robot
sindicebot
siteauditbot
sitesucker
slurp
sogou spider
sogou web spider
spbot
spinn3r
splitsignalbot
timpibot
trendiction
trendictionbot
ttd-content
turnitinbot
vocus
wesee
wikido
yandexbot
yioopbot
yisouspider
zoombot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.datathistle.com/sitemap-index.xml.gz
Back to top

Comments

  • sitemaps
  • all
  • Disallow: /places/*/what:*/
  • Disallow: /*/where:*/
  • Apple (and Amazon)
  • User-agent: Bingbot
  • stop Google News indexing non-news pages
  • banned
Back to top

Warnings

  • 1 invalid line.
Back to top