Log In Sign Up

blog.screenweek.it
robots.txt

Robots Exclusion Standard data for blog.screenweek.it

Resource Scan

Scan Details

Site Domain blog.screenweek.it
Base Domain screenweek.it
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-17T07:55:45+00:00
Next Scan 2024-06-16T07:55:45+00:00

Last Successful Scan

Scanned2024-03-26T07:53:47+00:00
URL https://blog.screenweek.it/robots.txt
Domain IPs 95.216.21.83
Response IP 95.216.21.83
Found Yes
Hash 34b637d899a198d6668c480ca3acfbbbc65a4391cb1d0293316d107fc02d6343
SimHash 4a840f45fe26

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 10

nutch

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

reap-crawler

Rule Path
Disallow /

tarantulas

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbotper

Rule Path
Disallow /

exabot

Rule Path
Disallow /

trendiction

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

yak

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

bdbrandprotect.comw

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

proximic

Rule Path
Disallow /

buck

Rule Path
Disallow /

sogou

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

jersey

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

yeti

Rule Path
Disallow /

contxbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

experibot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

v-bot

Rule Path
Disallow /

coccobot-web

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

mappy

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

gogolbot

Rule Path
Disallow /

quantcastbot

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

linkisbot

Rule Path
Disallow /

slack-imgproxy

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://blog.screenweek.it/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
Back to top