Log In Sign Up

henrywiltshire.ae
robots.txt

Robots Exclusion Standard data for henrywiltshire.ae

Resource Scan

Scan Details

Site Domain henrywiltshire.ae
Base Domain henrywiltshire.ae
Scan Status Ok
Last Scan2024-10-08T06:38:28+00:00
Next Scan 2024-11-07T06:38:28+00:00

Last Scan

Scanned2024-10-08T06:38:28+00:00
URL https://henrywiltshire.ae/robots.txt
Domain IPs 35.214.33.95
Response IP 35.214.33.95
Found Yes
Hash 0485d12193f2a3bb64732e5adfdf38471a9bcebfd420dcf0964460bf8265d0fa
SimHash ba444f70c4b0

Groups

ahrefsbot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

bpimagewalker

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

clockwork data vault

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

domain re-animator

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

getintent

Rule Path
Disallow /

grammarly

Rule Path
Disallow /

guardcrwlr

Rule Path
Disallow /

hubspot links crawler

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

kocmohabt

Rule Path
Disallow /

link checker

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkscrawler

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

link sleuth

Rule Path
Disallow /

linkspammer

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

nettrack

Rule Path
Disallow /

okhttp

Rule Path
Disallow /

openlinkprofiler

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pu_in

Rule Path
Disallow /

pulsepoint

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

rankvalbot

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

searchie

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

socialrankiobot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

vericitecrawler

Rule Path
Disallow /

webmeup-crawler

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

my server guy

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-snapshots/
Disallow wp-app.php
Disallow wp-atom.php
Disallow wp-blog-header.php
Disallow wp-comments-post.php
Disallow wp-config-sample.php
Disallow wp-config.php
Disallow wp-cron.php
Disallow wp-feed.php
Disallow wp-login.php
Disallow wp-links-opml.php
Disallow wp-mail.php
Disallow wp-pass.php
Disallow wp-rdf.php
Disallow wp-register.php
Disallow wp-rss.php
Disallow wp-rss2.php
Disallow wp-settings.php
Disallow wp-trackback.php
Disallow xmlrpc.php
Disallow /uploads/ithemes-security/
Disallow /uploads/shiftnav_backups/
Disallow /upgrade/

ninjabot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.henrywiltshire.ae/sitemap_index.xml
Back to top

Comments

  • Ezooms and dotbot
Back to top

Warnings

  • 4 invalid lines.
Back to top