Log In Sign Up

delicatedave.com
robots.txt

Robots Exclusion Standard data for delicatedave.com

Resource Scan

Scan Details

Site Domain delicatedave.com
Base Domain delicatedave.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-11-15T20:30:52+00:00
Next Scan 2024-11-22T20:30:52+00:00

Last Successful Scan

Scanned2024-10-15T20:30:21+00:00
URL https://delicatedave.com/robots.txt
Domain IPs 82.69.82.16
Response IP 82.69.82.16
Found Yes
Hash b9f571d03627e6279e91428f8d1f68b89dadcebf03b41a918d79f403b6293387
SimHash f234db88c770

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /*/*.css
Allow /*/*.js
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow *?attachment_id=

*

Rule Path
Disallow /wp-json/
Disallow /?rest_route=

*

Rule Path
Disallow /search/
Disallow /?s=

*

Rule Path
Disallow *?s=*
Disallow *?p=*
Disallow *%26p%3D*
Disallow *%26preview%3D*

*

Rule Path
Disallow /feed/
Disallow /feed/$
Disallow /comments/feed
Disallow */feed
Disallow */feed$
Disallow /?feed=
Disallow /wp-feed

*

Rule Path
Disallow /trackback/
Disallow */comments$
Disallow */trackback
Disallow */trackback$
Disallow /wp-comments
Disallow /wp-trackback
Disallow */replytocom%3D

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /wp-content/uploads/

mediapartners-google

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /wp-content/uploads/

applebot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /wp-content/uploads/

facebook

Rule Path
Disallow /

facebookplatform/1.0

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

facebookexternalhit/1.0

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

facebookscraper

Rule Path
Disallow /

facebot/1.0

Rule Path
Disallow /

visionutils/0.2

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

instagrambot

Rule Path
Disallow /

whatsapp bot

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

linkedinbot/1.0

Rule Path
Disallow /

pinterest bot

Rule Path
Disallow /

pinterest/0.1

Rule Path
Disallow /

pinterest/0.2

Rule Path
Disallow /

discordbot

Rule Path
Disallow /

*

Rule Path
Disallow /*.webp$

*

Rule Path
Disallow /*.png$

*

Rule Path
Disallow /*.jpg$

*

Rule Path
Disallow /*.gif$

*

Rule Path
Disallow /*.pdf$

*

Rule Path
Disallow /*.docx$

*

Rule Path
Disallow /*.php$

dotbot

Rule Path
Disallow /

giftghostbot

Rule Path
Disallow /

seznam

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

dataprovider/6.101

Rule Path
Disallow /

dataprovidersiteexplorer

Rule Path
Disallow /

dazoobot/1.0

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dubaiindex

Rule Path
Disallow /

ecommercebot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

feedbin

Rule Path
Disallow /

fetch/2.0a

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

focusbot/1.1

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

huaweisymantecspider/1.0

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lipperheylinkexplorer

Rule Path
Disallow /

lssrocketcrawler/1.0

Rule Path
Disallow /

lyt.srv1.5

Rule Path
Disallow /

miadev/0.0.1

Rule Path
Disallow /

najdi.si/3.1

Rule Path
Disallow /

bountiibot

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

bixocrawler testcrawler

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crowsnest/0.5

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

dataprovider/6.92

Rule Path
Disallow /

dblbot/1.0

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

digg deeper/v1

Rule Path
Disallow /

discobot/1.0

Rule Path
Disallow /

discobot/1.1

Rule Path
Disallow /

discobot/2.0

Rule Path
Disallow /

discoverybot/2.0

Rule Path
Disallow /

dlvr.it/1.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

drupact/0.7

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

fastbot crawler beta 2.0

Rule Path
Disallow /

fastbot crawler beta 4.0

Rule Path
Disallow /

feedly social

Rule Path
Disallow /

feedly/1.0

Rule Path
Disallow /

feedlybot/1.0

Rule Path
Disallow /

feedspot

Rule Path
Disallow /

feedspotbot/1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

classbot

Rule Path
Disallow /

cispa vulnerability notification

Rule Path
Disallow /

cirrusexplorer/1.1

Rule Path
Disallow /

checksem/nutch-1.10

Rule Path
Disallow /

catchbot/5.0

Rule Path
Disallow /

catchbot/3.0

Rule Path
Disallow /

catchbot/2.0

Rule Path
Disallow /

catchbot/1.0

Rule Path
Disallow /

camontspider/1.0

Rule Path
Disallow /

buzzbot/1.0

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

businessseek.biz_spider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

fyberspider/1.3

Rule Path
Disallow /

findlinks/1.1.6-beta5

Rule Path
Disallow /

g2reader-bot/1.0

Rule Path
Disallow /

findlinks/1.1.6-beta6

Rule Path
Disallow /

findlinks/2.0

Rule Path
Disallow /

findlinks/2.0.1

Rule Path
Disallow /

findlinks/2.0.2

Rule Path
Disallow /

findlinks/2.0.4

Rule Path
Disallow /

findlinks/2.0.5

Rule Path
Disallow /

findlinks/2.0.9

Rule Path
Disallow /

findlinks/2.1

Rule Path
Disallow /

findlinks/2.1.5

Rule Path
Disallow /

findlinks/2.1.3

Rule Path
Disallow /

findlinks/2.2

Rule Path
Disallow /

findlinks/2.5

Rule Path
Disallow /

findlinks/2.6

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

findlinks/1.0

Rule Path
Disallow /

findlinks/1.1.3-beta8

Rule Path
Disallow /

findlinks/1.1.3-beta9

Rule Path
Disallow /

findlinks/1.1.4-beta7

Rule Path
Disallow /

findlinks/1.1.6-beta1

Rule Path
Disallow /

findlinks/1.1.6-beta1 yacy

Rule Path
Disallow /

findlinks/1.1.6-beta2

Rule Path
Disallow /

findlinks/1.1.6-beta3

Rule Path
Disallow /

findlinks/1.1.6-beta4

Rule Path
Disallow /

bixo

Rule Path
Disallow /

bixolabs/1.0

Rule Path
Disallow /

crawlera/1.10.2

Rule Path
Disallow /

dataprovider site explorer

Rule Path
Disallow /
Back to top

Comments

  • Advanced Wordpress
  • Prevent Crawling of WordPress JSON API Endpoints
  • Block Search URLs /search/ and /?s=
  • Block Parameters
  • Block Feed
  • Block Spam Directories
  • Block archive.org bots
  • Block Google Images Bot
  • Block Google Media Partners Bot
  • Block Google AdsBot Bot
  • Block Google Mobile Bot
  • Block MSNBot Media Bot
  • Block Apple Bot
  • Block Yandex Images Bot
  • Block Facebook Bot
  • Block Instagram Bot
  • Block Whatsapp Bot
  • Block Telegram Bot
  • Block Twitter Bot
  • Block Linkedin Bot
  • Block Pinterest Bot
  • Block Discord Bot
  • Block Webp Images
  • Block Png Images
  • Block Jpg Images
  • Block Gif Images
  • Block PDF Files
  • Block DOCX Files
  • Block Php Files
  • Block Scrapper Bots
Back to top

Warnings

  • 8 invalid lines.
Back to top