Log In Sign Up

derman.com
robots.txt

Robots Exclusion Standard data for derman.com

Resource Scan

Scan Details

Site Domain derman.com
Base Domain derman.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-06-25T10:10:34+00:00
Next Scan 2024-07-02T10:10:34+00:00

Last Successful Scan

Scanned2024-06-17T10:09:59+00:00
URL https://derman.com/robots.txt
Redirect https://www.derman.com/robots.txt?DEIredir
Redirect Domain www.derman.com
Redirect Base derman.com
Domain IPs 24.207.40.122
Redirect IPs 24.207.40.122
Response IP 24.207.40.122
Found Yes
Hash 8e5fae875a204664d3fe4b4f06fba798462e2f3443623e9b09c966083574a979
SimHash bab64ddac658

Groups

a6-indexer

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

adstxtcrawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

betabot

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

bingpreview

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

clockwork data vault

Rule Path
Disallow /

dnyzbot

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

dusterio

Rule Path
Disallow /

exabot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

companybook

Rule Path
Disallow /

crawl

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

df bot

Rule Path
Disallow /

dispatch

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dnyzbot

Rule Path
Disallow /

dusterio

Rule Path
Disallow /

exabot

Rule Path
Disallow /

filterdb.iss.net

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

genieo

Rule Path
Disallow /

gimme60bot

Rule Path
Disallow /

go.mail.ru/help/robots

Rule Path
Disallow /

gocrawl

Rule Path
Disallow /

google-youtube-links

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

gowikibot

Rule Path
Disallow /

guardcrwlr

Rule Path
Disallow /

hatena

Rule Path
Disallow /

honeso spider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

icap-iod

Rule Path
Disallow /

iceweasel

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

kerrigan

Rule Path
Disallow /

knowledge ai

Rule Path
Disallow /

kocmohabt

Rule Path
Disallow /

kscan

Rule Path
Disallow /

lightspeedsystemscrawler

Rule Path
Disallow /

link checker

Rule Path
Disallow /

link sleuth

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lua-resty-http

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

mapping

Rule Path
Disallow /

mappy

Rule Path
Disallow /

masscan

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

melvil rawi

Rule Path
Disallow /

mixbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

nbot

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netscan

Rule Path
Disallow /

nettrack

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

openlinkprofiler

Rule Path
Disallow /

outclicksbot

Rule Path
Disallow /

panscient

Rule Path
Disallow /

pbot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

pooplebot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

rankvalbot

Rule Path
Disallow /

rarebits

Rule Path
Disallow /

riddler

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

safeassign

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

semrush

Rule Path
Disallow /

seznam

Rule Path
Disallow /

skypeuripreview

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

scan.trustnet.venafi

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spider

Rule Path
Disallow /

ssl checker

Rule Path
Disallow /

startmebot

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

v-bot

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

verboten

Rule Path
Disallow /

webmeup

Rule Path
Disallow /

whatcmsbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yoozbot

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

*

Rule Path Comment
Disallow *.js -
Disallow /node/23 Drupal's Error page
Disallow /node/13 Support Center's home page
Disallow /node/13?x=&mod_id=1 Support Center's home page
Disallow /node/13?mod_id=1 Support Center's home page
Disallow /cgi-bin/ -
Disallow /cache/ -
Disallow /database/ -
Disallow /files/ -
Disallow /includes/ -
Disallow /LC/ -
Disallow /Licensing/ -
Disallow /misc/ -
Disallow /modules/ -
Disallow /profiles/ -
Disallow /Resources/ -
Disallow /scripts/ -
Disallow /sites/ -
Disallow /themes/ -
Disallow /updates/ -
Disallow /cron.php -
Disallow /install.php -
Disallow /update.php -
Disallow /xmlrpc.php -
Disallow /admin/ -
Disallow /cart/ -
Disallow /comment/reply/ -
Disallow /contact/ -
Disallow /filter/tips/ -
Disallow /logout/ -
Disallow /node/add/ -
Disallow /print/ -
Disallow /search/ -
Disallow /search/user/ -
Disallow /search/user -
Disallow /support/ -
Disallow /system/ -
Disallow /tracker/ -
Disallow /tracker -
Disallow /user/ -
Disallow /user/login/ -
Disallow /user/login -
Disallow /user/password/ -
Disallow /user/password -
Disallow /user/register/ -
Disallow /user/register -
Disallow /?q=admin%2F -
Disallow /?q=cart%2F -
Disallow /?q=comment%2Freply%2F -
Disallow /?q=contact%2F -
Disallow /?q=filter%2Ftips%2F -
Disallow /?q=logout%2F -
Disallow /?q=node%2Fadd%2F -
Disallow /?q=print%2F -
Disallow /?q=search%2F -
Disallow /?q=search%2Fuser%2F -
Disallow /?q=search%2Fuser -
Disallow /?q=support%2F -
Disallow /?q=system%2F -
Disallow /?q=user%2F -

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Specific pages
  • Disallow: /node$
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Disallow: /?q=user/login/
  • Disallow: /?q=user/login
  • Disallow: /?q=user/password/
  • Disallow: /?q=user/password
  • Disallow: /?q=user/register/
  • Disallow: /?q=user/register
Back to top

Warnings

  • 4 invalid lines.
  • `request-rate` is not a known field.
Back to top