Log In Sign Up

digitalcompudev.biz
robots.txt

Robots Exclusion Standard data for digitalcompudev.biz

Resource Scan

Scan Details

Site Domain digitalcompudev.biz
Base Domain digitalcompudev.biz
Scan Status Ok
Last Scan2024-11-09T19:29:15+00:00
Next Scan 2024-11-16T19:29:15+00:00

Last Scan

Scanned2024-11-09T19:29:15+00:00
URL https://digitalcompudev.biz/robots.txt
Redirect https://www.digitalcompudev.biz/robots.txt
Redirect Domain www.digitalcompudev.biz
Redirect Base digitalcompudev.biz
Domain IPs 34.149.120.3, 34.160.17.71, 35.190.31.54, 35.244.153.44
Redirect IPs 34.149.120.3, 34.160.81.203, 35.190.31.54, 35.227.194.51
Response IP 34.120.190.48
Found Yes
Hash 6167b8f8d4d94f7880adab55b0e87be9afd3ab4085347dd27cfde10582270131
SimHash 6224ff6045f2

Groups

*

Rule Path Comment
Allow /wp-admin/admin-ajax.php -
Allow /*.js********************* -
Allow /*.css******************** -
Allow /*.jpg******************** -
Allow /*.gif******************** -
Allow /*.png******************** -
Allow /*/*.css -
Allow /*/*.js -
Disallow /wp-admin/ -
Disallow /wp-includes/ -
Disallow /readme.html -
Disallow /license.txt -
Disallow /xmlrpc.php -
Disallow /wp-login.php -
Disallow /wp-register.php -
Disallow */disclaimer/* -
Disallow *?attachment_id= -
Disallow /privacy-policy -
Disallow /mpn/ -
Disallow /upc/ -
Disallow /wp-admin/ -
Disallow /readme.html -
Disallow /license.txt -
Disallow /xmlrpc.php -
Disallow /wp-login.php -
Disallow /wp-register.php -
Disallow */disclaimer/* -
Disallow *?attachment_id= -
Disallow /privacy-policy -
Disallow /terms-and-conditions -
Disallow /feed/ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /trackback/ -
Disallow */?author=* -
Disallow */author/* -
Disallow /author* -
Disallow /author/ -
Disallow /armata/ -
Disallow */comments$ -
Disallow */feed -
Disallow */feed$ -
Disallow */trackback -
Disallow */trackback$ -
Disallow /?feed= -
Disallow /wp-comments -
Disallow /wp-feed -
Disallow /wp-trackback -
Disallow */replytocom%3D -
Disallow /cart/ -
Disallow /checkout/ -
Disallow /my-account/ -
Disallow /*?orderby=price -
Disallow /*?orderby=rating -
Disallow /*?orderby=date -
Disallow /*?orderby=price-desc -
Disallow /*?orderby=popularity -
Disallow /*?filter -
Disallow /*add-to-cart%3D* -
Disallow /search/ -
Disallow *?s=* -
Disallow *?p=* -
Disallow *%26p%3D* -
Disallow *%26preview%3D* -
Disallow /search -
Disallow allow/disallowads.txt
Disallow allow/disallowapp-ads.txt
Disallow /map -
Disallow /edit-profile -
Disallow /my-account -

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /wp-content/uploads/

applebot

Rule Path
Allow /

yandex

Rule Path
Allow /

yandeximages

Rule Path
Allow /wp-content/uploads/

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

qwantify

Rule Path
Allow /

baiduspider

Rule Path
Allow /

baiduspider/2.0

Rule Path
Allow /

baiduspider-video

Rule Path
Allow /

baiduspider-image

Rule Path
Allow /

sogou spider

Rule Path
Allow /

sogou web spider

Rule Path
Allow /

sosospider

Rule Path
Allow /

sosospider+

Rule Path
Allow /

sosospider/2.0

Rule Path
Allow /

yodao

Rule Path
Allow /

youdao

Rule Path
Allow /

youdaobot

Rule Path
Allow /

youdaobot/1.0

Rule Path
Allow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

moreover

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

obot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

curl

Rule Path
Disallow /

wget

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Disallow /feed/
Disallow /feed/$
Disallow /comments/feed
Disallow /trackback/
Disallow */?author=*
Disallow */author/*
Disallow /author*
Disallow /author/
Disallow */comments$
Disallow */feed
Disallow */feed$
Disallow */trackback
Disallow */trackback$
Disallow /?feed=
Disallow /wp-comments
Disallow /wp-feed
Disallow /wp-trackback
Disallow */replytocom%3D

giftghostbot

Rule Path
Disallow /

seznam

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

dataprovider/6.101

Rule Path
Disallow /

dataprovidersiteexplorer

Rule Path
Disallow /

dazoobot/1.0

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dubaiindex

Rule Path
Disallow /

ecommercebot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

feedbin

Rule Path
Disallow /

fetch/2.0a

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

focusbot/1.1

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

huaweisymantecspider/1.0

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lipperheylinkexplorer

Rule Path
Disallow /

lssrocketcrawler/1.0

Rule Path
Disallow /

lyt.srv1.5

Rule Path
Disallow /

miadev/0.0.1

Rule Path
Disallow /

najdi.si/3.1

Rule Path
Disallow /

bountiibot

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

bixocrawler testcrawler

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crowsnest/0.5

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

dataprovider/6.92

Rule Path
Disallow /

dblbot/1.0

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

digg deeper/v1

Rule Path
Disallow /

discobot/1.0

Rule Path
Disallow /

discobot/1.1

Rule Path
Disallow /

discobot/2.0

Rule Path
Disallow /

discoverybot/2.0

Rule Path
Disallow /

dlvr.it/1.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

drupact/0.7

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

fastbot crawler beta 2.0

Rule Path
Disallow /

fastbot crawler beta 4.0

Rule Path
Disallow /

feedly social

Rule Path
Disallow /

feedly/1.0

Rule Path
Disallow /

feedlybot/1.0

Rule Path
Disallow /

feedspot

Rule Path
Disallow /

feedspotbot/1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

classbot

Rule Path
Disallow /

cispa vulnerability notification

Rule Path
Disallow /

cirrusexplorer/1.1

Rule Path
Disallow /

checksem/nutch-1.10

Rule Path
Disallow /

catchbot/5.0

Rule Path
Disallow /

catchbot/3.0

Rule Path
Disallow /

catchbot/2.0

Rule Path
Disallow /

catchbot/1.0

Rule Path
Disallow /

camontspider/1.0

Rule Path
Disallow /

buzzbot/1.0

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

businessseek.biz_spider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

fyberspider/1.3

Rule Path
Disallow /

findlinks/1.1.6-beta5

Rule Path
Disallow /

g2reader-bot/1.0

Rule Path
Disallow /

findlinks/1.1.6-beta6

Rule Path
Disallow /

findlinks/2.0

Rule Path
Disallow /

findlinks/2.0.1

Rule Path
Disallow /

findlinks/2.0.2

Rule Path
Disallow /

findlinks/2.0.4

Rule Path
Disallow /

findlinks/2.0.5

Rule Path
Disallow /

findlinks/2.0.9

Rule Path
Disallow /

findlinks/2.1

Rule Path
Disallow /

findlinks/2.1.5

Rule Path
Disallow /

findlinks/2.1.3

Rule Path
Disallow /

findlinks/2.2

Rule Path
Disallow /

findlinks/2.5

Rule Path
Disallow /

findlinks/2.6

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

findlinks/1.0

Rule Path
Disallow /

findlinks/1.1.3-beta8

Rule Path
Disallow /

findlinks/1.1.3-beta9

Rule Path
Disallow /

findlinks/1.1.4-beta7

Rule Path
Disallow /

findlinks/1.1.6-beta1

Rule Path
Disallow /

findlinks/1.1.6-beta1 yacy

Rule Path
Disallow /

findlinks/1.1.6-beta2

Rule Path
Disallow /

findlinks/1.1.6-beta3

Rule Path
Disallow /

findlinks/1.1.6-beta4

Rule Path
Disallow /

bixo

Rule Path
Disallow /

bixolabs/1.0

Rule Path
Disallow /

crawlera/1.10.2

Rule Path
Disallow /

dataprovider site explorer

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /*?orderby=price
Disallow /*?orderby=rating
Disallow /*?orderby=date
Disallow /*?orderby=price-desc
Disallow /*?orderby=popularity
Disallow /*?filter
Disallow /*?orderby=title
Disallow /*?orderby=desc
Disallow /*?filter
Disallow /*add-to-cart%3D*
Disallow /*add_to_wishlist%3D*
Disallow /*?paged=&count=*
Disallow /*?count=*

*

Rule Path
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*
Allow /*.webp*
Disallow /search/
Disallow *?s=*
Disallow *?p=*
Disallow *%26p%3D*
Disallow *%26preview%3D*
Disallow /search

facebookexternalhit/1.0

Rule Path
Allow /

facebookexternalhit/1.1

Rule Path
Allow /

facebookplatform/1.0

Rule Path
Allow /

facebot/1.0

Rule Path
Allow /

visionutils/0.2

Rule Path
Allow /

datagnionbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot/1.0

Rule Path
Allow /

pinterest/0.1

Rule Path
Allow /

pinterest/0.2

Rule Path
Allow /

*

Rule Path
Allow /ads.txt

*

Rule Path
Allow /app-ads.txt

Other Records

Field Value
crawl-delay 1
Back to top

Comments

  • Popular chinese search engines
  • Block Bad Bots. "AI recommended setting" by ChatGPT
  • ChatGPT Bot Blocker - Block ChatGPT Bot from scrapping your content
  • Spam Backlink Blocker
  • Block Bad Bots. Powered by Better Robots.txt Pro
  • Backlink Protector. Powered by Better Robots.txt Pro
  • Loading Performance for Woocommerce
  • Image Crawlability by search engines
  • Avoid crawler traps causing crawl budget issues
  • Social Media Crawling
  • Allow/Disallow Ads.txt
  • Allow/Disallow App-ads.txt
  • DIGITALCOMPUDEV
  • This robots.txt file was created by Better Robots.txt (Index & Rank Booster by Pagup) Plugin. https://www.better-robots.com/
Back to top

Warnings

  • 8 invalid lines.
Back to top