Log In Sign Up

captionstobio.com
robots.txt

Robots Exclusion Standard data for captionstobio.com

Resource Scan

Scan Details

Site Domain captionstobio.com
Base Domain captionstobio.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-21T13:32:30+00:00
Next Scan 2024-12-20T13:32:30+00:00

Last Successful Scan

Scanned2023-11-03T09:10:02+00:00
URL https://captionstobio.com/robots.txt
Domain IPs 172.67.165.199
Response IP 172.67.165.199
Found Yes
Hash 9f188b29788dbd6d34f00e761e962fb7aa38b6b09e52b185ed73aa5dab7e9153
SimHash b234dbc887f2

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /*/*.css
Allow /*/*.js
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow *?attachment_id=

*

Rule Path
Disallow /wp-json/
Disallow /?rest_route=

*

Rule Path
Disallow /search/
Disallow /?s=

*

Rule Path
Disallow *?s=*
Disallow *?p=*
Disallow *%26p%3D*
Disallow *%26preview%3D*

*

Rule Path
Disallow /feed/
Disallow /feed/$
Disallow /comments/feed
Disallow */feed
Disallow */feed$
Disallow /?feed=
Disallow /wp-feed

*

Rule Path
Disallow /trackback/
Disallow */comments$
Disallow */trackback
Disallow */trackback$
Disallow /wp-comments
Disallow /wp-trackback
Disallow */replytocom%3D

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

*

Rule Path
Disallow /cart/

*

Rule Path
Disallow /checkout/

*

Rule Path
Disallow /my-account/

*

Rule Path
Disallow /login/

*

Rule Path
Disallow /*?orderby=price
Disallow /*?orderby=rating
Disallow /*?orderby=date
Disallow /*?orderby=price-desc
Disallow /*?orderby=popularity
Disallow /*?filter
Disallow /*?orderby=title
Disallow /*?orderby=desc
Disallow /*add-to-cart%3D*
Disallow /*add_to_wishlist%3D*
Disallow /*?paged=&count=*
Disallow /*?count=*

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /wp-content/uploads/

applebot

Rule Path
Allow /

facebook

Rule Path
Allow /

facebookplatform/1.0

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

facebookexternalhit/1.0

Rule Path
Allow /

facebookexternalhit/1.1

Rule Path
Allow /

facebookscraper

Rule Path
Allow /

facebot/1.0

Rule Path
Allow /

visionutils/0.2

Rule Path
Allow /

datagnionbot/1.0

Rule Path
Allow /

instagrambot

Rule Path
Allow /

whatsapp bot

Rule Path
Allow /

telegrambot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

linkedinbot/1.0

Rule Path
Allow /

pinterest bot

Rule Path
Allow /

pinterest/0.1

Rule Path
Allow /

pinterest/0.2

Rule Path
Allow /

discordbot

Rule Path
Allow /

*

Rule Path
Allow /*.webp$

*

Rule Path
Allow /*.jpg$

*

Rule Path
Allow /*.gif$

*

Rule Path
Allow /*.png$

dotbot

Rule Path
Disallow /

giftghostbot

Rule Path
Disallow /

seznam

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

dataprovider/6.101

Rule Path
Disallow /

dataprovidersiteexplorer

Rule Path
Disallow /

dazoobot/1.0

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dubaiindex

Rule Path
Disallow /

ecommercebot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

feedbin

Rule Path
Disallow /

fetch/2.0a

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

focusbot/1.1

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

huaweisymantecspider/1.0

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lipperheylinkexplorer

Rule Path
Disallow /

lssrocketcrawler/1.0

Rule Path
Disallow /

lyt.srv1.5

Rule Path
Disallow /

miadev/0.0.1

Rule Path
Disallow /

najdi.si/3.1

Rule Path
Disallow /

bountiibot

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

bixocrawler testcrawler

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crowsnest/0.5

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

dataprovider/6.92

Rule Path
Disallow /

dblbot/1.0

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

digg deeper/v1

Rule Path
Disallow /

discobot/1.0

Rule Path
Disallow /

discobot/1.1

Rule Path
Disallow /

discobot/2.0

Rule Path
Disallow /

discoverybot/2.0

Rule Path
Disallow /

dlvr.it/1.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

drupact/0.7

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

fastbot crawler beta 2.0

Rule Path
Disallow /

fastbot crawler beta 4.0

Rule Path
Disallow /

feedly social

Rule Path
Disallow /

feedly/1.0

Rule Path
Disallow /

feedlybot/1.0

Rule Path
Disallow /

feedspot

Rule Path
Disallow /

feedspotbot/1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

classbot

Rule Path
Disallow /

cispa vulnerability notification

Rule Path
Disallow /

cirrusexplorer/1.1

Rule Path
Disallow /

checksem/nutch-1.10

Rule Path
Disallow /

catchbot/5.0

Rule Path
Disallow /

catchbot/3.0

Rule Path
Disallow /

catchbot/2.0

Rule Path
Disallow /

catchbot/1.0

Rule Path
Disallow /

camontspider/1.0

Rule Path
Disallow /

buzzbot/1.0

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

businessseek.biz_spider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

fyberspider/1.3

Rule Path
Disallow /

findlinks/1.1.6-beta5

Rule Path
Disallow /

g2reader-bot/1.0

Rule Path
Disallow /

findlinks/1.1.6-beta6

Rule Path
Disallow /

findlinks/2.0

Rule Path
Disallow /

findlinks/2.0.1

Rule Path
Disallow /

findlinks/2.0.2

Rule Path
Disallow /

findlinks/2.0.4

Rule Path
Disallow /

findlinks/2.0.5

Rule Path
Disallow /

findlinks/2.0.9

Rule Path
Disallow /

findlinks/2.1

Rule Path
Disallow /

findlinks/2.1.5

Rule Path
Disallow /

findlinks/2.1.3

Rule Path
Disallow /

findlinks/2.2

Rule Path
Disallow /

findlinks/2.5

Rule Path
Disallow /

findlinks/2.6

Rule Path
Disallow /

ffbot/1.0

Rule Path
Disallow /

findlinks/1.0

Rule Path
Disallow /

findlinks/1.1.3-beta8

Rule Path
Disallow /

findlinks/1.1.3-beta9

Rule Path
Disallow /

findlinks/1.1.4-beta7

Rule Path
Disallow /

findlinks/1.1.6-beta1

Rule Path
Disallow /

findlinks/1.1.6-beta1 yacy

Rule Path
Disallow /

findlinks/1.1.6-beta2

Rule Path
Disallow /

findlinks/1.1.6-beta3

Rule Path
Disallow /

findlinks/1.1.6-beta4

Rule Path
Disallow /

bixo

Rule Path
Disallow /

bixolabs/1.0

Rule Path
Disallow /

crawlera/1.10.2

Rule Path
Disallow /

dataprovider site explorer

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://captionstobio.com/post-sitemap.xml
sitemap https://captionstobio.com/page-sitemap.xml
Back to top

Comments

  • Advanced Wordpress
  • Prevent Crawling of WordPress JSON API Endpoints
  • Block Search URLs /search/ and /?s=
  • Block Parameters
  • Block Feed
  • Block Spam Directories
  • Block archive.org bots
  • Block Chatgpt
  • Block Cart Page
  • Block Checkout Page
  • Block My Account Page
  • Block Login Page
  • Block Woocommerce Parameters
  • Rankmath Sitemap Link
  • Allow Google Bot
  • Allow Google Images Bot
  • Allow Google Media Partners Bot
  • Allow Google AdsBot Bot
  • Allow Google Mobile Bot
  • Allow Bing Bot
  • Allow MSN Bot
  • Allow MSNBot Media Bot
  • Allow Apple Bot
  • Allow Facebook Bot
  • Allow Instagram Bot
  • Allow Whatsapp Bot
  • Allow Telegram Bot
  • Allow Twitter Bot
  • Allow Linkedin Bot
  • Allow Pinterest Bot
  • Allow Discord Bot
  • Allow Webp Images
  • Allow Jpg Images
  • Allow Gif Images
  • Allow Png Images
  • Block Scrapper Bots
Back to top

Warnings

  • 8 invalid lines.
Back to top