better-robots.com
robots.txt

Robots Exclusion Standard data for better-robots.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	better-robots.com
Base Domain	better-robots.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-10-13T19:19:43+00:00
Next Scan	2024-12-12T19:19:43+00:00

Last Successful Scan

Scanned	2024-08-15T18:24:57+00:00
URL	https://better-robots.com/robots.txt
Domain IPs	104.21.38.124, 172.67.222.164, 2606:4700:3031::6815:267c, 2606:4700:3035::ac43:dea4
Response IP	172.67.222.164
Found	Yes
Hash	c5e1bf44836a4f6293001c670a10186712ea024875b335f8f5686672b9d3b11e
SimHash	6235fd6945f2

Groups

*

Rule	Path
Allow	/wp-admin/admin-ajax.php
Disallow	/wp-admin/
Disallow	/wp-includes/
Allow	//.css
Allow	//.js
Disallow	/disclaimer/
Disallow	*?attachment_id=
Disallow	/privacy-policy
Disallow	/search

Rule

Path

Allow

/wp-admin/admin-ajax.php

Disallow

/wp-admin/

Disallow

/wp-includes/

Allow

/*/*.css

Allow

/*/*.js

Disallow

*/disclaimer/*

Disallow

*?attachment_id=

Disallow

/privacy-policy

Disallow

/search

googlebot

Rule	Path
Allow	/

Rule

Path

Allow

googlebot-image

Rule	Path
Allow	/wp-content/uploads/

Rule

Path

Allow

/wp-content/uploads/

mediapartners-google

Rule	Path
Allow	/

Rule

Path

Allow

adsbot-google

Rule	Path
Allow	/

Rule

Path

Allow

adsbot-google-mobile

Rule	Path
Allow	/

Rule

Path

Allow

bingbot

Rule	Path
Allow	/

Rule

Path

Allow

msnbot

Rule	Path
Allow	/

Rule

Path

Allow

msnbot-media

Rule	Path
Allow	/wp-content/uploads/

Rule

Path

Allow

/wp-content/uploads/

applebot

Rule	Path
Allow	/

Rule

Path

Allow

yandex

Rule	Path
Disallow	/

Rule

Path

Disallow

yandeximages

Rule	Path
Allow	/wp-content/uploads/

Rule

Path

Allow

/wp-content/uploads/

slurp

Rule	Path
Allow	/

Rule

Path

Allow

duckduckbot

Rule	Path
Allow	/

Rule

Path

Allow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-video

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou spider

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou web spider

Rule	Path
Disallow	/

Rule

Path

Disallow

sosospider

Rule	Path
Disallow	/

Rule

Path

Disallow

sosospider+

Rule	Path
Disallow	/

Rule

Path

Disallow

sosospider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

yodao

Rule	Path
Disallow	/

Rule

Path

Disallow

youdao

Rule	Path
Disallow	/

Rule

Path

Disallow

youdaobot

Rule	Path
Disallow	/

Rule

Path

Disallow

youdaobot/1.0

Rule	Path
Disallow	/
Disallow	/feed/
Disallow	/feed/$
Disallow	/comments/feed
Disallow	/trackback/
Disallow	/?author=
Disallow	/author/
Disallow	/author*
Disallow	/author/
Disallow	*/comments$
Disallow	*/feed
Disallow	*/feed$
Disallow	*/trackback
Disallow	*/trackback$
Disallow	/?feed=
Disallow	/wp-comments
Disallow	/wp-feed
Disallow	/wp-trackback
Disallow	*/replytocom%3D

Rule

Path

Disallow

/feed/

Disallow

/feed/$

Disallow

/comments/feed

Disallow

/trackback/

Disallow

*/?author=*

Disallow

*/author/*

Disallow

/author*

Disallow

/author/

Disallow

*/comments$

Disallow

*/feed

Disallow

*/feed$

Disallow

*/trackback

Disallow

*/trackback$

Disallow

/?feed=

Disallow

/wp-comments

Disallow

/wp-feed

Disallow

/wp-trackback

Disallow

*/replytocom%3D

giftghostbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seznam

Rule	Path
Disallow	/

Rule

Path

Disallow

paperlibot

Rule	Path
Disallow	/

Rule

Path

Disallow

genieo

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovider/6.101

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovidersiteexplorer

Rule	Path
Disallow	/

Rule

Path

Disallow

dazoobot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

domainstatsbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

dubaiindex

Rule	Path
Disallow	/

Rule

Path

Disallow

ecommercebot

Rule	Path
Disallow	/

Rule

Path

Disallow

expertsearchspider

Rule	Path
Disallow	/

Rule

Path

Disallow

feedbin

Rule	Path
Disallow	/

Rule

Path

Disallow

fetch/2.0a

Rule	Path
Disallow	/

Rule

Path

Disallow

ffbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

focusbot/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

huaweisymantecspider

Rule	Path
Disallow	/

Rule

Path

Disallow

huaweisymantecspider/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

jobdiggerspider

Rule	Path
Disallow	/

Rule

Path

Disallow

lemurwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperheylinkexplorer

Rule	Path
Disallow	/

Rule

Path

Disallow

lssrocketcrawler/1.0

Rule

Path

Disallow

lyt.srv1.5

Rule

Path

Disallow

miadev/0.0.1

Rule

Path

Disallow

najdi.si/3.1

Rule

Path

Disallow

bountiibot

Rule

Path

Disallow

experibot_v1

Rule

Path

Disallow

bixocrawler

Rule

Path

Disallow

bixocrawler testcrawler

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

crowsnest/0.5

Rule

Path

Disallow

cukbot

Rule

Path

Disallow

dataprovider/6.92

Rule

Path

Disallow

dblbot/1.0

Rule

Path

Disallow

diffbot/0.1

Rule

Path

Disallow

digg deeper/v1

Rule

Path

Disallow

discobot/1.0

Rule

Path

Disallow

discobot/1.1

Rule

Path

Disallow

discobot/2.0

Rule

Path

Disallow

discoverybot/2.0

Rule

Path

Disallow

dlvr.it/1.0

Rule

Path

Disallow

domainstatsbot/1.0

Rule

Path

Disallow

drupact/0.7

Rule

Path

Disallow

ezooms/1.0

Rule

Path

Disallow

fastbot crawler beta 2.0

Rule

Path

Disallow

fastbot crawler beta 4.0

Rule

Path

Disallow

feedly social

Rule

Path

Disallow

feedly/1.0

Rule

Path

Disallow

feedlybot/1.0

Rule

Path

Disallow

feedspot

Rule

Path

Disallow

feedspotbot/1.0

Rule

Path

Disallow

clickagy intelligence bot v2

Rule

Path

Disallow

classbot

Rule

Path

Disallow

cispa vulnerability notification

Rule

Path

Disallow

cirrusexplorer/1.1

Rule

Path

Disallow

checksem/nutch-1.10

Rule

Path

Disallow

catchbot/5.0

Rule

Path

Disallow

catchbot/3.0

Rule

Path

Disallow

catchbot/2.0

Rule

Path

Disallow

catchbot/1.0

Rule

Path

Disallow

camontspider/1.0

Rule

Path

Disallow

buzzbot/1.0

Rule

Path

Disallow

buzzbot

Rule

Path

Disallow

businessseek.biz_spider

Rule

Path

Disallow

bubing

Rule

Path

Disallow

fyberspider/1.3

Rule

Path

Disallow

findlinks/1.1.6-beta5

Rule

Path

Disallow

g2reader-bot/1.0

Rule

Path

Disallow

findlinks/1.1.6-beta6

Rule

Path

Disallow

findlinks/2.0

Rule

Path

Disallow

findlinks/2.0.1

Rule

Path

Disallow

findlinks/2.0.2

Rule

Path

Disallow

findlinks/2.0.4

Rule

Path

Disallow

findlinks/2.0.5

Rule

Path

Disallow

findlinks/2.0.9

Rule

Path

Disallow

findlinks/2.1

Rule

Path

Disallow

findlinks/2.1.5

Rule

Path

Disallow

findlinks/2.1.3

Rule

Path

Disallow

findlinks/2.2

Rule

Path

Disallow

findlinks/2.5

Rule

Path

Disallow

findlinks/2.6

Rule

Path

Disallow

ffbot/1.0

Rule

Path

Disallow

findlinks/1.0

Rule

Path

Disallow

findlinks/1.1.3-beta8

Rule

Path

Disallow

findlinks/1.1.3-beta9

Rule

Path

Disallow

findlinks/1.1.4-beta7

Rule

Path

Disallow

findlinks/1.1.6-beta1

Rule

Path

Disallow

findlinks/1.1.6-beta1 yacy

Rule

Path

Disallow

findlinks/1.1.6-beta2

Rule

Path

Disallow

findlinks/1.1.6-beta3

Rule

Path

Disallow

findlinks/1.1.6-beta4

Rule

Path

Disallow

bixo

Rule

Path

Disallow

bixolabs/1.0

Rule

Path

Disallow

crawlera/1.10.2

Rule

Path

Disallow

dataprovider site explorer

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

alexibot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

xenu's

Rule

Path

Disallow

xenu's link sleuth 1.1c

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

semrushbot-ba

Rule

Path

Disallow

semrushbot-si

Rule

Path

Disallow

semrushbot-swa

Rule

Path

Disallow

semrushbot-ct

Rule

Path

Disallow

semrushbot-bm

Rule

Path

Disallow

dotbot/1.1

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

archive.org_bot

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

spbot

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

moreover

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

aboundexbot

Rule

Path

Disallow

proximic

Rule

Path

Disallow

obot

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

nutch

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

zoominfobot

Rule

Path

Disallow

zmeu

Rule

Path

Disallow

grapeshot

Rule

Path

Disallow

python-requests

Rule

Path

Disallow

go-http-client

Rule

Path

Disallow

apache-httpclient

Rule

Path

Disallow

libwww-perl

Rule

Path

Disallow

curl

Rule

Path

Disallow

wget

Rule

Path

Disallow

gptbot

Rule

Path

Disallow

/cart/

Disallow

/checkout/

Disallow

/my-account/

Disallow

/*?orderby=price

Disallow

/*?orderby=rating

Disallow

/*?orderby=date

Disallow

/*?orderby=price-desc

Disallow

/*?orderby=popularity

Disallow

/*?filter

Disallow

/*?orderby=title

Disallow

/*?orderby=desc

Disallow

/*?filter

Disallow

/*add-to-cart%3D*

Disallow

/*add_to_wishlist%3D*

Disallow

/*?paged=&count=*

Disallow

/*?count=*

*

Rule

Path

Allow

/*.png*

Allow

/*.jpg*

Allow

/*.gif*

Allow

/*.webp*

Disallow

/search/

Disallow

*?s=*

Disallow

*?p=*

Disallow

*%26p%3D*

Disallow

*%26preview%3D*

Disallow

/search

twitterbot

Rule

Path

Allow

linkedinbot/1.0

Rule

Path

Allow

pinterest/0.1

Rule

Path

Allow

pinterest/0.2

Rule

Path

Allow

*

Rule

Path

Allow

/ads.txt

*

Rule

Path

Allow

/app-ads.txt

*

Rule

Path

Disallow

/translate-our-plugins

Disallow

/?p=6247

Disallow

/seo-services

Other Records

Field

Value

crawl-delay

Other Records

Field

Value

sitemap

https://better-robots.com/sitemap_index.xml

Comments

Custom Rule for WPML
Popular chinese search engines
Spam Backlink Blocker
Block Bad Bots. Powered by Better Robots.txt Pro
Backlink Protector. Powered by Better Robots.txt Pro
Block Bad Bots. "AI recommended setting" by ChatGPT
ChatGPT Bot Blocker - Block ChatGPT Bot from scrapping your content
Loading Performance for Woocommerce
Image Crawlability by search engines
Avoid crawler traps causing crawl budget issues
Social Media Crawling
Allow/Disallow Ads.txt
Allow/Disallow App-ads.txt
Manual rules with Better Robots.txt Post Meta Box
--------------------------------------------------------------------------# \\\"The best place to hide a dead body is the second page of Google search\\\"# --------------------------------------------------------------------------
This robots.txt file was created by Better Robots.txt (Index & Rank Booster by Pagup) Plugin. https://www.better-robots.com/

Warnings

16 invalid lines.

better-robots.comrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

googlebot

googlebot-image

mediapartners-google

adsbot-google

adsbot-google-mobile

bingbot

msnbot

msnbot-media

applebot

yandex

yandeximages

slurp

duckduckbot

baiduspider

baiduspider/2.0

baiduspider-video

baiduspider-image

sogou spider

sogou web spider

sosospider

sosospider+

sosospider/2.0

yodao

youdao

youdaobot

youdaobot/1.0

giftghostbot

seznam

paperlibot

genieo

dataprovider/6.101

dataprovidersiteexplorer

dazoobot/1.0

diffbot

domainstatsbot/1.0

dubaiindex

ecommercebot

expertsearchspider

feedbin

fetch/2.0a

ffbot/1.0

focusbot/1.1

huaweisymantecspider

huaweisymantecspider/1.0

jobdiggerspider

lemurwebcrawler

lipperheylinkexplorer

lssrocketcrawler/1.0

lyt.srv1.5

miadev/0.0.1

najdi.si/3.1

bountiibot

experibot_v1

bixocrawler

bixocrawler testcrawler

crawler4j

crowsnest/0.5

cukbot

dataprovider/6.92

dblbot/1.0

diffbot/0.1

digg deeper/v1

discobot/1.0

discobot/1.1

discobot/2.0

discoverybot/2.0

dlvr.it/1.0

domainstatsbot/1.0

drupact/0.7

ezooms/1.0

fastbot crawler beta 2.0

fastbot crawler beta 4.0

feedly social

feedly/1.0

better-robots.com
robots.txt