Log In Sign Up

realniches.com
robots.txt

Robots Exclusion Standard data for realniches.com

Resource Scan

Scan Details

Site Domain realniches.com
Base Domain realniches.com
Scan Status Ok
Last Scan2024-10-08T15:16:32+00:00
Next Scan 2024-11-07T15:16:32+00:00

Last Scan

Scanned2024-10-08T15:16:32+00:00
URL https://realniches.com/robots.txt
Redirect https://www.realniches.com/robots.txt
Redirect Domain www.realniches.com
Redirect Base realniches.com
Domain IPs 67.231.253.33
Redirect IPs 67.231.253.33
Response IP 67.231.253.33
Found Yes
Hash ca949fa984e96281be41a941e4d09339e14ba98921551f0d77c3654a3ff5ddb7
SimHash 417d9492e38a

Groups

extractorpro

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver/1.6

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

zeus link scout

Rule Path
Disallow /

zeus 32297 webster pro v2.9 win32

Rule Path
Disallow /

getright/4.2

Rule Path
Disallow /

webreaper v9.8 - www.webreaper.net

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

download ninja 7.0

Rule Path
Disallow /

download ninja 5.0

Rule Path
Disallow /

download ninja 2.0

Rule Path
Disallow /

webreaper [info@webreaper.net]

Rule Path
Disallow /

webstripper/2.16

Rule Path
Disallow /

webstripper/2.61

Rule Path
Disallow /

wget/1.8.2

Rule Path
Disallow /

webcopier v3.6

Rule Path
Disallow /

webcopier v4.0

Rule Path
Disallow /

webstripper/2.59

Rule Path
Disallow /

webstripper/2.52

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

mozilla/5.0+(compatible;+barkrowler/0.9;++https://babbar.tech/crawler)

Rule Path
Disallow /

mozilla/5.0+(compatible; barkrowler/0.9; +https://babbar.tech/crawler)

Rule Path
Disallow /

rollcrawl

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

junglee bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

Rule Path
Disallow /

mozilla/5.0+(compatible;+megaindex.ru/2.0;++http://megaindex.com/crawler)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

mozilla/5.0+(compatible;+pinterestbot/1.0;++http://www.pinterest.com/bot.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; pinterestbot/1.0; http://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

mozilla/5.0+(compatible;+adsbot/3.1)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0+(compatible;+dotbot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0+(compatible;+blexbot/1.0;++http://webmeup-crawler.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

blexbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

ccbot/2.0 (http://commoncrawl.org/faq/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

ccbot/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0+(compatible;+dataforseobot/1.0;++https://dataforseo.com/dataforseo-bot)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2000

*

Rule Path
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /wp-content/plugins
Disallow /wp-content/cache
Disallow /wp-content/themes
Disallow /trackback
Disallow /cgi-bin
Disallow /search
Disallow /feed
Disallow /rss
Disallow /comments/feed
Disallow /feed/$
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Back to top

Comments

  • Block MegaIndex.ru
Back to top