Log In Sign Up

novelup.plus
robots.txt

Robots Exclusion Standard data for novelup.plus

Resource Scan

Scan Details

Site Domain novelup.plus
Base Domain novelup.plus
Scan Status Ok
Last Scan2024-11-09T14:01:52+00:00
Next Scan 2024-11-16T14:01:52+00:00

Last Scan

Scanned2024-11-09T14:01:52+00:00
URL https://novelup.plus/robots.txt
Domain IPs 3.165.102.111, 3.165.102.120, 3.165.102.56, 3.165.102.68
Response IP 3.165.102.111
Found Yes
Hash f597aa7d69701164aee4dc676e30cf84c08b0a2c99d349a834e808efa1dcfb0e
SimHash 934413146503

Groups

*

Rule Path
Disallow /my/
Disallow /edit/
Disallow /api/
Disallow /shop/point/

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

anewsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

clipper

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

http://firstra.in/fr3_0

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

linguee

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ias_crawler
ias-ie
peer39_crawler
imagesiftbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /
Back to top

Comments

  • Yahoo! (minute)
  • Microsoft Live Search (sec)
  • baidu (sec)
  • Naver
  • AhrefsBot
  • Anewsbot
  • Amazon
  • apple
  • webmeup BLEX
  • Buck trial
  • Clipper
  • Common Crawl
  • Common CriteoBot
  • DataForSeoBot
  • dotbot
  • ExaBot
  • Femtosearch
  • firstrain
  • flamingosearch.com
  • grapeshot
  • grapeshot
  • gptbot
  • Magpie Brandwatch
  • Mappy Crawler
  • cyber.harvard.edu
  • MJ12bot
  • Nutch
  • openindex
  • comscore
  • Linguee
  • Scrapy
  • sempi
  • Semrush
  • Semrush
  • sogou.com chinese search engine
  • toutiao trial
  • Trendiction
  • trendkite
  • turntin
  • theTradeDesk
  • Yandex
Back to top