Log In Sign Up

novel.firan.id
robots.txt

Robots Exclusion Standard data for novel.firan.id

Resource Scan

Scan Details

Site Domain novel.firan.id
Base Domain firan.id
Scan Status Ok
Last Scan2024-11-03T09:40:39+00:00
Next Scan 2024-12-03T09:40:39+00:00

Last Scan

Scanned2024-11-03T09:40:39+00:00
URL https://novel.firan.id/robots.txt
Domain IPs 202.10.43.31
Response IP 202.10.43.31
Found Yes
Hash 8ecffc3be4c0f5279ef57bc1d6b359f2c9c79d40491b05730f0e6d0e9addb1c0
SimHash 02dc1a16f49b

Groups

bingbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

feedburner

Rule Path
Disallow /

proximic

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

empty user agent string

Rule Path
Disallow /

feed

Rule Path
Disallow /

facebookexternalhit7

Rule Path
Disallow /

applebot

Rule Path
Disallow /

bingpreview

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

dalvik

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

mail.ru bot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

ias-au/3.3

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

ias-va

Rule Path
Disallow /

ias-va/3.1

Rule Path
Disallow /

ias-

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

ias-jp/3.1

Rule Path
Disallow /

ias-sg/3.1

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Rule Path
Disallow /

criteobot/0.1 (+https://www.criteo.com/criteo-crawler/)

Rule Path
Disallow /

mozilla/5.0 (compatible; dotbot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)

Rule Path
Disallow /

mozilla/5.0 (compatible;petalbot;+https://webmaster.petalsearch.com/site/petalbot)

Rule Path
Disallow /

mozilla/5.0 (compatible; linux x86_64; mail.ru_bot/2.0; +http://go.mail.ru/help/robots)

Rule Path
Disallow /

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

Rule Path
Disallow /

mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)

Rule Path
Disallow /

cutbot; 1.5; http://cutbot.net/

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/7.0; +http://ahrefs.com/robot/)

Rule Path
Disallow /

mozilla/5.0 (compatible; blexbot/1.0; +http://webmeup-crawler.com/)

Rule Path
Disallow /

ias-va/3.3 (former https://www.admantx.com + https://integralads.com/about-ias/)

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

updown.io daemon 2.9

Rule Path
Disallow /

ias-or/3.3

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

duckduckgo-favicons-bot

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

firefox version 10 and lower - various robots

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

unknown robot identified by bot\*

Rule Path
Disallow /

unknown robot (identified by hit on robots.txt)

Rule Path
Disallow /

empty user agent string

Rule Path
Disallow /

okhttp/4.9.3

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

webprosbot/2.0

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://novel.firan.id/sitemap.xml
Back to top