follifollie.it
robots.txt

Robots Exclusion Standard data for follifollie.it

Resource Scan

Scan Details

Site Domain follifollie.it
Base Domain follifollie.it
Scan Status Ok
Last Scan2024-10-22T19:53:34+00:00
Next Scan 2024-11-21T19:53:34+00:00

Last Scan

Scanned2024-10-22T19:53:34+00:00
URL https://follifollie.it/robots.txt
Redirect https://www.follifolliegroup.it/robots.txt
Redirect Domain www.follifolliegroup.it
Redirect Base follifolliegroup.it
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.9, 76.76.21.93
Response IP 76.76.21.98
Found Yes
Hash 3430b79ceccc1257ba8d7d5d78052233b68d820517fd9db86ccfa63cd329d98a
SimHash 6a11485ac237

Groups

*

Rule Path
Allow /
Disallow /*.mp4$
Disallow *?*
Disallow /*?*

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

linguee

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

pigafetta

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

sogou

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

storebot

Rule Path
Disallow /

twengabot-2.0

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

willybot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

npbot-1/2.0

Rule Path
Disallow /

npbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value Comment
crawl-delay 2 specifies a 2 second timeout

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.2

Other Records

Field Value
sitemap https://www.follifolliegroup.it/sitemap.xml