fr.pikbest.com
robots.txt

Robots Exclusion Standard data for fr.pikbest.com

Resource Scan

Scan Details

Site Domain fr.pikbest.com
Base Domain pikbest.com
Scan Status Ok
Last Scan2025-03-03T22:32:32+00:00
Next Scan 2025-04-02T22:32:32+00:00

Last Scan

Scanned2025-03-03T22:32:32+00:00
URL https://fr.pikbest.com/robots.txt
Domain IPs 104.18.26.9, 104.18.27.9
Response IP 104.18.27.9
Found Yes
Hash 1331fe53fef672c1ef18456461692571f8e4fe35de6851ab5d027203d0d02feb
SimHash 640452547c33

Groups

*

Rule Path
Allow /*?page=
Allow /so/*/*-*-0-0-0-0-c0_*-*.html
Allow /*/*?p=
Disallow /*?*
Disallow /vip/
Disallow /all/
Disallow /?m=download&id=*
Disallow /*xxx*
Disallow /aggregate/
Disallow /so/*/*-*-0-0-0-0-*.html
Disallow /so/*/*-*-*-*-*-*-*_*-*.html
Disallow /so/tooth-implant.html
Disallow /so/tooth-paste.html
Disallow /free-*/*_*.html
Disallow /sucai/
Disallow /amp/free-*/*_*.html
Disallow /cdn-cgi/

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.2

ahrefsbot/6.1

Rule Path
Disallow /

ahrefsbot/7.0

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

hyscore

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

tineye-bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

musobot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

bleriot

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

hubspot crawler

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou wap spider

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

coccocbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10