biz.webike.net
robots.txt

Robots Exclusion Standard data for biz.webike.net

Resource Scan

Scan Details

Site Domain biz.webike.net
Base Domain webike.net
Scan Status Ok
Last Scan2024-09-22T11:50:52+00:00
Next Scan 2024-10-22T11:50:52+00:00

Last Scan

Scanned2024-09-22T11:50:52+00:00
URL https://biz.webike.net/robots.txt
Domain IPs 104.22.50.209, 104.22.51.209, 172.67.14.254, 2606:4700:10::6816:32d1, 2606:4700:10::6816:33d1, 2606:4700:10::ac43:efe
Response IP 104.22.51.209
Found Yes
Hash 202b9113255b1464da7768ba6114633feca4ac0305bdf39b1ee87c79e827876a
SimHash 1b4cd060c0b8

Groups

*

Rule Path
Disallow */2324611/*
Disallow */wbs/*
Allow /wbs/genuine-estimate-input.html
Disallow */api/*
Disallow */i/*
Disallow */camp/*
Disallow */catalogue/*

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

twitterbot

Rule Path
Allow /wp-content/uploads/

Comments

  • Sitemap: https://www.webike.net/sitemap/sitemap_index.xml.gz
  • exclude annoyance bots
  • special case (bingbot)
  • Allow Twitter Bot