Log In Sign Up

webhero.be
robots.txt

Robots Exclusion Standard data for webhero.be

Resource Scan

Scan Details

Site Domain webhero.be
Base Domain webhero.be
Scan Status Ok
Last Scan2025-04-23T17:41:55+00:00
Next Scan 2025-05-23T17:41:55+00:00

Last Scan

Scanned2025-04-23T17:41:55+00:00
URL https://webhero.be/robots.txt
Redirect https://www.webhero.be/robots.txt
Redirect Domain www.webhero.be
Redirect Base webhero.be
Domain IPs 104.21.76.194, 172.67.199.232, 2606:4700:3031::ac43:c7e8, 2606:4700:3037::6815:4cc2
Redirect IPs 104.21.76.194, 172.67.199.232, 2606:4700:3031::ac43:c7e8, 2606:4700:3037::6815:4cc2
Response IP 172.67.199.232
Found Yes
Hash 3e24fc3d1b745228bd37e054973f56e14d2b8fa68dac3a9dc24570b2f046e16f
SimHash 6257fd50c8b1

Groups

*

Rule Path
Disallow /admin/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

blexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

serpstatbot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

megaindex.ru

Rule Path
Disallow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

oai-searchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://www.webhero.be/sitemap.xml
Back to top

Comments

  • Slow crawling rate for some agents
Back to top