webomator.com
robots.txt

Robots Exclusion Standard data for webomator.com

Resource Scan

Scan Details

Site Domain webomator.com
Base Domain webomator.com
Scan Status Ok
Last Scan2024-10-29T15:24:34+00:00
Next Scan 2024-11-28T15:24:34+00:00

Last Scan

Scanned2024-10-29T15:24:34+00:00
URL https://webomator.com/robots.txt
Domain IPs 23.254.252.9
Response IP 23.254.252.9
Found Yes
Hash 21f3dee3fcf5a4f9d73f7e137429a25b2ce114c0791adf51e93bdca38dd54563
SimHash 524ff150c8b2

Groups

*

Rule Path
Disallow /wp-login.php
Disallow /wordpress/wp-login.php

Other Records

Field Value
crawl-delay 30

exabot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

ezooms.bot
ezooms
ezooms.bot@gmail.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

http://help.soso.com/webspider.htm

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

http://help.soso.com/webspider.htm

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

yandex

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

umbot-ln

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 180

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 180