Log In Sign Up

veteranownedbusiness.com
robots.txt

Robots Exclusion Standard data for veteranownedbusiness.com

Resource Scan

Scan Details

Site Domain veteranownedbusiness.com
Base Domain veteranownedbusiness.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-29T04:34:02+00:00
Next Scan 2024-11-28T04:34:02+00:00

Last Successful Scan

Scanned2024-08-01T04:19:17+00:00
URL https://veteranownedbusiness.com/robots.txt
Domain IPs 172.66.40.172, 172.66.43.84, 2606:4700:3108::ac42:28ac, 2606:4700:3108::ac42:2b54
Response IP 172.66.43.84
Found Yes
Hash bca7e43bbdfdde3f2bc73e557083fd54c1ef417e0e7db75586e45226d526d0c8
SimHash 1848f1c18013

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ltx71 - (http://ltx71.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

meta-externalagent

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

siteauditbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot-ba

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot-si

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot-swa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot-ct

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot-bm

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

splitsignalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

baiduspider
baiduspider
baiduspider+
baiduspider-video
baiduspider-image

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

proximic

Rule Path
Disallow /php/

proximic

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sp_auditbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

spbot

Rule Path
Disallow /
Back to top

Comments

  • proximic.com/info/spider.php
  • Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • Block MegaIndex.ru
  • Block Sogou
  • Block SEOkicks
  • SEOProfiler
  • Block BlexBot
  • Block SISTRIX
  • Block WiseGuys Robot
  • Block dotbot
  • Block rogerbot
  • Block Turnitin Robot
  • Block Heritrix
  • Block bhcBot
  • Block SoGou
  • Block Youdao
  • Open Link Profiler
  • Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
  • http://OpenLinkProfiler.org/bot
Back to top