Log In Sign Up

hubinternational.com
robots.txt

Robots Exclusion Standard data for hubinternational.com

Resource Scan

Scan Details

Site Domain hubinternational.com
Base Domain hubinternational.com
Scan Status Ok
Last Scan2024-06-30T03:02:11+00:00
Next Scan 2024-07-30T03:02:11+00:00

Last Scan

Scanned2024-06-30T03:02:11+00:00
URL https://hubinternational.com/robots.txt
Redirect https://www.hubinternational.com/robots.txt
Redirect Domain www.hubinternational.com
Redirect Base hubinternational.com
Domain IPs 40.116.84.42
Redirect IPs 104.18.39.125, 172.64.148.131, 2606:4700:4400::6812:277d, 2606:4700:4400::ac40:9483
Response IP 172.64.148.131
Found Yes
Hash 740487480478099892216d35e6157e106b2a3fade4ed26841328f9faceca95a7
SimHash 70decf702ab0

Groups

*

No rules defined. All paths allowed.

aihitbot
barkrowler
bdcbot
blexbot
blp_bbot
http://brokenlinkcheck.com
buck
ccbot
cliqzbot
cyencebot
domaincrawler
dow jones searchbot
exabot
extlinksbot
femtosearchbot
fever
garlikcrawler
gigabot
gobuster
grapeshotcrawler
heritrix
istellabot
jersey
jobkicks
libwww-perl
linkdexbot
linkpadbot
ltx71 - (http://ltx71.com/)
lua-resty-http
lumtelbot
magpie-crawler
magus bot
mail.ru_bot
http://megaindex.ru
nl-crawler
onpagebot
riddler
scoutjet
scrapy
seekport
seznambot
smtbot
uptimerobot
velenpublicwebcrawler
wget
yacybot
yeti
yisouspider
yunsecuritybot
zoominfobot

Rule Path
Disallow /

ahrefsbot
ahrefssiteaudit
caliperbot
dotbot
hubspot
mj12bot
rogerbot
semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.hubinternational.com/sitemap.xml
Back to top

Comments

  • -- Unwanted Directories and URL Paths --
  • -- Allowed Directories and URL Paths --
  • -- Spam Bots and Other Unwanted Bots --
  • -- SEO Tools and Service - Set Crawl Delay for Optimal Performance --
  • -- XML Sitemap Locations --
Back to top

Warnings

  • 1 invalid line.
  • `request-rate` is not a known field.
Back to top