improove.it
robots.txt

Robots Exclusion Standard data for improove.it

Resource Scan

Scan Details

Site Domain improove.it
Base Domain improove.it
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-27T19:43:49+00:00
Next Scan 2025-11-26T19:43:49+00:00

Last Successful Scan

Scanned2025-09-28T11:21:29+00:00
URL https://improove.it/robots.txt
Domain IPs 46.252.146.246
Response IP 46.252.146.246
Found Yes
Hash f924d57bae7e11fed3c185561bcc11392c7bb7a011ecc10e443fc68003887a44
SimHash 012f435209b3

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /shop__trashed/
Allow /wp-content/uploads/
Allow /wp-content/plugin/
Allow /wp-admin/admin-ajax.php

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

deusu/5.0.2

Rule Path
Disallow /

awariorssbot/1.0

Rule Path
Disallow /

kraken

Rule Path
Disallow /

yak/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; yak/1.0; http://linkfluence.com/; bot@linkfluence.com)

Rule Path
Disallow /

awariosmartbot/1.0 (+https://awario.com/bots.html; bots@awario.com)

Rule Path
Disallow /

hubspot links crawler 1.0

Rule Path
Disallow /

hubspot webcrawler

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

hubspot crawler 1.0 http://www.hubspot.com/

Rule Path
Disallow /

hubspot links crawler 1.0 http://www.hubspot.com/

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

builtwith/1.0

Rule Path
Disallow /

builtwith/1.2

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

tpradstxtcrawler/1.1

Rule Path
Disallow /

proximic

Rule Path
Disallow /
Disallow /

Other Records

Field Value
sitemap https://improove.it/sitemap_index.xml

Comments

  • AwarioRssBot
  • AwarioSmartBot
  • Awario
  • AwarioRssBot
  • linkfluence.net
  • linkfluence.com
  • linkfluence.com
  • AwarioSmartBot
  • HubSpot

Warnings

  • 4 invalid lines.
  • `mozilla/5.0 (compatible; proximic; +http` is not a known field.
  • `mozilla/5.0 (compatible; yeti/1.1; +http` is not a known field.
  • `user-aget` is not a known field.