boeingservices.com
robots.txt

Robots Exclusion Standard data for boeingservices.com

Resource Scan

Scan Details

Site Domain boeingservices.com
Base Domain boeingservices.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-11T23:54:30+00:00
Next Scan 2024-12-10T23:54:30+00:00

Last Successful Scan

Scanned2022-03-01T11:54:05+00:00
URL https://boeingservices.com/robots.txt
Redirect https://services.boeing.com/robots.txt
Redirect Domain services.boeing.com
Redirect Base boeing.com
Response IP 23.64.122.40
Found Yes
Hash d0568f2cfcfe979589d53e2d411fbc7067dce60ccd8dbeefd15212e4c2a6cfae
SimHash e8421f1ecfea

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

Other Records

Field Value
sitemap /sitemap.xml

Comments

  • robots.txt for shop.jeppesen.com
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block AHREFS BOT
  • Block Voltron Bot
  • Block Etao Bot

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.