Log In Sign Up

machinetools.com
robots.txt

Robots Exclusion Standard data for machinetools.com

Resource Scan

Scan Details

Site Domain machinetools.com
Base Domain machinetools.com
Scan Status Ok
Last Scan2024-11-06T21:30:47+00:00
Next Scan 2024-12-06T21:30:47+00:00

Last Scan

Scanned2024-11-06T21:30:47+00:00
URL https://machinetools.com/robots.txt
Redirect https://www.machinetools.com/robots.txt
Redirect Domain www.machinetools.com
Redirect Base machinetools.com
Domain IPs 52.15.167.69
Redirect IPs 52.15.167.69
Response IP 52.15.167.69
Found Yes
Hash e74c87309051c7e2a2528dea8f5f10b232ea49ea8915be2f9f288447725bcc6c
SimHash a24a9aa1e135

Groups

ahrefsbot

Rule Path
Disallow /

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

petalbot

Rule Path
Disallow /

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

semrushbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

*

Rule Path
Disallow /control_panel/
Disallow /*/control_panel/
Disallow /login/
Disallow /*/login/
Disallow /search/
Disallow /*/search/
Disallow /*/update_state_select
Disallow /*/ads/
Disallow /*/feedback
Disallow /*/report_error
Disallow /*/responsive
Disallow /*/translation_suggestion
Disallow /*/uploads/
Disallow /*/utility/
Disallow /*?no_crawl=true
Disallow /*%26no_crawl%3Dtrue
Back to top

Other Records

Field Value
sitemap https://www.machinetools.com/xml_sitemaps/sitemap.xml.gz
Back to top

Comments

  • See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AhrefsBot
  • Common Crawl (https://commoncrawl.org/faq/)
  • DotBot (https://opensiteexplorer.org/dotbot)
  • Grapeshot crawler (making malformed URL requests)
  • Linguee (https://www.linguee.com/bot)
  • MJ12bot
  • MSNBot
  • PetalBot (http://aspiegel.com/petalbot)
  • Seekport (https://bot.seekport.com/)
  • SEMrushBot (crawling ancient links and not respecting limits)
  • Yahoo (Slurp)
  • Yahoo Slurp
  • Yandex
Back to top