Log In Sign Up

dlmf.nist.gov
robots.txt

Robots Exclusion Standard data for dlmf.nist.gov

Resource Scan

Scan Details

Site Domain dlmf.nist.gov
Base Domain nist.gov
Scan Status Ok
Last Scan2025-03-01T01:12:15+00:00
Next Scan 2025-03-31T01:12:15+00:00

Last Scan

Scanned2025-03-01T01:12:15+00:00
URL https://dlmf.nist.gov/robots.txt
Redirect https://math.nist.gov/robots.txt
Redirect Domain math.nist.gov
Redirect Base nist.gov
Domain IPs 129.6.13.19, 2610:20:6005:13::19
Redirect IPs 129.6.13.19, 2610:20:6005:13::19
Response IP 129.6.13.19
Found Yes
Hash 312824915185d8f253855b5d8829d4cbd9bc610833d0dd7a3ac681b942a3c6cf
SimHash 12c35d43eae2

Groups

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin/
Disallow /~SRessler/x3dom/x3domCaesar/

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

eyemonit

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /
Back to top

Comments

  • robots.txt, presumably for all virtual sites
  • PetalBot is dumb and creates millions of hits per week on dlmf.
  • https://www.semrush.com/bot/
  • block SemrushBot a webgraph of links:
  • block SemrushBot different SEO and technical issues:
  • block SemrushBot Backlink Audit tool:
  • block SemrushBot On Page SEO Checker tool and similar tools:
  • block SemrushBot from checking URLs on your site for SWA tool:
  • block SemrushBot Content Analyzer and Post Tracking tools:
  • block SemrushBot Brand Monitoring:
  • block SplitSignalBot SplitSignal tool:
  • block SemrushBot-COUB Content Outline Builder tool:
  • 2023-01-17 https://mj12bot.com/
  • 2023-01-17 https://naver.me/spd
  • 2023-01-17 https://zhanzhang.toutiao.com/
  • 2023-01-17 http://www.eyemon.it/
  • commented per ping by bruce 2023-05-11 since they're sending annoying email to lots of people at nist. :-)
  • 2023-01-17 http://www.bing.com/bingbot.htm
  • User-agent: bingbot
  • Disallow: /
  • User-agent: Bingbot
  • Disallow: /
  • 2023-01-17 http://yandex.com/bots
  • 2023-01-17 https://opensiteexplorer.org/dotbot
  • 2023-01-17 http://ahrefs.com/robot/
  • 2023-01-17 https://help.mail.ru/webmaster/indexing/robots
  • 2023-01-17 http://help.coccoc.com/searchengine
  • 2023-01-30 http://megaindex.com/crawler
Back to top