Log In Sign Up

antcat.org
robots.txt

Robots Exclusion Standard data for antcat.org

Resource Scan

Scan Details

Site Domain antcat.org
Base Domain antcat.org
Scan Status Ok
Last Scan2025-12-09T03:13:31+00:00
Next Scan 2026-01-08T03:13:31+00:00

Last Scan

Scanned2025-12-09T03:13:31+00:00
URL https://antcat.org/robots.txt
Domain IPs 147.182.238.155
Response IP 147.182.238.155
Found Yes
Hash 031008aa8845a799fabd6a55df0969c45d767a4c348e3dd9016f15b65f0b99b9
SimHash 412dd490fe72

Groups

*

Rule Path
Disallow /activities/
Disallow /documents/
Disallow /feedbacks/
Disallow /references/exports/
Disallow *.pdf
Disallow /*history$
Disallow /*exports/wikipedia$

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

linguee

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mauibot (crawler.feedback+dc@gmail.com)

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot/6~bl

Rule Path
Disallow /
Back to top

Comments

  • https://ahrefs.com/robot
  • http://webmeup-crawler.com/
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • http://www.linguee.com/bot
  • http://megaindex.com/crawler
  • https://megaindex.com/crawler
  • https://aspiegel.com/petalbot
  • https://www.semrush.com/bot/
Back to top