mailleartisans.org
robots.txt

Robots Exclusion Standard data for mailleartisans.org

Resource Scan

Scan Details

Site Domain mailleartisans.org
Base Domain mailleartisans.org
Scan Status Ok
Last Scan2025-09-01T18:16:12+00:00
Next Scan 2025-10-01T18:16:12+00:00

Last Scan

Scanned2025-09-01T18:16:12+00:00
URL https://mailleartisans.org/robots.txt
Redirect https://www.mailleartisans.org/robots.txt
Redirect Domain www.mailleartisans.org
Redirect Base mailleartisans.org
Domain IPs 198.72.127.235
Redirect IPs 198.72.127.235
Response IP 198.72.127.235
Found Yes
Hash 062f087a4ae054546686c3aefc2a8a5d38db3bc2de88743fca8627360bfa94c1
SimHash e47c1b9289b2

Groups

*

Rule Path
Disallow /articles/articlesubmit.php
Disallow /articles/articleedit.php
Disallow /weaves/weavesubmit.php
Disallow /weaves/weavedit.php
Disallow /gallery/gallerysubmit.php
Disallow /gallery/galleryedit.php
Disallow /lynx/linksmaintain.cgi
Disallow /teaching/teachingmaintain.cgi
Disallow /glossary/glossarymaintain.cgi
Disallow /members/register.cgi
Disallow /members/register.php
Disallow /members/register_submit.php

*

Rule Path
Disallow /board/admin/
Disallow /board/images/
Disallow /board/includes/
Disallow /board/language/
Disallow /board/templates/
Disallow /board/common.php
Disallow /board/config.php
Disallow /board/groupcp.php
Disallow /board/memberlist.php
Disallow /board/modcp.php
Disallow /board/posting.php
Disallow /board/profile.php
Disallow /board/privmsg.php
Disallow /board/viewonline.php
Disallow /board/search.php
Disallow /board/faq.php

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows nt; digext; dts agent

Rule Path
Disallow /

Comments

  • Spider Crawling File
  • Forum Stuff
  • Attempt ALL Bot Slowdown
  • Kill YanDex
  • Kill Goo
  • Kill Naver
  • Kill Baidu
  • Kill SoGou
  • Kill Youdao
  • Kill WebReaper
  • Kill DTS Agent