Log In Sign Up

cashflow-manager.com.au
robots.txt

Robots Exclusion Standard data for cashflow-manager.com.au

Resource Scan

Scan Details

Site Domain cashflow-manager.com.au
Base Domain cashflow-manager.com.au
Scan Status Ok
Last Scan2024-04-21T22:01:07+00:00
Next Scan 2024-05-21T22:01:07+00:00

Last Scan

Scanned2024-04-21T22:01:07+00:00
URL https://cashflow-manager.com.au/robots.txt
Domain IPs 2600:9000:260f:1200:1a:5d9d:1d40:93a1, 2600:9000:260f:1400:1a:5d9d:1d40:93a1, 2600:9000:260f:7e00:1a:5d9d:1d40:93a1, 2600:9000:260f:a600:1a:5d9d:1d40:93a1, 2600:9000:260f:b000:1a:5d9d:1d40:93a1, 2600:9000:260f:b800:1a:5d9d:1d40:93a1, 2600:9000:260f:e800:1a:5d9d:1d40:93a1, 2600:9000:260f:ec00:1a:5d9d:1d40:93a1, 3.163.24.109, 3.163.24.116, 3.163.24.29, 3.163.24.81
Response IP 13.226.2.6
Found Yes
Hash a50498bdc8ba5f90f7faef3618719110b805dd2073b27aee9f91cec4489d0870
SimHash e5347949c5c7

Groups

*

Rule Path
Disallow /cfmdevsite.cashflowmanagerwebinars.com.au/

doc

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

zao

Rule Path
Disallow /

fetch

Rule Path
Disallow /

httrack

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linko

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

xenu

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

wget

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

*

Rule Path
Disallow /*.php
Disallow /*/?comments=true
Disallow /*/?nomobile
Disallow /*/?postcomment=true
Disallow /*/?replytocom=*
Disallow /*/feed*
Disallow /*/feed/
Disallow /*/trackback
Disallow /*/trackback/
Disallow /?cat=
Disallow /?cat=*
Disallow /?p=*
Disallow /?s=*
Disallow /edit.php?*
Disallow /feed*
Disallow /over*
Disallow /statistics
Disallow /tweets
Disallow /tweets/
Disallow /tweets/*
Disallow /wp-admin
Disallow /wp-admin/*
Disallow /wp-login.php
Disallow /document/
Disallow /examples/
Disallow /js/
Disallow /LOG_DIR/
Disallow /logs/
Disallow /cgi-bin/
Disallow /author/*
Back to top

Other Records

Field Value
sitemap https://www.cashflow-manager.com.au/sitemap.xml
Back to top

Comments

  • A list of misbehaving crawlers.
  • originally from http://aardling.com/robots.txt
  • Some bots are known to be trouble, particularly those designed to copy entire sites.
  • Wget in its recursive mode is a frequent problem.
  • Hits many times per second, not acceptable.
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit.
  • http://www.webreaper.net/
  • These rules apply to everyone else.
  • Disallow: /*/page/
  • Disallow: /*/page/*
  • Disallow: /*/page/*/
Back to top