ghaya.online
robots.txt

Robots Exclusion Standard data for ghaya.online

Resource Scan

Scan Details

Site Domain ghaya.online
Base Domain ghaya.online
Scan Status Ok
Last Scan2025-04-07T12:13:34+00:00
Next Scan 2025-05-07T12:13:34+00:00

Last Scan

Scanned2025-04-07T12:13:34+00:00
URL https://ghaya.online/robots.txt
Domain IPs 67.225.152.36
Response IP 67.225.152.36
Found Yes
Hash ae229aaa42bb7e74bb8c0d74909fab529347f92b53788690e9eda76662293187
SimHash eb29b3678b67

Groups

*

Rule Path
Allow /
Disallow *.*
Disallow /.htaccess
Disallow /cgi-bin/$
Disallow */wp-*
Disallow /wp-content/$
Allow /wp-content/uploads/$
Allow /wp-content/plugins/$
Disallow /wp-content/plugins/wysija-newsletters/$
Disallow /wp-content/uploads/$
Allow /wp-content/themes/$
Allow /wp-content/cache/$
Disallow /wp-includes/$
Allow /wp-includes/js/$
Allow /wp-includes/images/$
Disallow */feed/$
Disallow /tmp/$
Disallow */comments
Disallow */trackback/$
Disallow */recomended/$
Disallow /wp-*
Disallow /wp-admin/$
Disallow *?wptheme=
Disallow *?comments=
Disallow *?replytocom
Allow *.css$
Allow *.js$
Allow *.html$
Allow *.htm$
Allow *.doc$
Allow *.docx$
Allow *.xls$
Allow *.xlt$
Allow *.xlsx$
Allow *.pp$
Allow *.sldx$
Allow *.pub$
Allow *.inc$
Allow *.wmv$
Allow *.gif$
Allow *.jpg$
Allow *.jpeg$
Allow *.pdf$
Allow *.png$
Allow *.wmv$
Allow *.tar$
Allow *.gz$
Allow *.tar$
Allow *.gz.tar$
Allow *.zip$
Allow *.xml$
Disallow /readme.html
Disallow /license.txt
Disallow .php$
Disallow *.*.php$
Allow /wp-cron.php
Allow /wp-feed.php
Allow /wp-login.php
Allow /xmlrpc.php
Allow /index.php
Allow /wp-admin/$
Disallow /%22

duggmirror

Rule Path
Disallow /

almaden

Rule Path
Disallow /

anarchie

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

attach

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

backweb

Rule Path
Disallow /

bandit

Rule Path
Disallow /

batchftp

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bot\ mailto:craftbot@yahoo.com

Rule Path
Disallow /

buddy

Rule Path
Disallow /

bumblebee

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

cicc

Rule Path
Disallow /

collector

Rule Path
Disallow /

copier

Rule Path
Disallow /

crescent

Rule Path
Disallow /

custo

Rule Path
Disallow /

da

Rule Path
Disallow /

diibot

Rule Path
Disallow /

disco

Rule Path
Disallow /

disco\ pump

Rule Path
Disallow /

download\ demon

Rule Path
Disallow /

download\ wonder

Rule Path
Disallow /

downloader

Rule Path
Disallow /

drip

Rule Path
Disallow /

dsurf15a

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

easydl/2.99

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

email [nc,or]

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

express\ webpictures

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

filehound

Rule Path
Disallow /

flashget

Rule Path
Disallow /

frontpage [nc,or]

Rule Path
Disallow /

getright

Rule Path
Disallow /

getsmart

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

gigabaz

Rule Path
Disallow /

go\!zilla

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

gotit

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

hmview

Rule Path
Disallow /

httrack

Rule Path
Disallow /

httpdown

Rule Path
Disallow /

image\ stripper

Rule Path
Disallow /

image\ sucker

Rule Path
Disallow /

indy*library

Rule Path
Disallow /

indy\ library [nc,or]

Rule Path
Disallow /

interget

Rule Path
Disallow /

internetlinkagent

Rule Path
Disallow /

internet\ ninja

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

iria

Rule Path
Disallow /

jbh*agent

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

joc\ web\ spider

Rule Path
Disallow /

justview

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lftp

Rule Path
Disallow /

link*sleuth

Rule Path
Disallow /

likse

Rule Path
Disallow /

link

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

magnet

Rule Path
Disallow /

mass\ downloader

Rule Path
Disallow /

memo

Rule Path
Disallow /

midown\ tool

Rule Path
Disallow /

mirror

Rule Path
Disallow /

mister\ pix

Rule Path
Disallow /

mozilla.*indy

Rule Path
Disallow /

mozilla.*newt

Rule Path
Disallow /

mozilla*msiecrawler

Rule Path
Disallow /

ms\ frontpage*

Rule Path
Disallow /

msfrontpage

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msproxy

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

netants

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netspider

Rule Path
Disallow /

net\ vampire

Rule Path
Disallow /

netzip

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

ninja

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline\ explorer

Rule Path
Disallow /

offline\ navigator

Rule Path
Disallow /

openfind

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

papa\ foto

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

ping

Rule Path
Disallow /

pingalink

Rule Path
Disallow /

pockey

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pump

Rule Path
Disallow /

qrva

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reaper

Rule Path
Disallow /

recorder

Rule Path
Disallow /

reget

Rule Path
Disallow /

scooter

Rule Path
Disallow /

seeker

Rule Path
Disallow /

siphon

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

snake

Rule Path
Disallow /

spacebison

Rule Path
Disallow /

sproose

Rule Path
Disallow /

stripper

Rule Path
Disallow /

sucker

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport\ pro

Rule Path
Disallow /

urlspiderpro

Rule Path
Disallow /

vacuum

Rule Path
Disallow /

voideye

Rule Path
Disallow /

web\ image\ collector

Rule Path
Disallow /

web\ sucker

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

web\ downloader

Rule Path
Disallow /

webemailextrac.*

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webgo\ is

Rule Path
Disallow /

webhook

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webminer

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website

Rule Path
Disallow /

website\ extractor

Rule Path
Disallow /

website\ quester

Rule Path
Disallow /

webster

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

whacker

Rule Path
Disallow /

widow

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

x-tractor

Rule Path
Disallow /

xaldon\ webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

zeus.*webster

Rule Path
Disallow /

zeus

Rule Path
Disallow /

Other Records

Field Value
sitemap http://motivationalspeakers4u.co.uk/sitemapindex.xml

Comments

  • Robots.txt wont keep hackers out. A bot is not forced to read it. It is only used by nice bots.
  • Crawl-delay: 5
  • Uncomment the below and change to reference you sitemap
  • Lets disallow stuff that could go wrong
  • should already be disallowed
  • Disallow: *.cgi
  • Disallow: *.exe
  • Disallow: *.xds
  • Disallow: *.xls
  • Disallow: *.bmp
  • Disallow: *.xhtml
  • needed to protect .php in the folders above, but has to be here to stop hacks such as jpg.php
  • ALLOW STUFF THROUGH
  • Allow: /wp-admin/admin.php
  • Allow: /wp-admin/admin-ajax.php
  • Allow: /wp-admin/load-styles.php
  • Allow: /wp-admin/load-scripts.php
  • Allow: /wp-admin/about.php
  • Allow: /wp-admin/media-new.php
  • Allow: /wp-admin/post-new.php
  • Allow: /wp-admin/page.php
  • Allow: /wp-admin/themes.php
  • Allow: /wp-admin/profile.php
  • Allow: /wp-admin/update-core.php
  • disallow urls starting with quote
  • DISALOW BOTS (not really needed but helps)
  • Internet Archiver Wayback Machine
  • User-agent: ia_archiver
  • Disallow: /

Warnings

  • 4 invalid lines.