Log In Sign Up

kwasistudios.com
robots.txt

Robots Exclusion Standard data for kwasistudios.com

Resource Scan

Scan Details

Site Domain kwasistudios.com
Base Domain kwasistudios.com
Scan Status Ok
Last Scan2025-09-23T20:08:13+00:00
Next Scan 2025-10-07T20:08:13+00:00

Last Scan

Scanned2025-09-23T20:08:13+00:00
URL https://kwasistudios.com/robots.txt
Redirect https://kwasi.com/robots.txt
Redirect Domain kwasi.com
Redirect Base kwasi.com
Domain IPs 43.250.142.135
Redirect IPs 54.79.191.104
Response IP 54.79.191.104
Found Yes
Hash fadc7e79a4bd694d55046fb37fcb93e67392ad6de5b347af69f03972fd33eadc
SimHash c43c594bed85

Groups

doc

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

zao

Rule Path
Disallow /

fetch

Rule Path
Disallow /

httrack

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linko

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

xenu

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

wget

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

*

Rule Path
Disallow /*.php
Disallow /*/?comments=true
Disallow /*/?nomobile
Disallow /*/?postcomment=true
Disallow /*/?replytocom=*
Disallow /*/feed*
Disallow /*/feed/
Disallow /*/trackback
Disallow /*/trackback/
Disallow /?cat=
Disallow /?cat=*
Disallow /?p=*
Disallow /?s=*
Disallow /edit.php?*
Disallow /feed*
Disallow /over*
Disallow /statistics
Disallow /tweets
Disallow /tweets/
Disallow /tweets/*
Disallow /wp-admin
Disallow /wp-admin/*
Disallow /wp-login.php
Disallow /document/
Disallow /examples/
Disallow /js/
Disallow /LOG_DIR/
Disallow /logs/
Disallow /cgi-bin/
Disallow /email/*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://kwasi.com/sitemap_index.xml
Back to top

Comments

  • A list of misbehaving crawlers.
  • originally from http://aardling.com/robots.txt
  • Some bots are known to be trouble, particularly those designed to copy entire sites.
  • Wget in its recursive mode is a frequent problem.
  • Hits many times per second, not acceptable.
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit.
  • http://www.webreaper.net/
  • These rules apply to everyone else.
  • Disallow: /*/page/
  • Disallow: /*/page/*
  • Disallow: /*/page/*/
  • And IIIIIIIIII EEEEE IIIIIII WILL ALWAYS LOOOOOOOOOOOOVE YOUUUUUUUUUUUUUUUU
  • RIP KWASI BOT - WE WON'T FORGET YOU
  • /#, ./(
  • % .. ,
  • .%........... (
  • (...........
  • %,.,,,,,/*** *,
  • %%.,,,,,#**
  • **(
  • (***(
  • ****
  • ..,*#((////(
  • *############(/*,... ....,*/(#############*
  • ,######/ ...,,,,.,,,,,,,,,,,,,,,,,,,,.... *######*
  • ,%% .........,......,,,,,,.,,,,,,,,,,,,,,,,,,,,,,,,.,,,,.. ,##(
  • %% , ..........,...... .,.,,,,,,,,,,,,,,,,,,,.,,,,,,,,,,,,,,.
  • *%% (##( .............. ,,,,,,.,,,,,,,,,, *,,,,,,,,,,,, **
  • .%% ####(( ............. ,,,,......... ,,,,,,,,,,
  • %% #####((( ........... (%##% .......... ,,,,,,,,,, #.
  • .%.#(####((((/.......... ( %(% ........ ,%#/ ,,,,,. ,.
  • ,% #(####((((## ........* . ,#%%%(# ........ . ( .,,,,,
  • *%.(####(((#### ........ %# /#####(% ,....* #%%#, ..,,,, (
  • %,####(((##### ,,.....* %%, *#######(# ..... #. /###(% ,...,,, ,
  • (%/###(((((((##,,,,,,..* .%##(((((((((((#(% ,.,* %#( ######(, ....., #,
  • *%###(((((((((((,,,.,,,* /###(((((((((((#(% *../ .##(((((((((((( ...... #/
  • %%((((((((((((#.,,,,,,* *%###((((((((((#(% *../ .##(((((((((#(( ....... #(
  • %%/(((((((((//# ,,,,,,* %####((((((((#### *,,* %###((((((##(, ....... #(
  • %%/(((/((((///( .....,* #%######((####(% *,,, ###########(% ....... #(
  • *%/(((((((//////........ #%##########(% .,,, %#########%. ........ #(
  • %%/(((((/////(#.......* ,%########%( /,,,,* %#######% *....... #*
  • %%/((((////(((# ......./ %%%%%%. /,,,,,./ .#%%* /,....... #.
  • .%/(((////((/(//......../ /.,,,,,,,/ *,,,......,
  • %%((///(((/(//#........., /........,,,. /,,.,,,,,,.(
  • %(///(((/(///( ..........,/ *..............*. *.,,,,,,,,,,
  • %//(((//(/////(............../(/**//,....................,/. */,...,,,,,,,,,, #(
  • %#((//((/////(# ..........................................................,,,,
  • %(//((///////(*.............................................................,.
  • %//((////(////# .... .,*(#%%%%%%%%%%%%%%%%%%%%%%%%%%%(/*,................(/
  • ,%((///(////#%&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&%%%#######%%&&&&&&%%%%(,
  • %(//(//(%##%%&&&&&&&&&&&&&&&&&&&&&&&&%%%%%%%%%%%%######%%%&&&&&&&&&&&&&&%%&.
  • .%/*/(#&&&&&%%%%&&&&&&&&&&&&&%(#*****,,,******//*.%&&&&&&&&&&&&&&&%####&,
  • %//&&&&&&&&&&&&&&&&&&%%%%*,/*,,,,..,,,,,,,.,,,,*/ %%&&&&%#########&&.
  • %&%##############%%%% ...,.,/,,,,.*((/**,,,,*/(/* (%% ./#%&*
  • ,%%* ..,/*,,,,,,,.*****,,,,,,,,,,** /%%%% *%%%%%.
  • %%. ,,,,,,,(,,,,,,,(****,,******,,** %%%**,%%/ % ,,,,,.(%
  • /% ..,,,,,,,*#,,,,,,.,,..**.....*,,*. %%*/,.,. .,,/%%%.
  • %# ,.,,,,,,,%/**,,,,,,,,,(**/...(*/,,* %%#**.,,/.,,.****(/
  • %% ,,,,****,%/***,*,,,,,,,..........,* %%#/*,,*/,, *******,,,,,,.,%(
  • % ,,,*****/,*%*****,*,,,,,........,, %%%(%///*/, *****/..,*(##*.,.%
  • %( ,****/**///#**********,,,,,,,,,** #%% %%,// ,,/(/,.,,,...*%%*
  • %% ////**///% ******************* %%% .(, **,,,,,.(#,*,%
  • .%%/**//(%.****/(************/%% %%,******,%
  • %/*****///(******.%%*%( #%%%%%
  • %#*****////////(/////%
  • %/(#####///////////*%
  • % **///////%(###(//*%,
  • *%%%%%%#* .# /////////#///////*% (%%%%%%%%/
  • %%%* ... ,%%%. /#####(///#///(####( %%.,*****, %%%
  • %%/ *************/////%(////////%/////////////////////******** (%%
  • %%, *///////////////////////(//////////////////////////////////*///, (%.
  • .......#% .**////////////////////////////(%%..,,,,,....,%%%(///////////////// %,....
  • ........,%%. *************//////////%%,......................./%%%%%(*********** %......
  • .. ...,%%%%%%%%%%%%%%%%%%%%%%%,.................................,/#%%%%%%%%#,......
  • WPStaq Auto-Added Rules - Start
  • WPStaq Auto-Added Rules - End
Back to top