Log In Sign Up

litpress.org
robots.txt

Robots Exclusion Standard data for litpress.org

Resource Scan

Scan Details

Site Domain litpress.org
Base Domain litpress.org
Scan Status Ok
Last Scan2024-09-19T05:12:36+00:00
Next Scan 2024-10-19T05:12:36+00:00

Last Scan

Scanned2024-09-19T05:12:36+00:00
URL https://litpress.org/robots.txt
Domain IPs 104.214.39.184
Response IP 104.214.39.184
Found Yes
Hash dff4991cd4c7eb51215c3eabcba6c656a5c4205450310e73768d2235858951fb
SimHash 4b79774b8d07

Groups

*

Rule Path
Disallow /Cart/
Disallow /Cart/*
Disallow /Catalogs/GetCatalog*
Disallow /Register/
Disallow /Products/Search
Disallow /MyAccount/
Disallow /Basket/
Disallow /Books/
Disallow /images/
Disallow /Controls/
Disallow /javascript/
Disallow /css/
Disallow /Content/
Disallow /pressroom/
Disallow /Marketing/
Disallow /excerpts/
Disallow /phpmyadmin
Disallow /Author/
Disallow /covers/
Disallow /Receipt.aspx
Disallow /ads.txt
Disallow /journals/bible_today.html
Disallow /about.html
Disallow /Subscriptions/
Disallow /Products/CategoryCenter/
Disallow /plus/
Disallow /Handlers/
Disallow /About_Us/
Disallow /Home/About
Disallow /About
Disallow /thumbs/
Disallow /wp-admin/
Disallow /Customer_Service/Catalogs/irs/default.html
Disallow /Products/GetImage/
Disallow /Products/GetSample/
Disallow /RequestCopy/
Disallow /*.aspx$
Disallow /StoreLocator
Disallow /MoreBy/
Disallow /MoreByCount?productid
Disallow /MoreBy?productid
Disallow /LPSuggest?productid
Disallow *start%3D*

Other Records

Field Value
crawl-delay 10

etaospider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

add catalog

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

spiceworks

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

ru_bot

Rule Path
Disallow /

wget

Rule Path
Disallow /

java/1.7.0_25

Rule Path
Disallow /

slurp

Rule Path
Disallow /

funwebproducts

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

acoirobot

Rule Path
Disallow /

acoon robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihit

Rule Path
Disallow /

alkalinebot

Rule Path
Disallow /

anzwerscrawl

Rule Path
Disallow /

arachnoidea

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

archive

Rule Path
Disallow /

autonomy spider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

benderthewebrobot

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bork-edition

Rule Path
Disallow /

bot mailto:craftbot@yahoo.com

Rule Path
Disallow /

botje

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

commoncrawl

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

covario

Rule Path
Disallow /

crawler

Rule Path
Disallow /

curl

Rule Path
Disallow /

custo

Rule Path
Disallow /

data mining development project

Rule Path
Disallow /

digext

Rule Path
Disallow /

disco

Rule Path
Disallow /

discobot

Rule Path
Disallow /

discoveryengine

Rule Path
Disallow /

doc

Rule Path
Disallow /

docomo

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

download demon

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

eurobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

express webpictures

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fetch

Rule Path
Disallow /

fetch api

Rule Path
Disallow /

filterdb

Rule Path
Disallow /

findfiles

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flightdeckreports

Rule Path
Disallow /

followsite bot

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

geniebot

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

gigablast

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

gt::www

Rule Path
Disallow /

hailoo

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hmview

Rule Path
Disallow /

houxou

Rule Path
Disallow /

http::lite

Rule Path
Disallow /

httrack

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ibm evv

Rule Path
Disallow /

id-search

Rule Path
Disallow /

idbot

Rule Path
Disallow /

image stripper

Rule Path
Disallow /

image sucker

Rule Path
Disallow /

indy library

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet ninja

Rule Path
Disallow /

internetmemory

Rule Path
Disallow /

isc systems irc search 2.1

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

larbin

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

libghttp

Rule Path
Disallow /

libwww

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

linko

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

metadatalabs

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

midown tool

Rule Path
Disallow /

missigua

Rule Path
Disallow /

missigua locator

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

morenet

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

naver

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

net vampire

Rule Path
Disallow /

netants

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netzip

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

offline navigator

Rule Path
Disallow /

omni-explorer

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

panscient

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

papa foto

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pecl::http

Rule Path
Disallow /

php/

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

picsearch

Rule Path
Disallow /

pipl

Rule Path
Disallow /

pmoz

Rule Path
Disallow /

predictyourbabysearchtoolbar

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

referrer karma

Rule Path
Disallow /

reget

Rule Path
Disallow /

reverseget

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

sogou

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchbot

Rule Path
Disallow /

seexie

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

servage robot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

sindice

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

smart.apnoti.com

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

snoopy

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

superpagesurlverifybot

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

synapse

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

uri::fetch

Rule Path
Disallow /

urllib

Rule Path
Disallow /

user-agent

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

web sucker

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webgo is

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website extractor

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wells search ii

Rule Path
Disallow /

wep search

Rule Path
Disallow /

widow

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

ybot

Rule Path
Disallow /

yesupbot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yolinkbot

Rule Path
Disallow /

youdao

Rule Path
Disallow /

zao

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Allow /Products
Back to top

Other Records

Field Value
sitemap https://litpress.org/sitemap.xml
Back to top

Warnings

  • 3 invalid lines.
Back to top