gjw.com
robots.txt

Robots Exclusion Standard data for gjw.com

Resource Scan

Scan Details

Site Domain gjw.com
Base Domain gjw.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-11-01T09:10:23+00:00
Next Scan 2026-01-30T09:10:23+00:00

Last Successful Scan

Scanned2022-12-23T11:06:16+00:00
URL https://gjw.com/robots.txt
Domain IPs 117.149.203.229, 123.129.227.229
Response IP 123.129.227.229
Found Yes
Hash fab1add3a04a3b36d06856a7e023bada2979d70f2583b495845ed239da087f07
SimHash 431e437e4932

Groups

abonti

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

bpimagewalker

Rule Path
Disallow /

check_http

Rule Path
Disallow /

checks.panopta.com

Rule Path
Disallow /

curl

Rule Path
Disallow /

dle_spider.exe

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

email exractor

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

gbplugin

Rule Path
Disallow /

gooblog

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

holmesbot

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

infoseek sidewinder

Rule Path
Disallow /

infoseek sidewinder/2.0b (linux 2.4 i686)

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

jakarta commons-httpclient

Rule Path
Disallow /

java

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

pear http_request class

Rule Path
Disallow /

php

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

rmnbot

Rule Path
Disallow /

ruby

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seoengworldbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

squider

Rule Path
Disallow /

ssearch_bot

Rule Path
Disallow /

suchmaschinenoptimierung.de

Rule Path
Disallow /

synapse

Rule Path
Disallow /

wget

Rule Path
Disallow /

woopingbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yeti

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

*

Rule Path
Disallow /99bill/
Disallow /alipay/
Disallow /api/
Disallow /Dialog/
Disallow /fckeditor/
Disallow /orders/
Disallow /Tenpay/
Disallow /tracert/
Disallow /user/
Disallow /login/
Disallow /script/
Disallow /ajax/
Disallow /Error/
Disallow /purchase/
Disallow /trydrink/item-id-*.htm
Disallow /error/404.htm
Disallow /*.aspx*