creditwritedowns.com
robots.txt

Robots Exclusion Standard data for creditwritedowns.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	creditwritedowns.com
Base Domain	creditwritedowns.com
Scan Status	Ok
Last Scan	2024-05-13T06:06:44+00:00
Next Scan	2024-05-20T06:06:44+00:00

Last Scan

Scanned	2024-05-13T06:06:44+00:00
URL	https://creditwritedowns.com/robots.txt
Domain IPs	172.66.40.76, 172.66.43.180, 2606:4700:3108::ac42:284c, 2606:4700:3108::ac42:2bb4
Response IP	172.66.40.76
Found	Yes
Hash	d205135b69922aef2b992952e8d8eefa086ae4e7d6daa8d15b175fe814f68cc8
SimHash	30087d7f4622

Groups

*

Rule	Path
Disallow	/?blackhole
Disallow	/comments/feed/
Disallow	/index.php
Disallow	/wp-admin/
Disallow	/trackback/
Disallow	/xmlrpc.php
Disallow	/feed
Disallow	/2018/03/do-not-click-page-you-will-be-banned-testing-rss-only.html

Rule

Path

Disallow

/?blackhole

Disallow

/comments/feed/

Disallow

/index.php

Disallow

/wp-admin/

Disallow

/trackback/

Disallow

/xmlrpc.php

Disallow

/feed

Disallow

/2018/03/do-not-click-page-you-will-be-banned-testing-rss-only.html

msnbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazoncloudservices

Rule	Path
Disallow	/

Rule

Path

Disallow

awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-video

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

bazqux

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

cliqzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

crawl13.exabot.com

Rule	Path
Disallow	/

Rule

Path

Disallow

crawl6.exabot.com

Rule	Path
Disallow	/

Rule

Path

Disallow

curl

Rule	Path
Disallow	/

Rule

Path

Disallow

daum

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot-images/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot-images/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

fetch api request

Rule	Path
Disallow	/

Rule

Path

Disallow

gt::www

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

http::lite

Rule	Path
Disallow	/

Rule

Path

Disallow

http client

Rule	Path
Disallow	/

Rule

Path

Disallow

http/1.1 zibber-v0.1(www.zibb.com/crawler/)

Rule	Path
Disallow	/

Rule

Path

Disallow

httplib

Rule	Path
Disallow	/

Rule

Path

Disallow

humanlinks

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archive

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver-web.archive.org

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver/1.6

Rule	Path
Disallow	/

Rule

Path

Disallow

inoreader

Rule	Path
Disallow	/

Rule

Path

Disallow

java

Rule	Path
Disallow	/

Rule

Path

Disallow

javabot

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.4.1_04

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.6.0_04

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.5.0_05

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.5.0_06

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.6.0_07

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.5.0_10

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.6.0_11

Rule	Path
Disallow	/

Rule

Path

Disallow

java/1.6.0_13

Rule

Path

Disallow

java bot 2.0

Rule

Path

Disallow

laserlikebot

Rule

Path

Disallow

libcurl-agent

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

libwww

Rule

Path

Disallow

libwww-perl

Rule

Path

Disallow

libwww-perl/5.65

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

linkdexbot/2.0

Rule

Path

Disallow

linkdexbot/2.1

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwp-trivial/1.34

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

manzama

Rule

Path

Disallow

mediatoolkitbot

Rule

Path

Disallow

mfc_tear_sample

Rule

Path

Disallow

mj12

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mj12bot/v1.2.0

Rule

Path

Disallow

mozilla

Rule

Path

Disallow

mozilla/2.0 (compatible; ask jeeves/teoma)

Rule

Path

Disallow

mozilla/3

Rule

Path

Disallow

mozilla/4

Rule

Path

Disallow

mozilla/4.0

Rule

Path

Disallow

mozilla/4.0 (compatible; bullseye; windows 95)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows 2000)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows 9

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows 95)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows 98)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows nt)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.0; windows xp)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.05; windows nt 3.51)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.05; windows nt 4.0)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.05; windows nt 5.0)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 6.0; windows nt 5.0)

Rule

Path

Disallow

mozilla/4.0+compatible+zyborg/1.0+

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

noozilla 1.1

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

offline explorer/1.5

Rule

Path

Disallow

offline explorer/1.9

Rule

Path

Disallow

offline explorer/2.5

Rule

Path

Disallow

omniexplorer

Rule

Path

Disallow

omniexplorer_bot

Rule

Path

Disallow

omniexplorer_bot/1.07

Rule

Path

Disallow

omniexplorer_bot/1.09

Rule

Path

Disallow

omniexplorer_bot/1.10

Rule

Path

Disallow

pecl::http

Rule

Path

Disallow

perl

Rule

Path

Disallow

php/

Rule

Path

Disallow

phpcrawl

Rule

Path

Disallow

phpversion

Rule

Path

Disallow

protopage

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

screaming frog seo spider/9.0

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

simplepie

Rule

Path

Disallow

socialrankiobot

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

sogou+spider

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

trendictionbot0.5.0

Rule

Path

Disallow

tweetmemebot/4.0

Rule

Path

Disallow

uri::fetch

Rule

Path

Disallow

urllib

Rule

Path

Disallow

web downloader/4.5

Rule

Path

Disallow

web downloader/5.7

Rule

Path

Disallow

web downloader/6.9

Rule

Path

Disallow

web image collector

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

webcollage

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webcopier v3.2a

Rule

Path

Disallow

webcopier v3.3

Rule

Path

Disallow

webcopier v3.3.0

Rule

Path

Disallow

webcopier v3.3.2

Rule

Path

Disallow

webcrawler

Rule

Path

Disallow

wget

Rule

Path

Disallow

wget/1.10.2

Rule

Path

Disallow

wget/1.5.3

Rule

Path

Disallow

wget/1.6

Rule

Path

Disallow

wget/1.9.1

Rule

Path

Disallow

wget/1.9+cvs-stable (red hat modified)

Rule

Path

Disallow

whois.de

Rule

Path

Disallow

whoisde.de

Rule

Path

Disallow

yandex

Rule

Path

Disallow

yandex/1.01.001 (compatible; win16; m)

Rule

Path

Disallow

yandex/1.01.001 (compatible; win16; h)

Rule

Path

Disallow

yandex/1.01.001 (compatible; win16; p)

Rule

Path

Disallow

yandex/1.01.001 (compatible; win16; i)

Rule

Path

Disallow

yandexmobilebot

Rule

Path

Disallow

yandexmobilebot/3.0

Rule

Path

Disallow

zend_http_client

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://pro.creditwritedowns.com/sitemap_index.xml

Comments

Now for the list of outright bans

Warnings

2 invalid lines.

creditwritedowns.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

msnbot

Other Records

ahrefsbot

amazoncloudservices

awariosmartbot

baiduspider

baiduspider/2.0

baiduspider-video

baiduspider-image

bazqux

blexbot

blexbot/1.0

blp_bbot

blp_bbot/0.1

cliqzbot

crawl13.exabot.com

crawl6.exabot.com

curl

daum

dotbot

dotbot

exabot

exabot

exabot-images/1.0

exabot-images/3.0

exabot/2.0

exabot/3.0

fetch api request

gt::www

heritrix

http::lite

http client

http/1.1 zibber-v0.1(www.zibb.com/crawler/)

httplib

humanlinks

ia_archive

ia_archiver-web.archive.org

ia_archiver/1.6

inoreader

java

javabot

java/1.4.1_04

java/1.6.0_04

java/1.5.0_05

java/1.5.0_06

java/1.6.0_07

java/1.5.0_10

java/1.6.0_11

java/1.6.0_13

java bot 2.0

laserlikebot

libcurl-agent

libweb/clshttp

libwww

libwww-perl

libwww-perl/5.65

linkdexbot

linkdexbot/2.0

linkdexbot/2.1

lwp-trivial

lwp-trivial/1.34

magpie-crawler

manzama

mediatoolkitbot

mfc_tear_sample

mj12

mj12bot

mj12bot/v1.2.0

mozilla

mozilla/2.0 (compatible; ask jeeves/teoma)

mozilla/3

mozilla/4

mozilla/4.0

mozilla/4.0 (compatible; bullseye; windows 95)

creditwritedowns.com
robots.txt