softgoat.com
robots.txt

Robots Exclusion Standard data for softgoat.com

Resource Scan

Scan Details

Site Domain softgoat.com
Base Domain softgoat.com
Scan Status Ok
Last Scan5/6/2025, 4:00:15 AM
Next Scan 6/5/2025, 4:00:15 AM

Last Scan

Scanned5/6/2025, 4:00:15 AM
URL https://softgoat.com/robots.txt
Domain IPs 104.22.6.17, 104.22.7.17, 172.67.8.14, 2606:4700:10::6816:611, 2606:4700:10::6816:711, 2606:4700:10::ac43:80e
Response IP 104.22.6.17
Found Yes
Hash 83200a7310417066aa73d843db8a37b681d98a8f6ff53380335164aec0b3a633
SimHash 8614fcc25097

Groups

*

Rule Path
Disallow /selection
Disallow /selection/*

adsbot-google

Rule Path
Disallow /selection
Disallow /selection/*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

nutch

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

alligator

Rule Path
Disallow /

sq\ webscanner

Rule Path
Disallow /

allsubmitter

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

anonymous

Rule Path
Disallow /

asterias

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

badass

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

bitacle

Rule Path
Disallow /

dragonfly

Rule Path
Disallow /

dreampassport

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

ebrowse

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

edgeio

Rule Path
Disallow /

dreampassport

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

fsurf

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

gamespy_arcade

Rule Path
Disallow /

geniebot

Rule Path
Disallow /

getbot

Rule Path
Disallow /

getright

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

goforitbot

Rule Path
Disallow /

email\ extractor

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

jeteye

Rule Path
Disallow /

jeteyebot

Rule Path
Disallow /

jobo

Rule Path
Disallow /

joc\ web\ spider

Rule Path
Disallow /

kapere

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechget

Rule Path
Disallow /

lightningdownload

Rule Path
Disallow /

linkie

Rule Path
Disallow /

mac\ finder

Rule Path
Disallow /

mail\ sweeper

Rule Path
Disallow /

outfoxbot

Rule Path
Disallow /

omniexplorer

Rule Path
Disallow /

p3p

Rule Path
Disallow /

psycheclone

Rule Path
Disallow /

puf

Rule Path
Disallow /

pussycat

Rule Path
Disallow /

puxarapido

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

redkernel

Rule Path
Disallow /

relevantnoise

Rule Path
Disallow /

y!tunnelpro

Rule Path
Disallow /

yahooysmcm

Rule Path
Disallow /

zade

Rule Path
Disallow /

zbot

Rule Path
Disallow /

zerxbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.softgoat.com/sitemap.xml

Comments

  • Google Adsbot ignores robots.txt unless specifically named
  • Slow down some bots
  • Disallow bad bots
  • Fix for googlebot