Log In Sign Up

spiritualsingles.com
robots.txt

Robots Exclusion Standard data for spiritualsingles.com

Resource Scan

Scan Details

Site Domain spiritualsingles.com
Base Domain spiritualsingles.com
Scan Status Ok
Last Scan2024-10-28T06:22:20+00:00
Next Scan 2024-11-27T06:22:20+00:00

Last Scan

Scanned2024-10-28T06:22:20+00:00
URL https://spiritualsingles.com/robots.txt
Redirect https://www.spiritualsingles.com/robots.txt
Redirect Domain www.spiritualsingles.com
Redirect Base spiritualsingles.com
Domain IPs 66.206.21.244
Redirect IPs 66.206.21.244
Response IP 66.206.21.244
Found Yes
Hash 51a59ad779e68177148be0a921c574df2baf16908d4d4ba7886246464b546b83
SimHash 5ab497704bf3

Groups

blp_bbot/0.1

Rule Path
Disallow /

libreprensabot/1.0

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

xenu link sleuth/1.3.8

Rule Path
Disallow /

owlin bot

Rule Path
Disallow /

superfeedr bot/2.0

Rule Path
Disallow /

flipboard/3.2.6 cfnetwork/711.0.6 darwin/14.0.0

Rule Path
Disallow /

simplepie/1.1.3

Rule Path
Disallow /

rome client

Rule Path
Disallow /

genieo/1.0

Rule Path
Disallow /

snarfer/1.0.2

Rule Path
Disallow /

moodlebot/1.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bbot

Rule Path
Disallow /

blp_bbot/0.1

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

psbot/0.1

Rule Path
Disallow /

psbot

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

yeti

Rule Path
Disallow /

snapbot/1.0

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

superfeedr

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

funwebproducts

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

trustpilotcrawler

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

blogsearch

Rule Path
Disallow /

powermarks

Rule Path
Disallow /

blogsearch/2 +http://www.icerocket.com/

Rule Path
Disallow /

jakarta commons-httpclient/3.1

Rule Path
Disallow /

rome client (http://tinyurl.com/64t5n) ver: 0.9

Rule Path
Disallow /

rome client

Rule Path
Disallow /

*

Rule Path
Disallow /author/*
Disallow /xtras/*
Back to top