Log In Sign Up

dcurbanmom.com
robots.txt

Robots Exclusion Standard data for dcurbanmom.com

Resource Scan

Scan Details

Site Domain dcurbanmom.com
Base Domain dcurbanmom.com
Scan Status Ok
Last Scan2024-11-02T12:36:02+00:00
Next Scan 2024-11-09T12:36:02+00:00

Last Scan

Scanned2024-11-02T12:36:02+00:00
URL https://dcurbanmom.com/robots.txt
Redirect https://www.dcurbanmom.com/robots.txt
Redirect Domain www.dcurbanmom.com
Redirect Base dcurbanmom.com
Domain IPs 209.190.218.10
Redirect IPs 209.190.218.10
Response IP 209.190.218.10
Found Yes
Hash 1004776e5ba1343ddc7c2c3a5571ed50b76e2a8b17a952ff4824f9209409617f
SimHash 2613daba4512

Groups

applebot

Product Comment
applebot apple
Rule Path
Disallow /reportpost

boardreader

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /search_rss
Disallow /search
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /login
Disallow /login_form

msnbot-udiscovery

Rule Path
Disallow /search_rss
Disallow /search
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /author
Disallow /login
Disallow /login_form

msnbot-udiscovery/2.0b

Rule Path
Disallow /search_rss
Disallow /search
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /login
Disallow /login_form

googlebot-image

Rule Path
Disallow /search_rss
Disallow /search
Disallow /jforum

technoratibot

Rule Path
Disallow /Members
Disallow /search_rss
Disallow /search

irlbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /search_rss
Disallow /search
Disallow /events/aggregator/atct_topic_view
Disallow /jforum/jforum.page?module=search
Disallow /login
Disallow /login_form

googlebot

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /search_rss
Disallow /search
Disallow /events/aggregator/atct_topic_view
Disallow /jforum/jforum.page?module=search
Disallow /login
Disallow /login_form

googlebot-image

Rule Path
Disallow /Members
Disallow /search_rss
Disallow /search
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

mediapartners-google

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

msnbot

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /login
Disallow /login_form

slurp

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /login
Disallow /login_form

teoma

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

gigabot

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

scrubby

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

robozilla

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

nutch

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser

ia_archiver

Rule Path
Disallow /Members

baiduspider

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /search_rss
Disallow /search

yahoo-mmcrawler

Rule Path
Disallow /Members
Disallow /jforum/captcha
Disallow /jforum/posts/listByUser
Disallow jforum/recentTopics/showTopicsByUser
Disallow /search_rss
Disallow /search
Disallow /login
Disallow /login_form

anthropic-ai
ccbot
claude-web
facebookbot
gptbot
google-extended
piplbot

Rule Path
Disallow /
Back to top