softcreatr.dev
robots.txt

Robots Exclusion Standard data for softcreatr.dev

Resource Scan

Scan Details

Site Domain softcreatr.dev
Base Domain softcreatr.dev
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned an unexpected status.
Last Scan2025-04-08T05:37:35+00:00
Next Scan 2025-07-07T05:37:35+00:00

Last Successful Scan

Scanned2024-12-03T05:20:21+00:00
URL https://softcreatr.dev/robots.txt
Domain IPs 168.119.156.34, 2a01:4f8:c012:cffa::1
Response IP 168.119.156.34
Found Yes
Hash 9f14b8290aaadb9fc20a42e47bd616857b137772f685d5dea001875a0b8baff5
SimHash 3339c152ec02

Groups

*

Rule Path
Allow *
Disallow /*/acp/
Disallow /*/attachments/
Disallow /*/cache/
Disallow /*/language/
Disallow /*/lib/
Disallow /*/log/
Disallow /*/templates/
Disallow /*/tmp/
Disallow /*/user/
Disallow /*/register/
Disallow /*/login/
Disallow /*/lost-password/
Disallow /*/github-auth/
Disallow /*/twitter-auth/
Disallow /*/google-auth/
Disallow /*/facebook-auth/
Disallow /*?*postID=
Disallow /*?*tagID=
Disallow /*?*pageNo=1$
Disallow /*?*action=firstNew
Disallow /*?*action=lastPost
Disallow /*/search-result/
Disallow /*/search/
Disallow /*/tag-search/
Disallow /*/tagged/
Disallow /*/user-search/
Disallow /*/members-list/
Disallow /*?*sortField=
Disallow /*?*sortOrder=
Disallow /*?*filterByDate=
Disallow /*?*filterByStatus=
Disallow /*?*highlight=
Disallow /*?*objectType=
Disallow /*?*utm_source=
Disallow /*?*utm_medium=
Disallow /*?*utm_campaign=
Disallow /*/product-terms-and-conditions/
Disallow /*/cancellation-policy/
Disallow /*/sc-terms-of-use/
Disallow /*/legal-notice/
Disallow /*/contact/
Disallow /*/datenschutzerklaerung/
Disallow /*/privacy-policy/
Disallow /*/politique-de-confidentialite/
Disallow /*/informativa-sulla-privacy/
Disallow /*/privacyverklaring/
Disallow /*/polityka-prywatnosci/
Disallow /*/gizlilik-politikasi/
Disallow /*/lizenzbedingungen/
Disallow /*/license-terms/
Disallow /*/conditions-de-licence/
Disallow /*/condizioni-licenza/
Disallow /*/licentievoorwaarden/
Disallow /*/warunki-licencji/
Disallow /*/lisans-kosullari/

ahrefsbot
semrushbot
blexbot
dotbot
mj12bot
barkrowler
yandexbot
petalbot
sentibot
dataforseobot
megaindex
neticle
censysinspect
um-ic
researchbot
mojeekbot
seocompany

Rule Path
Disallow /

ai2bot
amazonbot
anthropic-ai
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
diffbot
duckassistbot
facebookbot
google-extended
gptbot
kangaroo bot
meta-externalagent
meta-externalfetcher
oai-searchbot
omgili
perplexitybot
timpibot
webzio-extended
youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://softcreatr.dev/sitemap.xml

Comments

  • 2024-11-10 by SoftCreatR.dev
  • sitemap
  • block internal directories
  • block user profiles
  • block registration and login
  • block post links
  • block search
  • block filtered / ordered pages
  • block utm
  • block legal pages and contact form
  • block crawler
  • AI