Log In Sign Up

developmentscout.com
robots.txt

Robots Exclusion Standard data for developmentscout.com

Resource Scan

Scan Details

Site Domain developmentscout.com
Base Domain developmentscout.com
Scan Status Ok
Last Scan2025-12-17T14:19:08+00:00
Next Scan 2025-12-24T14:19:08+00:00

Last Scan

Scanned2025-12-17T14:19:08+00:00
URL https://www.developmentscout.com/robots.txt
Domain IPs 104.26.14.79, 104.26.15.79, 172.67.74.78, 2606:4700:20::681a:e4f, 2606:4700:20::681a:f4f, 2606:4700:20::ac43:4a4e
Response IP 172.67.74.78
Found Yes
Hash de1f442d0cf5aa2bad4ec83c4b5a6257239874711fe7a993eb76a48c78996cd2
SimHash 66b16a13cd55

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

*

Rule Path
Disallow /installation_old
Disallow /?tmpl=component
Disallow /component/content/*
Disallow /*?utm_source=*
Disallow /*?utm_medium=*
Disallow /*?utm_campaign=*
Disallow /*?acm=*
Disallow /*?idU=*
Disallow /*?layout=*
Disallow /*?amp%3Bview=*
Disallow /*?missing_content=*
Disallow /*?redirected=*
Disallow /*?start=*
Disallow /media/com_jchoptimize/
Disallow /component/acym/*
Disallow /components/com_adagency/*
Allow /component/adagency/*.*
Disallow /component/adagency/
Disallow /images/resized/*
Disallow /images/stories/slider/cache/*
Disallow /images/stories/ad_agency/*
Disallow /images/jch_optimize_backup_images/*
Disallow /cache/
Disallow /media/jui/js/
Disallow /media/gantry5/
Disallow /?subject=
Disallow /piwik/
Disallow /media/smartslider3/
Disallow /plugins/system/itcs_scroll2top/
Disallow /?option=dbz_bdlz&ctrl=*
Disallow /index.php?option=dbz_bdlz&ctrl=*
Disallow /newsletter/frontusers/
Disallow /plugins/content/jw_allvideos/
Disallow /components/com_roksprocket/
Disallow /?format=feed&type=rss
Disallow /*?format=feed&type=rss
Disallow /?format=feed&type=atom
Disallow /*?format=feed&type=atom
Allow /ads/preferences/
Allow /dtt/k
Allow /gpt/
Allow /pagead/show_ads.js
Allow /pagead/js/adsbygoogle.js
Allow /pagead/js/*/show_ads_impl.js
Allow /static/glade.js
Allow /static/ad_status.js
Allow /static/glade/
Allow /tag/js/

adsbot-google

Rule Path
Allow /maps/api/js
Allow /maps/api/place/js/
Allow /maps/api/staticmap
Allow /maps/api/streetview

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

yandex

Rule Path
Disallow /

youbot

Rule Path
Disallow /

deepseekbot

Rule Path
Disallow /

geminibot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

oasst

Rule Path
Disallow /

quora link preview

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

applebot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.developmentscout.com/index.php?option=com_jmap&view=sitemap&format=xml
sitemap https://www.developmentscout.com/index.php?option=com_jmap&view=sitemap&format=images
sitemap https://www.developmentscout.com/index.php?option=com_jmap&view=sitemap&format=videos
Back to top

Comments

  • As a condition of accessing this website, you agree to abide by the following
  • content signals:
  • (a) If a content-signal = yes, you may collect content for the corresponding
  • use.
  • (b) If a content-signal = no, you may not collect content for the
  • corresponding use.
  • (c) If the website operator does not include a content signal for a
  • corresponding use, the website operator neither grants nor restricts
  • permission via content signal with respect to the corresponding use.
  • The content signals and their meanings are:
  • search: building a search index and providing search results (e.g., returning
  • hyperlinks and short excerpts from your website's contents). Search does not
  • include providing AI-generated search summaries.
  • ai-input: inputting content into one or more AI models (e.g., retrieval
  • augmented generation, grounding, or other real-time taking of content for
  • generative AI search answers).
  • ai-train: training or fine-tuning AI models.
  • ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
  • RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
  • AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
  • BEGIN Cloudflare Managed content
  • END Cloudflare Managed Content
  • Joomla Parameter and JCH-Optimize exclusion
  • Bekannte KI-Crawler blockieren
  • JSitemap entries
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top