android-api.duolingo.com
robots.txt

Robots Exclusion Standard data for android-api.duolingo.com

Resource Scan

Scan Details

Site Domain android-api.duolingo.com
Base Domain duolingo.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-05-01T07:40:54+00:00
Next Scan 2025-07-30T07:40:54+00:00

Last Successful Scan

Scanned2023-01-12T11:20:13+00:00
URL https://android-api.duolingo.com/robots.txt
Domain IPs 100.24.89.241, 18.214.91.42, 34.193.62.144, 44.198.33.62, 44.205.183.51, 54.172.65.85, 54.173.177.237, 54.210.52.226
Response IP 54.147.159.108
Found Yes
Hash b290b818d87974d446fb79860879e4231e94346a5846d2d05ff4efef497425ca
SimHash 3c102d3bb1f1

Groups

*

Rule Path
Disallow /sessions/
Disallow /extend_session/
Disallow /session_element_solutions/
Disallow /friendships/
Disallow /vocabularies/
Disallow /diagnostics/
Disallow /skills/
Disallow /translations/
Disallow /translation/
Disallow /contribute/
Disallow /documents/
Disallow /matchmaker/
Disallow /logger/
Disallow /events/
Disallow /facebook/
Disallow /twitter/
Disallow /words/
Disallow /translate_jobs/
Disallow /oauth/
Disallow /data/
Disallow /unsubscribe
Disallow /deactivate
Disallow /pm/
Disallow /profile/

twitterbot

Rule Path
Allow /profile/

facebookexternalhit

Rule Path
Allow /profile/

Other Records

Field Value
sitemap https://android-api.duolingo.com/sitemap_index.xml

Comments

  • Certain social media sites are added to allow crawlers to access page markup when links to /profile are shared.