ios-api-2.duolingo.com
robots.txt

Robots Exclusion Standard data for ios-api-2.duolingo.com

Resource Scan

Scan Details

Site Domain ios-api-2.duolingo.com
Base Domain duolingo.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-22T04:23:42+00:00
Next Scan 2024-10-20T04:23:42+00:00

Last Successful Scan

Scanned2023-01-12T12:47:24+00:00
URL https://ios-api-2.duolingo.com/robots.txt
Domain IPs 100.24.89.241, 23.22.171.190, 3.224.58.229, 44.205.183.51, 52.2.118.205, 52.4.17.131, 52.55.4.20, 54.197.36.57
Response IP 52.205.25.193
Found Yes
Hash 86810a106756ff3d9dd33e8dadef446776fabba6db24be20208474340267b273
SimHash 3c302d3aa1f1

Groups

*

Rule Path
Disallow /sessions/
Disallow /extend_session/
Disallow /session_element_solutions/
Disallow /friendships/
Disallow /vocabularies/
Disallow /diagnostics/
Disallow /skills/
Disallow /translations/
Disallow /translation/
Disallow /contribute/
Disallow /documents/
Disallow /matchmaker/
Disallow /logger/
Disallow /events/
Disallow /facebook/
Disallow /twitter/
Disallow /words/
Disallow /translate_jobs/
Disallow /oauth/
Disallow /data/
Disallow /unsubscribe
Disallow /deactivate
Disallow /pm/
Disallow /profile/

twitterbot

Rule Path
Allow /profile/

facebookexternalhit

Rule Path
Allow /profile/

Other Records

Field Value
sitemap https://ios-api-2.duolingo.com/sitemap_index.xml

Comments

  • Certain social media sites are added to allow crawlers to access page markup when links to /profile are shared.