Log In Sign Up

ranker-dev.com
robots.txt

Robots Exclusion Standard data for ranker-dev.com

Resource Scan

Scan Details

Site Domain ranker-dev.com
Base Domain ranker-dev.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-30T23:22:49+00:00
Next Scan 2024-11-29T23:22:49+00:00

Last Successful Scan

Scanned2024-09-08T22:53:35+00:00
URL https://ranker-dev.com/robots.txt
Redirect https://www.ranker-dev.com/robots.txt
Redirect Domain www.ranker-dev.com
Redirect Base ranker-dev.com
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 199.232.45.91
Found Yes
Hash 0657478a521ef6d23b78ce7a8ab4922f44c310ebaf2274655c2b22695a2912cf
SimHash f45b911b6775

Groups

*

Rule Path
Disallow /app
Disallow /commentary
Disallow /listcommentary
Disallow /extlink
Disallow /node
Disallow /suggestions
Disallow /list/rerank.htm
Disallow /widget
Disallow /pics
Disallow /review
Disallow /amp
Disallow /*US_CENSUS_NAME*
Allow /app/faq.htm

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ranker.com/sitemaps/list/sitemap-list-index.xml
sitemap https://www.ranker.com/sitemaps/user/sitemap-user-index.xml
sitemap https://www.ranker.com/sitemaps/google-news/sitemap-google-news-index.xml
sitemap https://www.ranker.com/sitemaps/collection/sitemap-collection-index.xml
Back to top

Comments

  • Ranker content is made available for your personal, non-commercial
  • use subject to our Terms of Service here: https://www.ranker.com/terms
  • Use of any device, tool, or process designed to data mine or scrape the
  • content using automated means is prohibited without prior written permission
  • from Ranker, Inc.
  • Prohibited uses include but are not limited to:
  • (1) text and data mining activities under Art. 4 of the EU Directive on
  • Copyright in the Digital Single Market;
  • (2) the development of any software, machine learning, artificial
  • intelligence (AI), and/or large language models (LLMs);
  • (3) creating or providing archived or cached data sets containing our
  • content to others; and/or
  • (4) any commercial purposes.
  • Contact https://business.ranker.com/contact for assistance.
  • Sitemap files
Back to top