Log In Sign Up

datacadamia.com
robots.txt

Robots Exclusion Standard data for datacadamia.com

Resource Scan

Scan Details

Site Domain datacadamia.com
Base Domain datacadamia.com
Scan Status Ok
Last Scan2024-11-16T21:23:41+00:00
Next Scan 2024-11-23T21:23:41+00:00

Last Scan

Scanned2024-11-16T21:23:41+00:00
URL https://datacadamia.com/robots.txt
Domain IPs 104.21.30.14, 172.67.150.50, 2606:4700:3030::ac43:9632, 2606:4700:3035::6815:1e0e
Response IP 172.67.150.50
Found Yes
Hash cfc8362ba6a1a35b04e2e7156093230ab13c98e98a9160580536cd1a5da5bd07
SimHash 48345412ccd2

Groups

ahrefsbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

*

Rule Path
Disallow /*?*do=search*

*

Rule Path
Disallow /*?*do=diff*

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://datacadamia.com/doku.php?do=sitemap
Back to top

Comments

  • https://ahrefs.com/robot (too much email spam from them)
  • Seo: Moz
  • Seo Moz https://moz.com/help/moz-procedures/crawlers/dotbot
  • Seo: https://www.deepcrawl.com/
  • Chinese search engine spider so.360.cn
  • http://so.360.cn/index.htm
  • User-agent: 360Spider
  • Google: https://support.google.com/webmasters/answer/1061943?hl=en
  • Checks Android web page ad quality
  • User-agent: AdsBot-Google
  • User-agent: AdsBot-Google-Mobile
  • https://support.google.com/webmasters/answer/1061943?hl=en
  • User-agent: APIs-Google
  • User-agent: Applebot
  • User-agent: Googlebot
  • User-agent: Googlebot-Image
  • User-agent: Googlebot-Mobile
  • User-agent: Googlebot-News
  • User-agent: Googlebot-Video
  • Ad Sense: https://support.google.com/webmasters/answer/1061943
  • User-agent: Mediapartners-Google
  • User-agent: baiduspider
  • User-agent: Bingbot
  • User-agent: DuckDuckBot
  • User-agent: ia_archiver
  • https://developers.facebook.com/docs/sharing/webmasters/crawler
  • User-agent: facebookexternalhit/1.1
  • User-agent: Lycos
  • User-agent: msnbot
  • User-agent: msnbot-media
  • User-agent: OrangeBot
  • User-agent: OrangeBot-Collector
  • Yahoo
  • User-agent: Slurp
  • Search engine
  • https://en.wikipedia.org/wiki/Sogou
  • User-agent: Sogou
  • Russe
  • User-agent: StackRambler
  • https://en.wikipedia.org/wiki/Teoma
  • User-agent: teoma
  • A feed fetcher that retrieves details associated with external links displayed on Twitter.
  • https://developer.twitter.com/en/docs/tweets/optimize-with-cards/guides/getting-started#url-crawling-and-caching
  • User-agent: Twitterbot
  • User-agent: Yandex
  • User-agent: Mail.RU_Bot
  • A new page every 2 seconds
Back to top