Log In Sign Up

dudilink.com
robots.txt

Robots Exclusion Standard data for dudilink.com

Resource Scan

Scan Details

Site Domain dudilink.com
Base Domain dudilink.com
Scan Status Ok
Last Scan2025-11-23T23:17:59+00:00
Next Scan 2025-11-30T23:17:59+00:00

Last Scan

Scanned2025-11-23T23:17:59+00:00
URL https://dudilink.com/robots.txt
Domain IPs 104.21.2.67, 172.67.128.218, 2606:4700:3032::6815:243, 2606:4700:3037::ac43:80da
Response IP 104.21.2.67
Found Yes
Hash 796a7d8672834728416e6e994b11bb6274344751c644fec3f50a56e44d9890e0
SimHash 7e3a98414572

Groups

*

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

zgrab

Rule Path
Disallow /
Back to top

Comments

  • http://www.robotstxt.org/robotstxt.html
  • Dark Visitors robots.txt
  • https://darkvisitors.com/
  • - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  • AI Assistant
  • https://darkvisitors.com/agents/chatgpt-user
  • Website: https://openai.com/bot
  • AI Assistant
  • https://darkvisitors.com/agents/cohere-ai
  • Website: https://cohere.com/
  • AI Data Scraper
  • https://darkvisitors.com/agents/anthropic-ai
  • Website: https://anthropic.com/
  • AI Data Scraper
  • https://darkvisitors.com/agents/bytespider
  • Website: https://bytedance.com/
  • AI Data Scraper
  • https://darkvisitors.com/agents/ccbot
  • Website: https://commoncrawl.org/faq
  • AI Data Scraper
  • https://darkvisitors.com/agents/facebookbot
  • Website: https://developers.facebook.com/docs/sharing/bot/
  • AI Data Scraper
  • https://darkvisitors.com/agents/google-extended
  • Website: http://google.com/bot.html
  • AI Data Scraper
  • https://darkvisitors.com/agents/gptbot
  • Website: https://openai.com/gptbot
  • AI Data Scraper
  • https://darkvisitors.com/agents/omgilibot
  • AI Search Crawler
  • https://darkvisitors.com/agents/amazonbot
  • Website: https://developer.amazon.com/support/amazonbot
  • AI Search Crawler
  • https://darkvisitors.com/agents/applebot
  • Website: http://apple.com/go/applebot
  • AI Search Crawler
  • https://darkvisitors.com/agents/perplexitybot
  • Website: https://docs.perplexity.ai/docs/perplexitybot
  • AI Search Crawler
  • https://darkvisitors.com/agents/youbot
  • Website: https://about.you.com/youbot/
  • - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  • This blocks Go-http-client
  • This blocks zgrab
Back to top