Log In Sign Up

cdnintech.com
robots.txt

Robots Exclusion Standard data for cdnintech.com

Resource Scan

Scan Details

Site Domain cdnintech.com
Base Domain cdnintech.com
Scan Status Ok
Last Scan2024-10-26T14:33:24+00:00
Next Scan 2024-11-25T14:33:24+00:00

Last Scan

Scanned2024-10-26T14:33:24+00:00
URL https://cdnintech.com/robots.txt
Domain IPs 18.155.68.11, 18.155.68.111, 18.155.68.8, 18.155.68.96, 2600:9000:23d2:200:1f:3df1:7bc0:93a1, 2600:9000:23d2:4800:1f:3df1:7bc0:93a1, 2600:9000:23d2:7c00:1f:3df1:7bc0:93a1, 2600:9000:23d2:9600:1f:3df1:7bc0:93a1, 2600:9000:23d2:9e00:1f:3df1:7bc0:93a1, 2600:9000:23d2:a00:1f:3df1:7bc0:93a1, 2600:9000:23d2:a800:1f:3df1:7bc0:93a1, 2600:9000:23d2:ec00:1f:3df1:7bc0:93a1
Response IP 18.155.68.8
Found Yes
Hash fd1ed505abb171f5a048ddd66e9aac6c9ab131c2e27b48ac02fb2919bd11224e
SimHash 701eb250c552

Groups

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

omgili

Rule Path
Disallow /
Back to top

Comments

  • Block GPTBot (AI Search Crawler)
  • Block GPTBot (AI Data Scraper)
  • Block CCBot (AI Search Crawler)
  • Block CCBot (AI Data Scraper)
  • Block Google-Extended (AI Search Crawler)
  • Block Google-Extended (AI Data Scraper)
  • Block Amazonbot (AI Search Crawler)
  • Block Applebot (AI Search Crawler)
  • Block OAI-SearchBot (AI Search Crawler)
  • Block PerplexityBot (AI Search Crawler)
  • Block YouBot (AI Search Crawler)
  • Block ClaudeBot (AI Data Scraper)
  • Block Bytespider (AI Data Scraper)
  • Block Diffbot (AI Data Scraper)
  • Block FacebookBot (AI Data Scraper)
  • Block Meta-ExternalAgent (AI Data Scraper)
  • Block omgili (AI Data Scraper)
Back to top