Log In Sign Up

intechopen.com
robots.txt

Robots Exclusion Standard data for intechopen.com

Resource Scan

Scan Details

Site Domain intechopen.com
Base Domain intechopen.com
Scan Status Ok
Last Scan2024-11-13T13:31:46+00:00
Next Scan 2024-11-20T13:31:46+00:00

Last Scan

Scanned2024-11-13T13:31:46+00:00
URL https://intechopen.com/robots.txt
Redirect https://www.intechopen.com:443/robots.txt
Redirect Domain www.intechopen.com
Redirect Base intechopen.com
Domain IPs 2600:1f18:256f:5300:fd2e:67bb:9076:2b12, 2600:1f18:256f:5310:3245:29ce:a928:3f13, 34.204.153.147, 54.157.237.137
Redirect IPs 2600:1f18:256f:53c0:b444:6449:d5df:5774, 35.171.73.43
Response IP 35.171.73.43
Found Yes
Hash ee394c8f22a19a50643ad5536687d36e081cf70d5090629ef2863f71a1e45e25
SimHash 611c9251c510

Groups

*

Rule Path
Allow /
Disallow /_preview

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

omgili

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.intechopen.com/sitemap-v2/sitemap.xml
Back to top

Comments

  • Block GPTBot (AI Search Crawler)
  • Block GPTBot (AI Data Scraper)
  • Block CCBot (AI Search Crawler)
  • Block CCBot (AI Data Scraper)
  • Block Google-Extended (AI Search Crawler)
  • Block Google-Extended (AI Data Scraper)
  • Block Amazonbot (AI Search Crawler)
  • Block Applebot (AI Search Crawler)
  • Block OAI-SearchBot (AI Search Crawler)
  • Block PerplexityBot (AI Search Crawler)
  • Block YouBot (AI Search Crawler)
  • Block ClaudeBot (AI Data Scraper)
  • Block Bytespider (AI Data Scraper)
  • Block Diffbot (AI Data Scraper)
  • Block FacebookBot (AI Data Scraper)
  • Block Meta-ExternalAgent (AI Data Scraper)
  • Block omgili (AI Data Scraper)
Back to top