Log In Sign Up

ibt.com
robots.txt

Robots Exclusion Standard data for ibt.com

Resource Scan

Scan Details

Site Domain ibt.com
Base Domain ibt.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-02T22:25:15+00:00
Next Scan 2026-02-16T22:25:15+00:00

Last Successful Scan

Scanned2026-01-18T12:09:05+00:00
URL https://ibt.com/robots.txt
Redirect https://www.ibtimes.com/robots.txt
Redirect Domain www.ibtimes.com
Redirect Base ibtimes.com
Domain IPs 18.210.29.93, 3.213.234.254
Redirect IPs 44.218.35.136, 52.71.183.243
Response IP 44.218.35.136
Found Yes
Hash 4faff9e393a7aa5049c2c07ccf841ad255736dce0fa49aac394b2c9cb8c886a5
SimHash fd125148e7e4

Groups

*

Rule Path
Disallow /feeds/*
Disallow /system/
Disallow /rss/articles/specialcat/
Disallow /addineyeV2l
Disallow /eyeblaster
Disallow /doubleclick
Disallow /ads/sponsored
Disallow /comment/*
Disallow /search?q=*
Disallow /search/site*
Disallow /tool/logout*
Disallow /corporate/newsletter/signup?*

applebot-extended
aliyunsecbot
amazonbot
arquivo-web-crawler
awariorssbot
awariosmartbot
anthropic-ai
audigentadbot
blexbot
bytespider
buck
ccbot
coccocbot-web
cohere-ai
claudebot
claude-searchbot
claude-user
claude-web
dataforseobot
duckassistbot
facebookbot
imagesift
jenkersbot
jetslide
moodlebot
meta-externalagent
meta-externalagent
meta-externalfetcher
meta-externalfetcher
mycentralaiscraperbot
mojeek
news-please
oai-searchbot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
petalbot
perplexitybot
perplexity-user
poseidon research crawler
quillbot.com
quora-bot
scrapy
seekr
seekrbot
seznambot
seznamhomepagecrawler
scalepostai
sentione
taragroup intelligent bot
timpibot
turnitinbot
viennatinybot
yacy
yandexadditional
yandexadditionalbot
youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ibtimes.com/sitemap.xml
sitemap https://www.ibtimes.com/googlenews.xml
Back to top

Comments

  • This is the robots.txt file for ibtimes.com
  • The content on IBTimes is protected by our terms of service and copyright policies.
  • Unauthorized use of our content is prohibited. This includes, but is not limited to:
  • use by large language models (LLMs), machine learning systems, artificial intelligence
  • technologies, automated scraping, or any commercial exploitation without prior written consent.
  • For licensing or permission requests, please contact: licensing@ibt.com
Back to top