Log In Sign Up

myibtimes.com
robots.txt

Robots Exclusion Standard data for myibtimes.com

Resource Scan

Scan Details

Site Domain myibtimes.com
Base Domain myibtimes.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-20T17:59:15+00:00
Next Scan 2026-03-22T17:59:15+00:00

Last Successful Scan

Scanned2026-01-22T17:16:29+00:00
URL https://myibtimes.com/robots.txt
Redirect https://www.ibtimes.com/robots.txt
Redirect Domain www.ibtimes.com
Redirect Base ibtimes.com
Domain IPs 34.218.63.216, 52.38.65.34
Redirect IPs 3.218.130.139, 98.85.24.227
Response IP 98.85.24.227
Found Yes
Hash 4faff9e393a7aa5049c2c07ccf841ad255736dce0fa49aac394b2c9cb8c886a5
SimHash fd125148e7e4

Groups

*

Rule Path
Disallow /feeds/*
Disallow /system/
Disallow /rss/articles/specialcat/
Disallow /addineyeV2l
Disallow /eyeblaster
Disallow /doubleclick
Disallow /ads/sponsored
Disallow /comment/*
Disallow /search?q=*
Disallow /search/site*
Disallow /tool/logout*
Disallow /corporate/newsletter/signup?*

applebot-extended
aliyunsecbot
amazonbot
arquivo-web-crawler
awariorssbot
awariosmartbot
anthropic-ai
audigentadbot
blexbot
bytespider
buck
ccbot
coccocbot-web
cohere-ai
claudebot
claude-searchbot
claude-user
claude-web
dataforseobot
duckassistbot
facebookbot
imagesift
jenkersbot
jetslide
moodlebot
meta-externalagent
meta-externalagent
meta-externalfetcher
meta-externalfetcher
mycentralaiscraperbot
mojeek
news-please
oai-searchbot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
petalbot
perplexitybot
perplexity-user
poseidon research crawler
quillbot.com
quora-bot
scrapy
seekr
seekrbot
seznambot
seznamhomepagecrawler
scalepostai
sentione
taragroup intelligent bot
timpibot
turnitinbot
viennatinybot
yacy
yandexadditional
yandexadditionalbot
youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ibtimes.com/sitemap.xml
sitemap https://www.ibtimes.com/googlenews.xml
Back to top

Comments

  • This is the robots.txt file for ibtimes.com
  • The content on IBTimes is protected by our terms of service and copyright policies.
  • Unauthorized use of our content is prohibited. This includes, but is not limited to:
  • use by large language models (LLMs), machine learning systems, artificial intelligence
  • technologies, automated scraping, or any commercial exploitation without prior written consent.
  • For licensing or permission requests, please contact: licensing@ibt.com
Back to top