researchtrend.ai
robots.txt
Robots Exclusion Standard data for researchtrend.ai
Resource Scan
Scan Details
| Site Domain | researchtrend.ai |
| Base Domain | researchtrend.ai |
| Scan Status | Ok |
| Last Scan | 2026-02-06T04:56:18+00:00 |
| Next Scan | 2026-03-08T04:56:18+00:00 |
Last Scan
| Scanned | 2026-02-06T04:56:18+00:00 |
| URL | https://researchtrend.ai/robots.txt |
| Domain IPs | 13.35.238.24, 13.35.238.54, 13.35.238.67, 13.35.238.77, 2600:9000:2085:200:13:5450:ba40:93a1, 2600:9000:2085:5800:13:5450:ba40:93a1, 2600:9000:2085:6800:13:5450:ba40:93a1, 2600:9000:2085:8200:13:5450:ba40:93a1, 2600:9000:2085:ac00:13:5450:ba40:93a1, 2600:9000:2085:b200:13:5450:ba40:93a1, 2600:9000:2085:de00:13:5450:ba40:93a1, 2600:9000:2085:fc00:13:5450:ba40:93a1 |
| Response IP | 13.35.238.54 |
| Found | Yes |
| Hash | 984074f570d6182d2d98637e90aa62d04a079290b0caa6a18c0e042190ff48d7 |
| SimHash | 7018a945a295 |
Groups
googlebot
bingbot
slurp
yandexbot
baiduspider
yeti
duckduckbot
facebookexternalhit
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /blog |
| Allow | /authors/* |
| Allow | /blog-posts/* |
| Allow | /communities/* |
| Allow | /communities/*/blog-posts |
| Allow | /organizations/* |
| Allow | /papers |
| Allow | /papers/* |
| Allow | /social-events/* |
| Allow | /users/* |
| Allow | /terms-and-conditions |
| Allow | /pricing |
| Disallow | /auth/* |
| Disallow | /blog-posts/*/edit |
| Disallow | /blog-posts/create |
| Disallow | /only-author |
| Disallow | /organizations/suggest |
| Disallow | /profile |
| Disallow | /social-events/create |
| Disallow | /api/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gptbot
claudebot
claude-web
anthropic-ai
perplexitybot
meta-externalagent
facebookbot
bytespider
ccbot
google-extended
diffbot
cohere-ai
imagesiftbot
omgilibot
chatgpt-user
oai-searchbot
amazonbot
applebot-extended
| Rule | Path |
|---|---|
| Allow | /papers/* |
| Allow | /authors/* |
| Allow | /blog/* |
| Allow | /blog-posts/* |
| Allow | /communities/* |
| Allow | /organizations/* |
| Allow | / |
| Disallow | /auth/* |
| Disallow | /profile |
| Disallow | /social-events/create |
| Disallow | /blog-posts/create |
| Disallow | /blog-posts/*/edit |
| Disallow | /only-author |
| Disallow | /organizations/suggest |
| Disallow | /api/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /blog |
| Allow | /authors/* |
| Allow | /blog-posts/* |
| Allow | /communities/* |
| Allow | /communities/*/blog-posts |
| Allow | /organizations/* |
| Allow | /papers |
| Allow | /papers/* |
| Allow | /social-events/* |
| Allow | /users/* |
| Allow | /terms-and-conditions |
| Allow | /pricing |
| Disallow | /auth/* |
| Disallow | /blog-posts/*/edit |
| Disallow | /blog-posts/create |
| Disallow | /only-author |
| Disallow | /organizations/suggest |
| Disallow | /profile |
| Disallow | /social-events/create |
| Disallow | /api/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://researchtrend.ai/sitemap.xml |
| sitemap | https://researchtrend.ai/sitemap/paper.xml |
| sitemap | https://researchtrend.ai/sitemap/author.xml |
| sitemap | https://researchtrend.ai/sitemap/organization.xml |
| sitemap | https://researchtrend.ai/sitemap/community.xml |
| sitemap | https://researchtrend.ai/sitemap/blogPost.xml |
| sitemap | https://researchtrend.ai/sitemap/socialEvent.xml |