theinternet.io
robots.txt
Robots Exclusion Standard data for theinternet.io
Resource Scan
Scan Details
| Site Domain | theinternet.io |
| Base Domain | theinternet.io |
| Scan Status | Ok |
| Last Scan | 2025-08-05T06:50:59+00:00 |
| Next Scan | 2025-08-12T06:50:59+00:00 |
Last Scan
| Scanned | 2025-08-05T06:50:59+00:00 |
| URL | https://theinternet.io/robots.txt |
| Redirect | https://www.theinternet.io/robots.txt |
| Redirect Domain | www.theinternet.io |
| Redirect Base | theinternet.io |
| Domain IPs | 34.233.163.247 |
| Redirect IPs | 35.173.69.207 |
| Response IP | 35.173.69.207 |
| Found | Yes |
| Hash | 7cfdf3efc38d2b0a6e0a2724ae91e74171f2e6ed522baab8173ca59d18b7ccf9 |
| SimHash | ad4059484b13 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /articles/writing-competitions/like-entry/ |
| Disallow | /articles/writing-competitions/vote-entry/ |
| Disallow | /articles/ask-ai/get-new-ai-answer/ |
| Disallow | /articles/ai-assistant/run-prompt/ |
| Disallow | /articles/ai-assistant/create-prompt-response/ |
| Disallow | /articles/ai-assistant/save-messages/ |
| Disallow | /articles/get-existing-comments/ |
| Disallow | /articles/post-article-comment/ |
| Disallow | /articles/ask-ai/read-all-notifications/ |
| Disallow | /articles/fetch_relevant_questions/ |
| Disallow | /articles/ask-ai/vote_for_question/ |
| Disallow | /articles/ask-ai/vote_for_tag/ |
| Disallow | /articles/ask-ai/vote_for_answer/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.theinternet.io/sitemap.xml |