www1.folha.uol.com.br
robots.txt
Robots Exclusion Standard data for www1.folha.uol.com.br
Resource Scan
Scan Details
| Site Domain | www1.folha.uol.com.br |
| Base Domain | uol.com.br |
| Scan Status | Ok |
| Last Scan | 2024-11-06T07:55:37+00:00 |
| Next Scan | 2024-11-20T07:55:37+00:00 |
Last Scan
| Scanned | 2024-11-06T07:55:37+00:00 |
| URL | https://www1.folha.uol.com.br/robots.txt |
| Domain IPs | 13.226.61.34, 13.226.61.35, 13.226.61.91, 13.226.61.94 |
| Response IP | 13.35.210.25 |
| Found | Yes |
| Hash | 516cc1d1ec57c6cbe91eaddf2ed3f86da9ad0b8a69258ea5988938f0f1c2066c |
| SimHash | f11e01510480 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /folha/furniture/ |
| Disallow | /folha/virtual/ |
| Disallow | /virtual/ |
| Allow | /virtual/hunting/podcasts/google-assistant/ |
| Disallow | /folha/backend/ |
amazonbot
anthropic-ai
applebot-extended
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
dataforseobot
diffbot
facebookbot
google-extended
gptbot
imagesiftbot
magpie-crawler
meta-externalagent
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
timpibot
webzio-extended
youbot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www1.folha.uol.com.br/news-sitemap.xml |
| sitemap | https://www1.folha.uol.com.br/folha-online-sitemap.xml |
| sitemap | https://www1.folha.uol.com.br/sitemap.xml |
| sitemap | https://www1.folha.uol.com.br/ondemorar/sitemap.xml |
Comments