afnoticias.com.br
robots.txt

Robots Exclusion Standard data for afnoticias.com.br

Resource Scan

Scan Details

Site Domain afnoticias.com.br
Base Domain afnoticias.com.br
Scan Status Ok
Last Scan2024-06-15T21:28:35+00:00
Next Scan 2024-06-22T21:28:35+00:00

Last Scan

Scanned2024-06-15T21:28:35+00:00
URL https://afnoticias.com.br/robots.txt
Domain IPs 96.126.108.196
Response IP 96.126.108.196
Found Yes
Hash f8f818b7876a8079854291240a4480581887964285f6ce3bfb576b9038388677
SimHash ea004ac66793

Groups

*

Rule Path
Disallow /banner
Allow /

proximic

Rule Path
Disallow /

starkbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

voluumdsp-content-bot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

*

Rule Path
Disallow /busca?query=*

Other Records

Field Value
sitemap https://afnoticias.com.br/sitemap.xml

Warnings

  • 1 invalid line.