mtenoticia.com.br
robots.txt

Robots Exclusion Standard data for mtenoticia.com.br

Resource Scan

Scan Details

Site Domain mtenoticia.com.br
Base Domain mtenoticia.com.br
Scan Status Ok
Last Scan2024-09-25T11:07:16+00:00
Next Scan 2024-10-02T11:07:16+00:00

Last Scan

Scanned2024-09-25T11:07:16+00:00
URL https://mtenoticia.com.br/robots.txt
Domain IPs 104.21.37.72, 172.67.205.75, 2606:4700:3033::6815:2548, 2606:4700:3037::ac43:cd4b
Response IP 172.67.205.75
Found Yes
Hash 37fe8de309ab0b22c70ca430f57e6717adbcc3f06912a259aeaf78ffba8b21aa
SimHash 6854d200c5a3

Groups

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

naverbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

seznambot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

teoma

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yeti

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

adsbot-google

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookcatalog

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /

Other Records

Field Value
sitemap https://mtenoticia.com.br/sitemap_index.xml