forum.macmagazine.com.br
robots.txt

Robots Exclusion Standard data for forum.macmagazine.com.br

Resource Scan

Scan Details

Site Domain forum.macmagazine.com.br
Base Domain macmagazine.com.br
Scan Status Ok
Last Scan2024-11-11T16:47:20+00:00
Next Scan 2024-11-25T16:47:20+00:00

Last Scan

Scanned2024-11-11T16:47:20+00:00
URL https://forum.macmagazine.com.br/robots.txt
Domain IPs 104.26.0.126, 104.26.1.126, 172.67.74.114, 2606:4700:20::681a:17e, 2606:4700:20::681a:7e, 2606:4700:20::ac43:4a72
Response IP 172.67.74.114
Found Yes
Hash a2415d9d6e54b97c5e92d9399289a3185546d043a367a809fc3304ec2a70bd8a
SimHash e20a4a0217b3

Groups

*

Rule Path
Disallow */admin
Disallow */api
Disallow */cgi-bin
Disallow */oauth
Disallow */system
Disallow /*.php$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow */*.inc$
Disallow */*.gz$
Disallow */*.wmv$
Allow */applications
Allow */datastore
Allow */themes
Allow */uploads
Allow */*.js
Allow */*.css

duggmirror

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow
Allow /*

yandex
yandexblogs

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sogou

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10