Log In Sign Up

arsenallatest.com
robots.txt

Robots Exclusion Standard data for arsenallatest.com

Resource Scan

Scan Details

Site Domain arsenallatest.com
Base Domain arsenallatest.com
Scan Status Ok
Last Scan2024-09-05T03:11:12+00:00
Next Scan 2024-10-05T03:11:12+00:00

Last Scan

Scanned2024-09-05T03:11:12+00:00
URL https://arsenallatest.com/robots.txt
Domain IPs 104.21.8.117, 172.67.130.112, 2606:4700:3030::6815:875, 2606:4700:3037::ac43:8270
Response IP 172.67.130.112
Found Yes
Hash d269a7aee22485f608ed785fd7a2c403458ec927bbf8b02b36798f1fa6ea1b50
SimHash 0c3486c29693

Groups

googlebot

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

*

Rule Path
Disallow /tmp/

twiceler

Rule Path
Disallow /

discobot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

yahoo pipes 1.0

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

spbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /
Back to top

Warnings

  • 2 invalid lines.
Back to top