Log In Sign Up

computergaga.com
robots.txt

Robots Exclusion Standard data for computergaga.com

Resource Scan

Scan Details

Site Domain computergaga.com
Base Domain computergaga.com
Scan Status Ok
Last Scan2024-10-05T05:49:42+00:00
Next Scan 2024-10-12T05:49:42+00:00

Last Scan

Scanned2024-10-05T05:49:42+00:00
URL https://computergaga.com/robots.txt
Domain IPs 104.21.41.135, 172.67.147.127, 2606:4700:3031::6815:2987, 2606:4700:3031::ac43:937f
Response IP 104.21.41.135
Found Yes
Hash f7088fbdaad95c2d7b8f3eb36f3964199151f3946ba532e5baadc65b5beff633
SimHash a2567192cca3

Groups

*

Rule Path
Disallow /cgi-bin/

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

python-urllib/1.17

Rule Path
Disallow /

boardtracker

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

discobot

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

voyager

Rule Path
Disallow /

kscrawler

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

purebot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

antbot

Rule Path
Disallow /

bender

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

mozilla/5.0 (compatible; bender; http://sites.google.com/site/bendercrawler)

Rule Path
Disallow /

mozilla/5.0 (compatible; bender; http://benderthewebrobot.tumblr.com)

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.computergaga.com/sitemap.xml
Back to top

Warnings

  • 2 invalid lines.
Back to top