webxicon.org
robots.txt

Robots Exclusion Standard data for webxicon.org

Archived Snapshots

Resource Scan

Scan Details

Site Domain	webxicon.org
Base Domain	webxicon.org
Scan Status	Ok
Last Scan	2025-08-31T13:11:53+00:00
Next Scan	2025-09-07T13:11:53+00:00

Last Scan

Scanned	2025-08-31T13:11:53+00:00
URL	https://webxicon.org/robots.txt
Domain IPs	104.21.46.149, 172.67.140.53, 2606:4700:3031::ac43:8c35, 2606:4700:3033::6815:2e95
Response IP	104.21.46.149
Found	Yes
Hash	0c474736d7e7e02c26ed5145c4170aba6384d19ee312365f2fd9977ffc891b24
SimHash	a1d759c2c5f5

Groups

*

Rule	Path
Disallow	/search.php
Disallow	/add_translations.php
Disallow	/switchlanguage.php
Disallow	/cat.php

Rule

Path

Disallow

/search.php

Disallow

/add_translations.php

Disallow

/switchlanguage.php

Disallow

/cat.php

twiceler

Rule	Path
Disallow	/

Rule

Path

Disallow

linkwalker

Rule	Path
Disallow	/

Rule

Path

Disallow

msiecrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver

Rule	Path
Disallow	/

Rule

Path

Disallow

inktomi slurp

Rule	Path
Disallow	/

Rule

Path

Disallow

ibm_planetwide

Rule	Path
Disallow	/

Rule

Path

Disallow

gazz

Rule	Path
Disallow	/

Rule

Path

Disallow

ubicrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

doc

Rule	Path
Disallow	/

Rule

Path

Disallow

zao

Rule	Path
Disallow	/

Rule

Path

Disallow

sitecheck.internetseer.com

Rule	Path
Disallow	/

Rule

Path

Disallow

zealbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sitesnagger

Rule	Path
Disallow	/

Rule

Path

Disallow

webstripper

Rule	Path
Disallow	/

Rule

Path

Disallow

webcopier

Rule	Path
Disallow	/

Rule

Path

Disallow

fetch

Rule	Path
Disallow	/

Rule

Path

Disallow

offline explorer

Rule	Path
Disallow	/

Rule

Path

Disallow

teleport

Rule	Path
Disallow	/

Rule

Path

Disallow

teleportpro

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip

Rule	Path
Disallow	/

Rule

Path

Disallow

linko

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack

Rule	Path
Disallow	/

Rule

Path

Disallow

microsoft.url.control

Rule	Path
Disallow	/

Rule

Path

Disallow

xenu

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin

Rule	Path
Disallow	/

Rule

Path

Disallow

libwww

Rule	Path
Disallow	/

Rule

Path

Disallow

zyborg

Rule	Path
Disallow	/

Rule

Path

Disallow

download ninja

Rule	Path
Disallow	/

Rule

Path

Disallow

grub-client

Rule	Path
Disallow	/

Rule

Path

Disallow

grub

Rule	Path
Disallow	/

Rule

Path

Disallow

slurp

Rule	Path
Disallow	/

Rule

Path

Disallow

mediapartners-google

Rule	Path
Allow	/

Rule

Path

Allow

googlebot

Rule	Path
Allow	/
Disallow	/add_translations.php
Disallow	/switchlanguage.php
Disallow	/links.php

Rule

Path

Allow

Disallow

/add_translations.php

Disallow

/switchlanguage.php

Disallow

/links.php

msnbot

Rule	Path
Allow	/
Disallow	/add_translations.php
Disallow	/switchlanguage.php
Disallow	/links.php

Rule

Path

Allow

Disallow

/add_translations.php

Disallow

/switchlanguage.php

Disallow

/links.php

bingbot

Rule	Path
Allow	/
Disallow	/add_translations.php
Disallow	/switchlanguage.php
Disallow	/links.php

Rule

Path

Allow

Disallow

/add_translations.php

Disallow

/switchlanguage.php

Disallow

/links.php

yandex

Rule	Path
Allow	/
Disallow	/add_translations.php
Disallow	/switchlanguage.php
Disallow	/links.php

Rule

Path

Allow

Disallow

/add_translations.php

Disallow

/switchlanguage.php

Disallow

/links.php

mediapartners-google

Rule	Path
Disallow

Rule

Path

Disallow

webxicon.orgrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

twiceler

linkwalker

msiecrawler

ia_archiver

inktomi slurp

ibm_planetwide

gazz

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

grub-client

grub

slurp

mediapartners-google

googlebot

msnbot

bingbot

yandex

mediapartners-google

webxicon.org
robots.txt