Log In Sign Up

maxigadget.com
robots.txt

Robots Exclusion Standard data for maxigadget.com

Resource Scan

Scan Details

Site Domain maxigadget.com
Base Domain maxigadget.com
Scan Status Ok
Last Scan2024-11-10T19:38:51+00:00
Next Scan 2024-11-17T19:38:51+00:00

Last Scan

Scanned2024-11-10T19:38:51+00:00
URL https://maxigadget.com/robots.txt
Domain IPs 91.134.190.184
Response IP 91.134.190.184
Found Yes
Hash c63a2b23212cda2feace8030da4f750fa06966462976d5b6e2ce76ac41ad8bbf
SimHash 634841400e9a

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /go/
Disallow /archives/
Disallow /wp-*
Disallow /author
Disallow /comments/feed/
Disallow /plugins/feedback.php
Allow /*.js$
Allow /*.css$

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

y!j-asr

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

orangebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

meltawer

Rule Path
Disallow /

pulp/2.5.3

Rule Path
Disallow /

iceweasel

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

libwww

Rule Path
Disallow /

wget

Rule Path
Disallow /

moreover

Rule Path
Disallow /

crystalsemantics

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

semantissimo

Rule Path
Disallow /

netseer

Rule Path
Disallow /

ltbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

screaming frog

Rule Path
Disallow /

netseer

Rule Path
Disallow /

netseer.com

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

ning/1.0

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

genieo.com

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

wasalive

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

openoffice

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

metauri

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

inoreader.com-like

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

rssing.com

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

maxpointinteractive

Rule Path
Disallow /

maxpoint

Rule Path
Disallow /

tiscali.it

Rule Path
Disallow /

nuzzel

Rule Path
Disallow /

photon/1.0

Rule Path
Disallow /

livelapbot/0.2

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

j12bot/v1.4.5

Rule Path
Disallow /

chilkat/1.0.0

Rule Path
Disallow /

*

Rule Path
Disallow /*blackhole
Disallow /?blackhole
Back to top

Other Records

Field Value
sitemap https://www.maxigadget.com/sitemap_index.xml
Back to top

Comments

  • disallow all files in these directories
  • slow down Search Engine
  • Robots exclus de toute indexation.
Back to top