Log In Sign Up

universeguide.com
robots.txt

Robots Exclusion Standard data for universeguide.com

Resource Scan

Scan Details

Site Domain universeguide.com
Base Domain universeguide.com
Scan Status Ok
Last Scan2024-09-28T23:58:39+00:00
Next Scan 2024-10-05T23:58:39+00:00

Last Scan

Scanned2024-09-28T23:58:39+00:00
URL https://universeguide.com/robots.txt
Redirect https://www.universeguide.com/robots.txt
Redirect Domain www.universeguide.com
Redirect Base universeguide.com
Domain IPs 2406:da18:9d0:143f:2124:4e9c:36a9:d9de, 52.221.42.138
Redirect IPs 104.21.18.97, 172.67.181.142, 2606:4700:3031::ac43:b58e, 2606:4700:3037::6815:1261
Response IP 172.67.181.142
Found Yes
Hash fdcf9fa161ef3ddf52798fbc070d97ac6c1470637fbf0c0d41590a2032695f9e
SimHash da5d5070c7e0

Groups

barkrowler/0.9

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

*

Rule Path
Disallow /members/*
Disallow /*/postcomment
Disallow /fetch/getsearch

npbot

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

*

Rule Path
Disallow *.axd$

*

Rule Path
Disallow /WebResource.axd

blexbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

serpstatbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow *

weborama

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

linespider

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.universeguide.com/sitemaps/ugaliens.xml
sitemap https://www.universeguide.com/sitemaps/ugasteroids.xml
sitemap https://www.universeguide.com/sitemaps/ugblogarticles.xml
sitemap https://www.universeguide.com/sitemaps/ugcharacters.xml
sitemap https://www.universeguide.com/sitemaps/ugcluster.xml
sitemap https://www.universeguide.com/sitemaps/ugcomet.xml
sitemap https://www.universeguide.com/sitemaps/ugconstellation.xml
sitemap https://www.universeguide.com/sitemaps/ugdeepspace.xml
sitemap https://www.universeguide.com/sitemaps/ugentertainment.xml
sitemap https://www.universeguide.com/sitemaps/ugepisode.xml
sitemap https://www.universeguide.com/sitemaps/ugexoplanets.xml
sitemap https://www.universeguide.com/sitemaps/ugextrasolar.xml
sitemap https://www.universeguide.com/sitemaps/ugfacts.xml
sitemap https://www.universeguide.com/sitemaps/ugfilms.xml
sitemap https://www.universeguide.com/sitemaps/uggalaxy.xml
sitemap https://www.universeguide.com/sitemaps/ugkuiperbelt.xml
sitemap https://www.universeguide.com/sitemaps/ugmain.xml
sitemap https://www.universeguide.com/sitemaps/ugmeteorshowerlist.xml
sitemap https://www.universeguide.com/sitemaps/ugmeteorshowers.xml
sitemap https://www.universeguide.com/sitemaps/ugmissions.xml
sitemap https://www.universeguide.com/sitemaps/ugnebula.xml
sitemap https://www.universeguide.com/sitemaps/ugobject.xml
sitemap https://www.universeguide.com/sitemaps/ugorganisations.xml
sitemap https://www.universeguide.com/sitemaps/ugpeople.xml
sitemap https://www.universeguide.com/sitemaps/ugplanetmoon.xml
sitemap https://www.universeguide.com/sitemaps/ugplanets.xml
sitemap https://www.universeguide.com/sitemaps/ugportal.xml
sitemap https://www.universeguide.com/sitemaps/ugquasar.xml
sitemap https://www.universeguide.com/sitemaps/ugrealplanets.xml
sitemap https://www.universeguide.com/sitemaps/ugreview.xml
sitemap https://www.universeguide.com/sitemaps/ugspaceship.xml
sitemap https://www.universeguide.com/sitemaps/ugstarlist.xml
sitemap https://www.universeguide.com/sitemaps/ugstars.xml
sitemap https://www.universeguide.com/sitemaps/ugstars1.xml
sitemap https://www.universeguide.com/sitemaps/ugsupernova.xml
sitemap https://www.universeguide.com/sitemaps/ugtaglist.xml
Back to top

Comments

  • https://megaindex.com/crawler
Back to top