edugeek.net
robots.txt

Robots Exclusion Standard data for edugeek.net

Resource Scan

Scan Details

Site Domain edugeek.net
Base Domain edugeek.net
Scan Status Ok
Last Scan2024-09-10T02:29:41+00:00
Next Scan 2024-10-10T02:29:41+00:00

Last Scan

Scanned2024-09-10T02:29:41+00:00
URL https://edugeek.net/robots.txt
Redirect https://www.edugeek.net/robots.txt
Redirect Domain www.edugeek.net
Redirect Base edugeek.net
Domain IPs 172.66.40.233, 172.66.43.23
Redirect IPs 172.66.40.233, 172.66.43.23
Response IP 172.66.40.233
Found Yes
Hash 4f1d1a658f364c99edb6d9e7223a5046b0fc324c87e6a255d92e0d2d4d666e55
SimHash 687840d0c4c0

Groups

trendkite-akashic-crawler

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

neevabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

netseer

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

um-ln

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 6

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

coccocbot-web

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

vegebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 15

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

archive.org_bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

blekkobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

mojeekbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

exabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

spbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

haosouspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

linkdexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

boardtracker

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mail.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

proximic

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

*

Rule Path
Disallow /https.php
Disallow /misc.php?do=whoposted*
Disallow /misc.php?do=whoposted
Disallow /wikiUser%3A*
Disallow /tfh/living.php
Disallow /tfh/eg/
Disallow /cdn-cgi/
Disallow /b/
Disallow privacy.php
Disallow /outlet/
Disallow /a/
Disallow /as/
Disallow /?pageid=outlet
Disallow /forums/member.php
Disallow /forums/vbgooglemapme.php
Disallow /forums/vaispy.php
Disallow /forums/ajax.php
Disallow /forums/attachment.php
Disallow /forums/calendar.php
Disallow /forums/cron.php
Disallow /forums/editpost.php
Disallow /forums/global.php
Disallow /forums/image.php
Disallow /forums/faq.php
Disallow /forums/inlinemod.php
Disallow /forums/joinrequests.php
Disallow /forums/login.php
Disallow /forums/misc.php
Disallow /forums/moderator.php
Disallow /forums/newattachment.php
Disallow /forums/newreply.php
Disallow /forums/newthread.php
Disallow /forums/online.php
Disallow /forums/poll.php
Disallow /forums/postings.php
Disallow /forums/printthread.php
Disallow /forums/private.php
Disallow /forums/profile.php
Disallow /forums/register.php
Disallow /forums/report.php
Disallow /forums/reputation.php
Disallow /forums/search.php
Disallow /forums/sendmessage.php
Disallow /forums/payments.php
Disallow /forums/showpost.php
Disallow /forums/spiders.php
Disallow /forums/subscription.php
Disallow /forums/threadrate.php
Disallow /forums/usercp.php
Disallow /forums/usernote.php
Disallow /forums/admincp/
Disallow /forums/cgi-bin/
Disallow /forums/includes/
Disallow /forums/install/
Disallow /forums/modcp/
Disallow /forums/customavatars/
Disallow /forums/xmlrpc.php
Disallow /forums/archive/
Disallow /forums/sitemap/
Disallow /forums/blogs/list/
Disallow /wiki/vb_wiki_link.php
Disallow /vb_projects_link.php
Disallow /vb_buy_link.php
Disallow /editpost.php
Disallow /gamercard.php
Disallow /inlinemod.php
Disallow /member.php
Disallow /memberlist.php
Disallow /newreply.php
Disallow /newthread.php
Disallow /payments.php
Disallow /printthread.php
Disallow /private.php
Disallow /profile.php
Disallow /report.php
Disallow /search.php
Disallow /sendmessage.php
Disallow /showpost.php
Disallow /usercp.php
Disallow /usernote.php
Disallow /redirect/edugeek/1
Disallow /files/n/
Disallow /wiki/api.php
Disallow /online.php
Disallow /subscription.php
Disallow /vaispy.php
Disallow /login.php?do=login
Disallow /profile.php?do=dismissnotice

Other Records

Field Value
crawl-delay 5

mediapartners-google

Rule Path
Allow /member.php
Allow /private.php
Allow /usercp.php

Other Records

Field Value
sitemap https://www.edugeek.net/sitemap_index.xml.gz
sitemap https://www.edugeek.net/wiki/sitemap-index-edugeek_v5_1-mw_.xml

Comments

  • Block trendkite-akashic-crawler

Warnings

  • 4 invalid lines.