Log In Sign Up

tamut.edu
robots.txt

Robots Exclusion Standard data for tamut.edu

Resource Scan

Scan Details

Site Domain tamut.edu
Base Domain tamut.edu
Scan Status Ok
Last Scan2024-08-31T19:08:25+00:00
Next Scan 2024-09-30T19:08:25+00:00

Last Scan

Scanned2024-08-31T19:08:25+00:00
URL https://tamut.edu/robots.txt
Domain IPs 199.79.56.116
Response IP 199.79.56.116
Found Yes
Hash e82a9c604def3d554bfea29c17e88f6faef654a02b214d619d17e98515ab4af5
SimHash 63e15efe8698

Groups

*

Rule Path
Disallow /css/
Disallow /dev/
Disallow /error/
Disallow /ico/
Disallow /images/
Disallow /img/
Disallow /includes/
Disallow /js/
Disallow /less/
Disallow /news/bringle/index.php
Disallow /oldwebpages
Disallow /techde/support/

almaden

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

axmo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

booch

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

downloader

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

expired domain sleuth

Rule Path
Disallow /

franklin locator

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

grub

Rule Path
Disallow /

hughcrawler

Rule Path
Disallow /

iaea.org

Rule Path
Disallow /

lcabotaccept

Rule Path
Disallow /

iconsurf

Rule Path
Disallow /

iltrovatore-setaccio

Rule Path
Disallow /

indy library

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

iupui

Rule Path
Disallow /

kittiecentral

Rule Path
Disallow /

iaea.org

Rule Path
Disallow /

larbin

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

metatagrobot

Rule Path
Disallow /

missigua locator

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

nextgensearch

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

peerbot

Rule Path
Disallow /

pictureofinternet

Rule Path
Disallow /

plantynet

Rule Path
Disallow /

quepasacreep

Rule Path
Disallow /

scspider

Rule Path
Disallow /

soft411

Rule Path
Disallow /

spider.acont.de

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

ssm agent

Rule Path
Disallow /

tamu

Rule Path
Disallow /

theusefulbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tutorial crawler

Rule Path
Disallow /

tutorgig

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webzip

Rule Path
Disallow /

zipppbot

Rule Path
Disallow /

xenu

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wget

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

mozdex

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

msnbot-news

Rule Path
Disallow /

msnbot-products

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

lanshanbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

slurp

Rule Path
Disallow /sacs

entireweb

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

scooter

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

bigsearch

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

sensis

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

modspider

Rule Path
Disallow /

fast enterprise crawler/6.4

Rule Path
Disallow /

teoma

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

libwww-perl/5.803

Rule Path
Disallow /

libwww-perl/5.805

Rule Path
Disallow /

libwww-perl/5.808

Rule Path
Disallow /

yandex

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.tamut.edu/sitemap.xml
Back to top