webster.edu
robots.txt
Robots Exclusion Standard data for webster.edu
Resource Scan
Scan Details
| Site Domain | webster.edu |
| Base Domain | webster.edu |
| Scan Status | Ok |
| Last Scan | 2024-11-07T09:19:06+00:00 |
| Next Scan | 2024-12-07T09:19:06+00:00 |
Last Scan
| Scanned | 2024-11-07T09:19:06+00:00 |
| URL | https://webster.edu/robots.txt |
| Domain IPs | 3.209.26.193, 34.236.193.193, 52.73.2.219 |
| Response IP | 34.236.193.193 |
| Found | Yes |
| Hash | fd506b790c7d27df20ce6aa14400c224d360443bd418814f9b0d6f5e2c137d3a |
| SimHash | ccc430c9d7c9 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /accel/ |
| Disallow | /404.php |
| Disallow | /_resources |
| Disallow | /alumni/ |
| Disallow | /apply/ |
| Disallow | /arc/ |
| Disallow | /catalog/_resources/ |
| Disallow | /catalog/_spif/ |
| Disallow | /catalog/catalog/ |
| Disallow | /catalog/catalog_program_archives/ |
| Disallow | /catalog/clean-original-catalog-for-review/ |
| Disallow | /catalog/documents/ |
| Disallow | /catalog/images/ |
| Disallow | /chancellor/ |
| Disallow | /chess/ |
| Disallow | /cms*/ |
| Disallow | /community/ |
| Disallow | /conversations/ |
| Disallow | /counseling/ |
| Disallow | /cybersecurity/ |
| Disallow | /dadah/ |
| Disallow | /december-toast/ |
| Disallow | /deiconference2024/ |
| Disallow | /documents/ |
| Disallow | /dualadmission/ |
| Disallow | /edd/ |
| Disallow | /events/ |
| Disallow | /filmseries/ |
| Disallow | /findyouradvisor/ |
| Disallow | /finearts/ |
| Disallow | /florida/ |
| Disallow | /fontbonne/ |
| Disallow | /forensic-accounting/ |
| Disallow | /fortbelvoir/ |
| Disallow | /fortgordon/ |
| Disallow | /fortliberty/ |
| Disallow | /forward/ |
| Disallow | /fye/ |
| Disallow | /galmichescholarship/ |
| Disallow | /global-citizen/ |
| Disallow | /globalimpact/ |
| Disallow | /gmc/ |
| Disallow | /gorloksgive/ |
| Disallow | /gowebster/ |
| Disallow | /grad/ |
| Disallow | /gradcatalog/ |
| Disallow | /halloffame/ |
| Disallow | /human-resources.php |
| Disallow | /humanrights/ |
| Disallow | /huntgallery/ |
| Disallow | /images/ |
| Disallow | /index-219125.php |
| Disallow | /iris/ |
| Disallow | /jointbaseandrews/ |
| Disallow | /kindermusik/ |
| Disallow | /light-cooperative/ |
| Disallow | /login/ |
| Disallow | /logs/ |
| Disallow | /lr/ |
| Disallow | /maygallery/ |
| Disallow | /media/ |
| Disallow | /mentor/ |
| Disallow | /militaryinfo/ |
| Disallow | /militaryonline/ |
| Disallow | /militarytimes/ |
| Disallow | /nominate/ |
| Disallow | /nursing-camp/ |
| Disallow | /ocp/ |
| Disallow | /openhouse/ |
| Disallow | /orientation/ |
| Disallow | /payment/ |
| Disallow | /phagecamp/ |
| Disallow | /pla/ |
| Disallow | /previewdays/ |
| Disallow | /rad/ |
| Disallow | /registernow/ |
| Disallow | /registration/ |
| Disallow | /rightingwrongs/ |
| Disallow | /rolla/ |
| Disallow | /sanantonio/ |
| Disallow | /sanantonio-metro/ |
| Disallow | /schedule/ |
| Disallow | /scholarship-application/ |
| Disallow | /scholarships/ |
| Disallow | /sie/ |
| Disallow | /soc/ |
| Disallow | /soe/ |
| Disallow | /stl/ |
| Disallow | /studyabroad/ |
| Disallow | /succeed/ |
| Disallow | /supportcms/ |
| Disallow | /sustainability/ |
| Disallow | /tap/ |
| Disallow | /tesl/ |
| Disallow | /texas/ |
| Disallow | /titleix |
| Disallow | /today/ |
| Disallow | /tour/ |
| Disallow | /transfer/ |
| Disallow | /transfercredit/ |
| Disallow | /transferevents/ |
| Disallow | /transferguide/ |
| Disallow | /unlimited/ |
| Disallow | /unlock/ |
| Disallow | /unlockyourgorlok/ |
| Disallow | /video_defer.js |
| Disallow | /virtualvisit/ |
| Disallow | /visit/ |
| Disallow | /vote/ |
| Disallow | /walkerschool/ |
| Disallow | /webnet/ |
| Disallow | /website-training-resources/ |
| Disallow | /websitetraining-resources.php |
| Disallow | /websteralerts/ |
| Disallow | /websterleads/ |
| Disallow | /websterspeaks/ |
| Disallow | /wins.php |
| Disallow | /winsabroad/ |
| Disallow | /wsa/ |
| Disallow | /wsbt/ |
| Disallow | /wsbt-mha/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.webster.edu/sitemap.xml |