Log In Sign Up

tws.edu
robots.txt

Robots Exclusion Standard data for tws.edu

Resource Scan

Scan Details

Site Domain tws.edu
Base Domain tws.edu
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2/28/2025, 12:20:54 AM
Next Scan 5/29/2025, 12:20:54 AM

Last Successful Scan

Scanned7/16/2023, 9:15:13 PM
URL https://tws.edu/robots.txt
Redirect https://www.tws.edu/robots.txt
Redirect Domain www.tws.edu
Redirect Base tws.edu
Domain IPs 104.21.67.10, 172.67.167.112, 2606:4700:3030::6815:430a, 2606:4700:3034::ac43:a770
Redirect IPs 104.21.67.10, 172.67.167.112, 2606:4700:3030::6815:430a, 2606:4700:3034::ac43:a770
Response IP 172.67.167.112
Found Yes
Hash b4354e5667a5cc3632ff3fec6679995cd53013f7ed1cf84296e7c915460d2180
SimHash 6810c241d9ff

Groups

*

Rule Path
Disallow /success/
Disallow /welcome/
Disallow /beta/
Disallow /dev/
Disallow /dev2/
Disallow /dev3/
Disallow /dev4/
Disallow /redesign/
Disallow /application/
Disallow /landing/
Disallow /example-landing-page/
Disallow /financial-aid/tuition-calculator/
Disallow /includes/NPC
Disallow /get-started/
Disallow /live-stream/
Disallow /bracket/
Disallow /blog/review/
Disallow /calculator/
Disallow /leaderboard/
Disallow /empezar/

googlebot-image

Rule Path
Disallow
Allow /*

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

duggmirror

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /trackback/
Disallow /feed/
Disallow /comments/
Disallow /category/*/*
Disallow */trackback/
Disallow */feed/
Disallow */comments/
Allow /wp-content/uploads/
Allow /wp-content/themes/

twitterbot

Rule Path
Allow /images

facebookexternalhit

Rule Path
Allow /images
Back to top

Other Records

Field Value
sitemap https://www.tws.edu/sitemap_index.xml
Back to top

Comments

  • Google Image
  • Google AdSense
  • Google Ads
  • digg mirror
  • global
  • Disallow: /wp-includes/ REMOVED 07/29/2015 - BM (https://yoast.com/wordpress-robots-txt-example/)
  • Disallow: /wp-content/plugins/ REMOVED 07/29/2015 - BM (https://yoast.com/wordpress-robots-txt-example/)
  • Disallow: /*? REMOVED 07/29/2015 - BM
  • Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. (12/10/18 - BM)
Back to top

Warnings

  • `noindex` is not a known field.
Back to top