Log In Sign Up

web.njit.edu
robots.txt

Robots Exclusion Standard data for web.njit.edu

Resource Scan

Scan Details

Site Domain web.njit.edu
Base Domain njit.edu
Scan Status Ok
Last Scan2025-09-01T02:30:57+00:00
Next Scan 2025-10-01T02:30:57+00:00

Last Scan

Scanned2025-09-01T02:30:57+00:00
URL https://web.njit.edu/robots.txt
Domain IPs 128.235.208.46
Response IP 128.235.208.46
Found Yes
Hash 6bfd1a230f1a48311714ee4c9419a4ab74026ae40a728632d3a7ae04bab77cb1
SimHash 2254c56c2d75

Groups

*

Rule Path
Disallow /~bieber/cis*
Disallow /~bieber/cis365f02
Disallow /~bieber/cis365s03
Disallow /~bieber/CIS*
Disallow /~bieber/CIS105
Disallow /~bieber/CIS105-CDROM
Disallow /~bieber/CIS105-stuff
Disallow /~bieber/CIS365
Disallow /~bieber/CIS365-stuff
Disallow /~bieber/CIS365F97
Disallow /~bieber/CIS365F98
Disallow /~bieber/CIS365F99
Disallow /~bieber/CIS365F99DL
Disallow /~bieber/CIS677
Disallow /~bieber/CIS677-CDROM
Disallow /~bieber/CIS677F00
Disallow /~bieber/CIS677F04
Disallow /~bieber/CIS677F98
Disallow /~bieber/CIS677F99
Disallow /~bieber/CIS677S00
Disallow /~bieber/CIS677S01
Disallow /~bieber/CIS677S02
Disallow /~bieber/CIS677S05
Disallow /~bieber/CIS677S98
Disallow /~bieber/CIS677S99
Disallow /~bieber/itr01*
Disallow /~bieber/itr01-supplemental
Disallow /~bieber/todo
Disallow /~bieber/wu-pt
Back to top

Comments

  • NJIT update notes:
  • This file resides at /afs/.cad.njit.edu/solaris/apache.shared/robots.txt
  • therefore a 'vos release apache.shared' is required after update!
  • 7-Jun-2010 /KJW
  • Following requested by M.Bieber on 2-Jun-2010 /KJW
  • following disallows requested by Aslam on 2/11/2014. /krane
  • to disallow any tilde user dir ending in numbers.
  • removed 9/19/2014 temporarily
  • Disallow: /~*0
  • Disallow: /~*1
  • Disallow: /~*2
  • Disallow: /~*3
  • Disallow: /~*4
  • Disallow: /~*5
  • Disallow: /~*6
  • Disallow: /~*7
  • Disallow: /~*8
  • Disallow: /~*9
Back to top