ecareerfairs.com
robots.txt

Robots Exclusion Standard data for ecareerfairs.com

Resource Scan

Scan Details

Site Domain ecareerfairs.com
Base Domain ecareerfairs.com
Scan Status Ok
Last Scan2024-09-15T11:40:53+00:00
Next Scan 2024-10-15T11:40:53+00:00

Last Scan

Scanned2024-09-15T11:40:53+00:00
URL https://www.ecareerfairs.com/robots.txt
Domain IPs 52.170.41.29
Response IP 52.170.41.29
Found Yes
Hash 15a247184e603b87c3ea0c08f3910a589a13b6588704607bcde0b0d2faff06e3
SimHash b318d8482dda

Groups

*

Rule Path
Disallow /FormsLogin.asp
Disallow /formslogin.asp
Disallow /EmployerX/LoginForm.asp
Disallow /employerx/loginform.asp
Disallow /JobSeekerX/LoginForm.asp
Disallow /jobseekerx/loginform.asp
Disallow /JobSeekerX/ViewSpecialJobs.asp
Disallow /jobseekerx/viewspecialjobs.asp
Disallow /JobSeekerX/ViewSpecialJobsTable.asp
Disallow /jobseekerx/viewspecialjobstable.asp
Disallow /JobSeekerX/SearchCompanyProfiles.asp
Disallow /jobseekerx/searchcompanyprofiles.asp
Disallow /JobSeekerX/ResetLoginForm.asp
Disallow /jobseekerx/resetloginform.asp
Disallow /EmployerX/ResetLoginForm.asp
Disallow /employerx/resetloginform.asp

*

Rule Path
Disallow /SiteSpecificScripts/
Disallow /Media/
Disallow /SiteSpecificAdmin/
Disallow /Help/
Disallow /Messages/
Disallow /Services/
Disallow /ExportTemplates/
Disallow /Auth/
Disallow /Bin/
Disallow /sitespecificscripts/
Disallow /media/
Disallow /sitespecificadmin/
Disallow /help/
Disallow /messages/
Disallow /services/
Disallow /exporttemplates/
Disallow /auth/
Disallow /bin/
Disallow /scripts/
Disallow /includes/
Disallow /components/

*

Rule Path
Disallow /*.zip$
Disallow /*.doc$
Disallow /*.exe$

atspider

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

elitesys entry

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mail sweeper

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

munky

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

webemailextrac

Rule Path
Disallow /

xget

Rule Path
Disallow /

wget

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

vobsub

Rule Path
Disallow /

templeton

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ssearcher100

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

shai'hulud

Rule Path
Disallow /

pbwf

Rule Path
Disallow /

lightningdownload

Rule Path
Disallow /

kdd exploror

Rule Path
Disallow /

jeeves

Rule Path
Disallow /

internet explore

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

httrack

Rule Path
Disallow /

havindex

Rule Path
Disallow /

geturl

Rule Path
Disallow /

getbot

Rule Path
Disallow /

esirover

Rule Path
Disallow /

download wonder

Rule Path
Disallow /

collage

Rule Path
Disallow /

mozilla/2.0 (compatible; ms frontpage 4.0)

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

Comments

  • Robots.txt
  • Disallow indexing of specific pages
  • Disallow directories
  • Disallow directories - lower case
  • Disallow indexing of specific file extensions
  • Backlink Analysis
  • Disallow Collectors and Spam
  • https://megaindex.com/crawler
  • Disallow Offline Browsers