units.academia.edu
robots.txt

Robots Exclusion Standard data for units.academia.edu

Resource Scan

Scan Details

Site Domain units.academia.edu
Base Domain academia.edu
Scan Status Ok
Last Scan2024-11-08T09:57:29+00:00
Next Scan 2024-11-22T09:57:29+00:00

Last Scan

Scanned2024-11-08T09:57:29+00:00
URL https://units.academia.edu/robots.txt
Domain IPs 108.157.254.49, 108.157.254.61, 108.157.254.8, 108.157.254.9
Response IP 108.157.254.8
Found Yes
Hash 6adb97adaa29a93540eb5ad577081481520b38709a997d1db5bb2d919d7ab0eb
SimHash 6a144110c591

Groups

googlebot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

mediapartners-google

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

msnbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

bingbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

slurp

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

duckduckbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

baiduspider

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

seznambot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

naverbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

teoma

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

yandex

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

scoutjet

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

gurujibot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

exabot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

solbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

fast

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

voilabot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

daumoa

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

architextspider

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

twitterbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

socscibot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

ia_archiver

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

coccoc

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

browsershots

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

sitearc

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

cliqzbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

proximic

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

oncrawlbot

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

botify

Rule Path
Disallow /ads.txt
Disallow /attachments/*/download_file
Disallow /Jobs/Search
Disallow /people/search
Disallow /ac_search
Disallow /users/search
Disallow /ac_universities
Disallow /ac_university_departments
Disallow /document_viewer/build_vu_embeds/
Disallow /v0/*
Allow /v0/users/*/following
Allow /v0/users/*/coauthors
Allow /

*

Rule Path
Disallow /

Other Records

Field Value
sitemap https://sitemap.academia.edu/works_index.xml

Comments

  • If you run a search engine and would like to index Academia.edu, please email support@academia.edu.