www.cos.youth4work.com
robots.txt

Robots Exclusion Standard data for www.cos.youth4work.com

Resource Scan

Scan Details

Site Domain www.cos.youth4work.com
Base Domain youth4work.com
Scan Status Ok
Last Scan2024-11-03T10:09:05+00:00
Next Scan 2024-12-03T10:09:05+00:00

Last Scan

Scanned2024-11-03T10:09:05+00:00
URL https://www.cos.youth4work.com/robots.txt
Domain IPs 20.192.170.11
Response IP 20.192.170.11
Found Yes
Hash 24a68acaadf0aae3a11a992239d6bdb6f997d56184884633e7d6f9f5a00c71ff
SimHash 0c56db5261c1

Groups

googlebot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /bn/
Disallow /de/
Disallow /fr/
Disallow /gu/
Disallow /it/
Disallow /ja/
Disallow /kn/
Disallow /ml/
Disallow /mr/
Disallow /or/
Disallow /pa/
Disallow /ru/
Disallow /ta/
Disallow /te/
Disallow /tr/
Disallow /ur/
Disallow /zh/
Disallow /ar/
Disallow /pt/

applebot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

bingbot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

Other Records

Field Value
crawl-delay 1

msnbot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

Other Records

Field Value
crawl-delay 10

googlebot-image

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /bn/
Disallow /de/
Disallow /fr/
Disallow /gu/
Disallow /it/
Disallow /ja/
Disallow /kn/
Disallow /ml/
Disallow /mr/
Disallow /or/
Disallow /pa/
Disallow /ru/
Disallow /ta/
Disallow /te/
Disallow /tr/
Disallow /ur/
Disallow /zh/
Disallow /ar/
Disallow /pt/

googlebot-news

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /bn/
Disallow /de/
Disallow /fr/
Disallow /gu/
Disallow /it/
Disallow /ja/
Disallow /kn/
Disallow /ml/
Disallow /mr/
Disallow /or/
Disallow /pa/
Disallow /ru/
Disallow /ta/
Disallow /te/
Disallow /tr/
Disallow /ur/
Disallow /zh/
Disallow /ar/
Disallow /pt/

googlebot-video

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /bn/
Disallow /de/
Disallow /fr/
Disallow /gu/
Disallow /it/
Disallow /ja/
Disallow /kn/
Disallow /ml/
Disallow /mr/
Disallow /or/
Disallow /pa/
Disallow /ru/
Disallow /ta/
Disallow /te/
Disallow /tr/
Disallow /ur/
Disallow /zh/
Disallow /ar/
Disallow /pt/

googlebot-mobile

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /bn/
Disallow /de/
Disallow /fr/
Disallow /gu/
Disallow /it/
Disallow /ja/
Disallow /kn/
Disallow /ml/
Disallow /mr/
Disallow /or/
Disallow /pa/
Disallow /ru/
Disallow /ta/
Disallow /te/
Disallow /tr/
Disallow /ur/
Disallow /zh/
Disallow /ar/
Disallow /pt/

mediapartners-google

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/

adsbot-google

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/

adsbot-google-mobile

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/

adsbot-google-mobile-apps

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/

baiduspider

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /es/
Disallow /hi/
Disallow /bn/
Disallow /it/

seznambot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

teoma

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

yandex

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /hi/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

Other Records

Field Value
crawl-delay 1

yeti

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

msnbot-media

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

duckduckbot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

daumoa

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

orangebot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

orangebot-collector

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

stackrambler

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

twitterbot/1.0

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

mail.ru_bot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /hi/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

naverbot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

vebidoobot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

mj12bot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

sogou

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

rogerbot

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

deepcrawl

Rule Path
Disallow /Documents/New_Openings_Resume/
Disallow /Documents/Colleges/attache/
Disallow /Documents/PrepTest/StudyMaterial/
Disallow /tr/
Disallow /gu/
Disallow /fr/
Disallow /te/
Disallow /ta/
Disallow /ml/
Disallow /mr/
Disallow /ur/
Disallow /kn/
Disallow /ar/
Disallow /pt/
Disallow /ru/
Disallow /de/
Disallow /or/
Disallow /ja/
Disallow /pa/
Disallow /zh/
Disallow /bn/
Disallow /it/

*

Rule Path
Disallow /

Warnings

  • 23 invalid lines.