career.aero
robots.txt

Robots Exclusion Standard data for career.aero

Resource Scan

Scan Details

Site Domain career.aero
Base Domain career.aero
Scan Status Ok
Last Scan2024-09-30T05:38:44+00:00
Next Scan 2024-10-30T05:38:44+00:00

Last Scan

Scanned2024-09-30T05:38:44+00:00
URL https://career.aero/robots.txt
Redirect https://www.career.aero/robots.txt
Redirect Domain www.career.aero
Redirect Base career.aero
Domain IPs 213.183.200.231, 2a00:4e00:2000:169::231
Redirect IPs 213.183.200.231, 2a00:4e00:2000:169::231
Response IP 213.183.200.231
Found Yes
Hash 55fc813d5854cfa38b97342aa3559b2884a7bb8ddbda55e8e618e9b4478514d3
SimHash 531e5440cc9a

Groups

mediapartners-google

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

Rule Path
Disallow /

botpoke

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

domcopbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

freshbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

jaddjabot

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

paqlebot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

python

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

vsusearchspider

Rule Path
Disallow /

wpbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /