hupro-job.com
robots.txt

Robots Exclusion Standard data for hupro-job.com

Resource Scan

Scan Details

Site Domain hupro-job.com
Base Domain hupro-job.com
Scan Status Ok
Last Scan2024-11-12T01:44:59+00:00
Next Scan 2024-11-19T01:44:59+00:00

Last Scan

Scanned2024-11-12T01:44:59+00:00
URL https://hupro-job.com/robots.txt
Domain IPs 52.194.173.55, 54.199.5.6, 57.180.114.134
Response IP 54.199.5.6
Found Yes
Hash 9f7389d1fc9e9bbd32f91b17f056e91a6b98feead0eac1cd7853fad510136565
SimHash 625ccfb0cebb

Groups

*

Rule Path
Disallow /accounts
Disallow /articles/*/tracker
Disallow /business/thanks
Disallow /contacts
Disallow /jobs?*companies_job%5B*
Disallow /jobs/accounting?*companies_job%5B*
Disallow /jobs/human_resources?*companies_job%5B*
Disallow /jobs/legal?*companies_job%5B*
Disallow /jobs/recommends
Disallow /jobs/*/preview
Disallow /jobs/*/tracker
Disallow /oauth
Disallow /optout
Disallow /password_resets
Disallow /upgrade
Disallow /unlocks

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

proximic

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

integralads

Rule Path
Disallow /

jet-bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /