sg.jobsdb.com
robots.txt
Robots Exclusion Standard data for sg.jobsdb.com
Resource Scan
Scan Details
| Site Domain | sg.jobsdb.com |
| Base Domain | jobsdb.com |
| Scan Status | Ok |
| Last Scan | 2024-11-16T00:25:08+00:00 |
| Next Scan | 2024-11-30T00:25:08+00:00 |
Last Scan
| Scanned | 2024-11-16T00:25:08+00:00 |
| URL | https://sg.jobsdb.com/robots.txt |
| Domain IPs | 104.18.32.227, 172.64.155.29, 2606:4700:4400::6812:20e3, 2606:4700:4400::ac40:9b1d |
| Response IP | 172.64.155.29 |
| Found | Yes |
| Hash | bda68b1c2c8cee3317ce9990691992f69af00ea5b369f10c9e9a8768dc313cc2 |
| SimHash | 0836f0d0c581 |
Groups
googlebot
bingbot
amazonbot
baiduspider
blekkobot
duckduckbot
ecosia
exabot
facebookexternalhit
yeti/naverbot
slurp
seznambot
sogou spider
soso spider
yandexbot
twitterbot
alexabot
apis-google
adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
duplexweb-google
googlebot-image
googlebot-news
googlebot-video
adidxbot
bingpreview
ahrefsbot
architextspider
crawler4j
rogerbot
semrushbot
ia_archiver
lycos_spider_(t-rex)
speedy_spider
teoma
| Product | Comment |
|---|---|
| ia_archiver | Alexa web-wide crawler |
| Rule | Path |
|---|---|
| Disallow | /rpc/ |
| Disallow | /job/rd/ |
| Disallow | /job/description/ |
| Disallow | /job-search/ |
| Disallow | /view-job/ |
| Disallow | /vanity/ |
| Disallow | /rss |
| Disallow | /style-guide/ |
| Disallow | /iniciar-sesion/ |
| Disallow | /cdn-cgi/ |
| Disallow | /*nofollow%3Dtrue* |
*
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 1 invalid line.