sg.jobsdb.com
robots.txt
Robots Exclusion Standard data for sg.jobsdb.com
Resource Scan
Scan Details
| Site Domain | sg.jobsdb.com |
| Base Domain | jobsdb.com |
| Scan Status | Ok |
| Last Scan | 2024-10-04T19:12:02+00:00 |
| Next Scan | 2024-10-18T19:12:02+00:00 |
Last Scan
| Scanned | 2024-10-04T19:12:02+00:00 |
| URL | https://sg.jobsdb.com/robots.txt |
| Domain IPs | 104.18.32.227, 172.64.155.29, 2606:4700:4400::6812:20e3, 2606:4700:4400::ac40:9b1d |
| Response IP | 104.18.32.227 |
| Found | Yes |
| Hash | 330ccb33d9b70da69351b19e27667b3310496cbdd8ece7b7425c42c38a6fa23c |
| SimHash | 0876f0d0c501 |
Groups
googlebot
bingbot
amazonbot
baiduspider
blekkobot
duckduckbot
ecosia
exabot
facebookexternalhit
yeti/naverbot
slurp
seznambot
sogou spider
soso spider
yandexbot
twitterbot
alexabot
apis-google
adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
duplexweb-google
googlebot-image
googlebot-news
googlebot-video
adidxbot
bingpreview
ahrefsbot
architextspider
crawler4j
rogerbot
semrushbot
ia_archiver
lycos_spider_(t-rex)
speedy_spider
teoma
| Product | Comment |
|---|---|
| ia_archiver | Alexa web-wide crawler |
| Rule | Path |
|---|---|
| Disallow | /rpc/ |
| Disallow | /rd/ |
| Disallow | /job/description/ |
| Disallow | /job-search/ |
| Disallow | /view-job/ |
| Disallow | /vanity/ |
| Disallow | /rss |
| Disallow | /style-guide/ |
| Disallow | /iniciar-sesion/ |
| Disallow | /cdn-cgi/ |
| Disallow | /*nofollow%3Dtrue* |
*
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 1 invalid line.