cyberworkers.com
robots.txt
Robots Exclusion Standard data for cyberworkers.com
Resource Scan
Scan Details
| Site Domain | cyberworkers.com |
| Base Domain | cyberworkers.com |
| Scan Status | Ok |
| Last Scan | 2024-11-12T11:51:31+00:00 |
| Next Scan | 2024-11-19T11:51:31+00:00 |
Last Scan
| Scanned | 2024-11-12T11:51:31+00:00 |
| URL | https://cyberworkers.com/robots.txt |
| Domain IPs | 2001:41d0:301::29, 51.91.236.255 |
| Response IP | 51.91.236.255 |
| Found | Yes |
| Hash | 003ac5e69de95e8714af9e705767193729b4cdb53c6275ac30cf8dd7a7dcb5c8 |
| SimHash | 5e1b4aa2c3f3 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /dir/tarifs-permanence-telephonique-25%-mois-cher.html |
| Disallow | /azurrelance |
| Disallow | /creation-bases-de-donnees-et-fichiers-2 |
| Disallow | /creation-site-web-suite-campagne-google |
| Disallow | /comment-on-buzz-sur-vos-services |
| Disallow | /si-vous-ne-signez-pas-daffaire |
| Disallow | /mise-a-jour-boutique-en-ligne |
| Disallow | /silverhamaro |
| Disallow | /benefices-du-teletravail |
| Disallow | /ralentissement-des-affaires |
| Disallow | /pige-immobiliere-agence-belfort |
| Disallow | /gouvernement-et-le-teletravail |
| Disallow | /plaquette-pour-festival |
| Disallow | /lien-vers-votre-site-depuis-cyberworkers-com |
| Disallow | /hgadrey |
| Disallow | /retranscription-dune-video |
| Disallow | /gpoptimum |
| Disallow | /agendadeglv |
| Disallow | /votreassistante |
| Disallow | /quotidien-teletravailleurstele |
| Disallow | /dir/creation-de-site-web-dynamique-100%-personalise-web-2.0-et-html-5.html |
| Disallow | /dir/tarifs-permanence-telephonique-25%-mois-cher.html |
a6-indexer
abachobot
abcdatosbotlink
acontbot
adsacomponent
admantx
aesop_com_spiderman
aibot
aisearchbot
aitcsrobot
aleadsoftbot
amznkassocbot
aol
aphpscript
ariadne
aspseek
aspsimply
atnworldwide
atn_worldwide
auresys
avsearch
awscloudbased
awsbot
abachobot
abonti
aboundex
aboundexbot
about
aboutusbot
aboutwebsite
academicbot
acadiauniversitywebcensusclient
accelobot
accoona
accoona-ai-agent
acoon
acoon-robot
acoonbot
activeagent
activecacherequest
acunetix
acunetixsecurityscanner
adfenixsegmentationcrawler
adidxbot
admediabot
adreport
adshadow
adsonarbot
adamaticsolutions
addcatalog
addsearchbot
addsugarspiderbot
addthis
addurlrobot
adidum
adidxbot
admantx
adnormcrawler
baiduspider
baiduspider-image
baiduspider-video
bytespider
mj12bot
naverbot
petalbot
sogou inst spider
sogou web spider
yandex
yeti
yisouspider
youdaobot
adsbot
ichiro
moget
sogou spider
semrushbot
semrushbot
friendica
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | http://www.cyberworkers.com/sitemap.xml |
| sitemap | http://www.cyberworkers.com/sitemap_projets.xml |
| sitemap | http://www.cyberworkers.com/sitemap_dir.xml |
| sitemap | http://www.cyberworkers.com/sitemap_dircat.xml |
| sitemap | http://www.cyberworkers.com/sitemap_static.xml |
| sitemap | http://www.cyberworkers.com/post-sitemap.xml |
Warnings
- 8 invalid lines.