cyberworkers.com
robots.txt

Robots Exclusion Standard data for cyberworkers.com

Resource Scan

Scan Details

Site Domain cyberworkers.com
Base Domain cyberworkers.com
Scan Status Ok
Last Scan2024-06-15T19:05:02+00:00
Next Scan 2024-06-22T19:05:02+00:00

Last Scan

Scanned2024-06-15T19:05:02+00:00
URL https://cyberworkers.com/robots.txt
Domain IPs 2001:41d0:301::29, 51.91.236.255
Response IP 51.91.236.255
Found Yes
Hash 003ac5e69de95e8714af9e705767193729b4cdb53c6275ac30cf8dd7a7dcb5c8
SimHash 5e1b4aa2c3f3

Groups

*

Rule Path
Allow /
Disallow /dir/tarifs-permanence-telephonique-25%-mois-cher.html
Disallow /azurrelance
Disallow /creation-bases-de-donnees-et-fichiers-2
Disallow /creation-site-web-suite-campagne-google
Disallow /comment-on-buzz-sur-vos-services
Disallow /si-vous-ne-signez-pas-daffaire
Disallow /mise-a-jour-boutique-en-ligne
Disallow /silverhamaro
Disallow /benefices-du-teletravail
Disallow /ralentissement-des-affaires
Disallow /pige-immobiliere-agence-belfort
Disallow /gouvernement-et-le-teletravail
Disallow /plaquette-pour-festival
Disallow /lien-vers-votre-site-depuis-cyberworkers-com
Disallow /hgadrey
Disallow /retranscription-dune-video
Disallow /gpoptimum
Disallow /agendadeglv
Disallow /votreassistante
Disallow /quotidien-teletravailleurstele
Disallow /dir/creation-de-site-web-dynamique-100%-personalise-web-2.0-et-html-5.html
Disallow /dir/tarifs-permanence-telephonique-25%-mois-cher.html

a6-indexer
abachobot
abcdatosbotlink
acontbot
adsacomponent
admantx
aesop_com_spiderman
aibot
aisearchbot
aitcsrobot
aleadsoftbot
amznkassocbot
aol
aphpscript
ariadne
aspseek
aspsimply
atnworldwide
atn_worldwide
auresys
avsearch
awscloudbased
awsbot
abachobot
abonti
aboundex
aboundexbot
about
aboutusbot
aboutwebsite
academicbot
acadiauniversitywebcensusclient
accelobot
accoona
accoona-ai-agent
acoon
acoon-robot
acoonbot
activeagent
activecacherequest
acunetix
acunetixsecurityscanner
adfenixsegmentationcrawler
adidxbot
admediabot
adreport
adshadow
adsonarbot
adamaticsolutions
addcatalog
addsearchbot
addsugarspiderbot
addthis
addurlrobot
adidum
adidxbot
admantx
adnormcrawler
baiduspider
baiduspider-image
baiduspider-video
bytespider
mj12bot
naverbot
petalbot
sogou inst spider
sogou web spider
yandex
yeti
yisouspider
youdaobot
adsbot
ichiro
moget
sogou spider
semrushbot
semrushbot
friendica

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.cyberworkers.com/sitemap.xml
sitemap http://www.cyberworkers.com/sitemap_projets.xml
sitemap http://www.cyberworkers.com/sitemap_dir.xml
sitemap http://www.cyberworkers.com/sitemap_dircat.xml
sitemap http://www.cyberworkers.com/sitemap_static.xml
sitemap http://www.cyberworkers.com/post-sitemap.xml

Warnings

  • 8 invalid lines.