CyberArk Secret management : CloudOne DevOps

CloudOne DevOps Pipeline v4.6 introduces CyberArk Integration with Rancher. Conjur performs certificate-based authentication with each Rancher cluster and securely supplies secrets to each namespace/application within the cluster. Secrets are securely stored in the CyberArk vault and securely supplied. You would need to upgrade your pipeline version to v4.6 to use this cyberark integration.

Safes created for your application:

We have provisioned 6 safes for each application for your team to use:

3 Application Safes: Each safe is designated for a specific landscape. Application teams should store their application secrets here, based on the landscape they belong to.

3 API Safes: These safes should be used for storing usernames and API keys specifically for accessing your secrets via api for development purposes using cyberark/summon.

Safe Permissions (RBAC):

Note : Member's would need to be directly added to the ng group to get access to the safe. Nested NG group will not have access.

How to create Account Manually in CyberArk:

Please refer to KB0007418 - Privileged Access Management: Vault Account Manually to know more on how to vault an account. You can "customize the account name" to specify a simpler account name, it has to be unique within the safe .

Setting Up Secrets in Values Files for appstacks:

The application team must use the CyberArk account to build secrets. You can specify the username and/or password for each secret. To enable CyberArk integration with CloudOne, you must have a properly formed values.yaml file.

Step 1:

Upgrade your pipeline to v4.6. See release notes and pipeline upgrade tool.

Step 2:

cyberark:
  secrets:
    DB_USER: "sample_db/username"
    DB_PASSWORD: "sample_db/password"
    # API_TOKEN: "sample_api/password"

• Key Name (DB_USER): The name of the environment variable to create in your container.
• Account Name (sample_db): unique name of the account which you create in cyberark.
• Variable Name (username)/(password): Specify if the username or password field of the CyberArk secret should be retrieved.

Step 3:

Copy our cyberark-secrets.yml file and paste it directly into the templates directory of your repo.

Step 4:

Update the deployment.yml in your repo and add the 4 lines as done in our deployment.yml.

{{- if .Values.cyberark.secrets }}
  - secretRef:
      name: {{ include "app.fullname" . }}-cyberark-secrets
{{- end }}

Database Modifications:

Database passwords were previously stored in variable groups; with Cyberark, we can vault the secrets .

Upgrading your databases to pipeline v4.6 requires that you migrate/store your database credentials to CyberArk using the method described above.

In addition to the steps outline above, you will also need to perform the following steps for databases.

Step 1:

You will need to copy our secrets.yml to the top-level directory of your database repo. *Please note: The secrets.yml file is specific to each database type/technology.

The link above is for mongodb. Please select our repo that matches your database technology and select the release/v4.6 branch.

Step 2:

Copy our cyberark-secrets.yml file and paste it directly into the templates directory of your repo. cyberark-secrets.yml

Step 3:

Update your values file as we have in our values.yml.

The link above is for mongodb. Please select our repo that matches your database technology and select the release/v4.6 branch.

We recommend that the account name that you create in CyberArk is unique for each database, since your safe may contain db credentials for multiple databases. For example, in your safe, create a secret that contains your mongoAdmin credentials and use <namespace>-mongoAdmin as the account name for easy identification and separation. You should add the created account to two safes: one based on the landscape and another with the devint/workspace safe "S-D-C-RNCHR-CNJR-{{ appcode }}". This ensures that the workspace can access the password as well .

Step 4:

Remove the *.props reference from your azure-pipelines.yml file.

REMOVE:

- { group: mongodb.props }

Step 5:

Copy our deployment file for your database stack file and paste it directly into the your repo. , example for mongo - deployment-standalone.yaml . change from previous version is "secretRef" .

Conjur in CI/CD:

Secrets are deployed before the application pod deployment, ensuring that when the pod comes up, the secrets are already available for consumption. Additionally, We have added to the sync Conjur manifest as a CI/CD step, allowing users to view and fix any errors before the deployment.

If the secret path defined in your values.yml file is incorrect, this task will fail.

If you rotate your password in CyberArk, you will want to manually trigger the sync and restart your pods for your application to consume the new password. Refer to custom cronjob run to manually trigger sync and restart.

How conjur works:

More details about what is happening "under-the-hood": Secrets Provider for Kubernetes - Kubernetes Job - (cert-based authn)

Summon:

cyberark/summon is a command-line tool that reads a file in secrets.yml format and injects secrets as environment variables into processes. In the current setup, secrets might be stored as variables on the local developer's laptop. By integrating Summon with Conjur, the developer community can securely store secrets in a vault, mitigating the risks associated with storing secrets insecurely on end-user laptops.

For more information on integrating Conjur with Summon, please refer to Developers’ Guide to inject Conjur Secrets using Summon.

If you have any questions or issues configuring Summon, please open a ticket assigned to IT > FS > EUS > Identity Management Engineering

Troubleshooting:

If the Sync conjur manifests task in your pipeline fails, it should contain information about the issue.

For example, if it gives an "exceeded quota" error, you will need to perform one of the following:

1. Manually delete the current deployment in Kubernetes, update the limits in your values file, and redeploy.
2. Manually update your existing deployment in Rancher and decrease the CPU limits, update the limits in your values file, and redeploy.
3. Increase your hostspace tshirt size through the Service-Now DevOps portal.