Overview

CloudOne is NetApp's solution for providing DevOps-as-a-Service to application teams, combining a well-defined software development framework and workflow with the underlying infrastructure for the deployment and execution of applications from the development and test stages all the way through to production.

CloudOne is an environment that implements container technology and lends itself particularly well to the delivery and execution of application Microservices and web-based UI components, providing a continuous pipeline needed to build and deploy these components as well as cloud-based environments in which these components will run.

CloudOne is built on top of several technologies, starting with the following:

• Docker – a container technology; a container is a very small execution environment for small application components such as Microservices and small web UI components.
• Kubernetes – an orchestration environment for Docker containers that organizes and manages individual Docker containers into deployable units (e.g. X number of containers running a particular microservice where X is a quantity meant to address high availability and scalability needs for that particular microservice)
• Rancher – A distribution of Kubernetes from Suse that adds on top of Kubernetes additional management capabilities, security and resource isolation capabilities, log collection capabilities and more
• OpenShift – A distribution of Kubernetes from Red Hat that adds on top of Kubernetes additional management capabilities, security and resource isolation capabilities, log collection capabilities and more
• Git – A source code repository within Azure DevOps where the code for applications in CloudOne will be hosted in specific locations according to CloudOne conventions
• Pull Request – when making updates to source code stored in a Git repository, a local copy is made and updated, then a pull request is issued to "request" the updates to be merged into the master copy of the repository. The pull request is subject to approval before it will be merged in, and in the CloudOne environment, this approval is issued automatically by the Release pipeline when the source code and associated build reaches a certain threshold of verification.
• Artifactory – A repository for container images that will be produced by the build process from source code and ultimately delivered and deployed into CloudOne for execution
• Helm Charts – A Kubernetes distribution-agnostic mechanism for defining all the components that make up a deployable unit of an application and instruct Kubernetes in CloudOne how to deploy the application components, accounting for differences in the deployment between differing target environments using a templating mechanism
• Kustomize File – An alternative (an alternative to Helm charts) Kubernetes mechanism for defining all the components that make up a deployable unit of an application and instruct Kubernetes in CloudOne how to deploy the application components, accounting for differences in the deployment between differing target environments using a file overlay mechanism
• Azure DevOps – The underlying technology for Continuous Integration (Build) and Continuous Delivery (Deploy) pipeline stages that will deliver application components from source code to deployment in runtime environments

Provisioned Components

When onboarding an application into CloudOne or when modifying the application within CloudOne, there are 3 types of spaces can be provisioned for the application:

• Workspaces
• Hostspaces
• Dataspaces

Each of these workspaces are needed for distinct purposes and stages in the life-cycle of the application development, testing and promotion to production.

Workspaces

A workspace is the space in which developers work to produce application code. It consists of the development tools, including source code repository, build and deployment pipelines, an image repository and other development tools as well as an environment into which application components will be deployed to run during the development and initial (unit) testing of a release. This deployment environment is ephemeral in nature and expected to be short-lived. This deployment environment is implemented in Kubernetes hosted in AWS, with a limited quota for memory and CPU capacity and with a short (2-hour) time span for deployment application components to live in the environment for testing.

Hostspaces

Hostspaces are Kubernetes environments for the deployment of application components after development and initial unit testing have completed successfully. The components in Hostspaces are expected to remain deployed indefinitely. These spaces are used for Staging, Pre-Production and Production environments and are hosted in-house in NetApp data centers. When Hostspaces are provisioned, quotas can be selected according to needs to accommodate varying capacities for memory, storage and CPU size.

Dataspaces

Dataspaces are predefined database stacks within the CloudOne environment that can vary in size and other specifications, but match specific patterns for types of data technology. Currently there are 3 types of Open Source database technologies that can be configured in dataspaces :

• MongoDB – a NoSQL JSON document oriented database
• Redis – a key-value in-memory database supporting a range of different data structures for stored values
• PostgreSQL – A SQL-compatible relational database (RDBMS) system

Provisioning Procedures

Onboarding an application

The process of onboarding an application into CloudOne is handled through a custom-built ServiceNow portal named "Onboard CloudOne Application". Before this onboarding can be carried out, an application must be registered into ServiceNow so that it will be known to the onboarding mechanism. Then, the onboarding process can begin, consisting of the following:

• Inventory is updated in ServiceNow, including a Configuration Management Database (CMDB) containing information about the application and the components to be provisioned in CloudOne
• A workspace is provisioned with a fixed-sized quota (there are no options to vary the size of the workspace as what is deployed in here is intended for only short-term usage)
• App Stacks are provisioned, one for every application component listed by the application team (e.g. microservices and front-end UIs)
• Hostspaces are provisioned as requested based on the size chosen
• When provisioning Hostspaces , the network connectivity is also specified to indicate whether it will be accessible externally or internally on the network
• Active Directory groups are generated for the application (see RBAC section below) and the assignment of users into these groups is propagated to Rancher, requiring a 30-minute pause in provisioning activity before completion
• Azure DevOps pipelines and projects are created
• Artifactory (image repository) area is provisioned
• Kubernetes namespaces are provisioned ( Workspace in AWS and Hostspaces in NetApp datacenters)
• Git source code repositories are generated for every listed App Stack with an initial structure of files are directories based on the type of App Stack for each application component
• Helm Charts that will guide the deployment of components into Kubernetes are generated (but may require customization later - Helm charts are generated by default but can be replaced with Kustomize files as an alternative)

Managing an application

Managing an application already onboarded to CloudOne is very similar to the onboarding process except for steps carried out only once at the start of an application's life in CloudOne. After invoking the ServiceNow portal for managing an already onboarded application, it will load up the currently provisioned components and then allow the addition of new App Stacks (components) to the application. Additional Hostspaces can be provisioned as well. Dataspaces are also provisioned from here (please note that at this time, Dataspaces cannot be provisioned at Onboarding time and must be provisioned from the "Manage CloudOne DevOps Application" portal).

Role-Based Access Control (RBAC)

Access and permissions within the CloudOne Workspaces and Hostspaces and in Azure DevOps are determined for every user based on their assignments into a set of Active Directory groups. For every application onboarded into CloudOne, six (6) Active Directory groups are generated, as follows (where XXX is a, usually, 3-letter application code serving as a unique identifier for the application):

• ng-ngdc-XXX-admin : Admin group – The owners of the application, e.g. development lead(s) and management; this group is pre-populated from the ServiceNow CMDB and also includes the Operations group (described below) as well as the CloudOne DevOps support team. The Admin group is also included in each of the following groups as the initial users assigned. This group has very limited access and should only be used for granting access to other users. This group should contain no more than 4 users.
• ng-ngdc-XXX-dev : Developer Group – the group containing all developers with access permissions for git, pipelines, etc.
• ng-ngdc-XXX-leads : Development Leads Group – this group is authorized to approve a deployment pipeline to deploy into the Stage Kubernetes Hostspaces
• ng-ngdc-XXX-ops: Operations Group – group of users providing operational support for the application; this group has permissions to approve the deployment of the application into production Hostspaces
• ng-ngdc-XXX-dba: DBA Group – this group now has the same permissions as the dev group
• ng-ngdc-XXX-biz: Business Approvers group – a group of users outside of the development team that approves the completion of testing and readiness of the application components to proceed to Stage Hostspaces
• cg-pam-devops-XXX-dev : Developer Group – read-write access to Kubernetes Workspace and DevInt Hostspaces
• cg-pam-devops-XXX-leads : Development Leads Group – read-write access to Kubernetes Stage Hostspaces
• cg-pam-devops-XXX-ops : Operations Group – read-write access to all Kubernetes Hostspaces and Dataspaces
• cg-pam-devops-XXX-dba : DBA Group – read-write access to all Kubernetes Dataspaces

The members of the Admin group have the authority to assign additional users to any of the above-listed groups. This is done via the NEAT application at the URL: https://neat.netapp.com

These Active Directory group memberships also determine what permissions which users have in the source code repository and in Kubernetes environments, as follows:

Members of the following groups have the ability to make updates to the source code repositories for the application:

• ng-ngdc-XXX-dev
• ng-ngdc-XXX-leads
• ng-ngdc-XXX-dba

Members of the follow groups can only read the contents of the source code repositories but cannot make updates:

• ng-ngdc-XXX-ops
• ng-ngdc-XXX-biz

Member so the following groups can make changes to the Kubernetes Workspaces (while members of the other groups can view apps in these spaces):

• cg-pam-devops-XXX-dev
• cg-pam-devops-XXX-ops
• cg-pam-devops-XXX-leads

Member so the following groups can make changes to the Kubernetes Stage Hostspaces (while members of the other groups can view apps in these spaces):

• cg-pam-devops-XXX-ops
• cg-pam-devops-XXX-leads

Member so the following groups can make changes to the Kubernetes Production Hostspaces (while members of the other groups can view apps in these spaces):

• cg-pam-devops-XXX-ops

Member so the following groups can make changes to the Kubernetes Dataspaces (while members of the other groups can view data stacks in these spaces):

• cg-pam-devops-XXX-ops
• cg-pam-devops-XXX-dba

RBAC Reference Matrix:

For quick reference, see the following role-based access matrix:

AD Group	Azure DevOps
ng-ngdc-<appcode>-admin	Contributor
ng-ngdc-<appcode>-dev	Contributor
ng-ngdc-<appcode>-leads	Contributor
ng-ngdc-<appcode>-ops	Reader
ng-ngdc-<appcode>-biz	Reader
ng-ngdc-<appcode>-dba	Contributor

Access Package Reference Matrix for Kubernetes:

IMPORTANT: Each user should only request and be granted access to a single Access Package below. Requesting access to multiple access package for a single application/appcode can cause issues and will be periodically audited.

For Kubernetes access (matrix below), users will need to log into Kubernetes using their a-sso account. Users can request an a-sso acount using the A-SSO Admin Account Request form. Users will need to request access to the necessary "Access Package" by logging into the "My Access" portal using their a-sso and following the steps in KB0007629. Once approved, they can log into Kubernetes using their a-sso. Keep in mind that it might take 1-2 hours for the access to apply once approved.

The individuals who approve the following access packages are the Application Development Owner and Application Support Owner. Please note that if these application owners change, the Access Package approvers are not updated with the new owners.

WS = Workspace, HS = Hostspace, DS = Dataspace (reference)

Access Package	CG Group	WS + HS (dnt)	HS (stg)	HS (prd)	DS (all)
IT DevOps <appcode> Developers	cg-pam-devops-<appcode>-dev	Edit	View	View	View
IT DevOps <appcode> Leads	cg-pam-devops-<appcode>-leads	Edit	Edit	View	View
IT DevOps <appcode> Operations	cg-pam-devops-<appcode>-ops	Edit	Edit	Edit	Edit
IT DevOps <appcode> DBA	cg-pam-devops-<appcode>-dba	View	View	View	Edit

Note: Only users with Edit Persmission can access Secrets in Kubernetes.

CloudOne Continuous Integration / Continuous Delivery (CI/CD) Pipeline

Continuous Integration stage - Build and Quality Checks

The Continuous Integration (CI) pipeline stage is triggered by a merge request issued to the master branch of the source code repository (in Git). Upon successful completion of the build, a container image will be generated and readied for deployment into a Workspace and for subsequent deployments into downstream Hostspaces.

Even before the CI pipeline stage is initiated, certain prerequisites are managed and enforced within Azure DevOps based on Branch Policies maintained in the source code repository. The policies include:

• All comments opened against the branch must be resolved
• All work must be linked to other work (attached within a hierarchy)
• A criteria called "PR Check" must be met in order to complete a merge to the master branch – in this manner, the Release pipeline will indicate success to allow the merge to be completed
• Multiple merges are consolidated to one before the pipeline is initiated

The workflow of the CI pipeline stage is as follows:

1. The flow is triggered by the creation of a merge request into an application component's master branch in Git
2. Initial validation checks are completed to ensure certain variables are defined for the pipeline (e.g. the appCode that identifies the application and the appVersion as a valid SemVer-compliant version number see Reference below for details) and that the intended type of build output is supported by the pipeline
3. Check version to ensure it is not a duplicate of a previously deployed/released version
4. Initiate the build of the code – the specific steps to build the application will depend on the type of build output; some builds are done using Docker S2I images (source-to-image) to have predefined contained perform the code builds, while others are done using explicitly defined build commands
5. Generate Helm Chart and publish to Artifactory
6. Run automated unit tests
7. Generate the Docker image containing the base image and the application and publish to Artifactory
8. Initiate deployment into Workspace (OpenShift in AWS)
9. Initiate a code scan (e.g. using Jacoco as defined in a pom.xml file)
10. Initiate a security code scan using JFrog XRay
11. The results of both code scans are decorated to the Git pull request
12. Score from the security code scan must exceed a specific threshold to allow the build to pass, however this can be overridden by setting the "Security Scan Exempted" flag, if approved to be done; a link to the contact of the person who grants the exemption will be decorated to the Git pull request as well

Note: Exemptions are granted by the security team – A new tool is under development for use by the security team to grant the exemptions and automatically update the pipeline; this tool will be available in the CloudOne Self Assist portal

14. Run the "Complete Pull Request" task to merge the new code into the master branch in Git
15. CI pipeline stage is finished

The completion of the CI pipeline stage triggers the initiation of the deployment pipeline stages, described below.

Deployment Pipeline Stages – Continuous Delivery

The Deployment pipeline stages are triggered by the successful completion of the CI pipeline stage and will deploy the code first to the Workspace and then to all the Hostspaces , each pending a manual approval indicating completion of certain validations and tests.

The workflow of the Release pipeline is as follows:

1. Using the Helm Chart (or Kustomize overlay files), deploy the application component to the Workspace
2. Pipeline pauses and waits for completion of Unit Tests, indicated either by a status of "Resume" (to continue the pipeline) or "Reject" (indicating to NOT proceed); this approval step is done by a member of the Dev Leads Group
3. Finalize release tags and version number in the container repository
4. Using the Helm Chart (or Kustomize overlay files), deploy the application component to the Staging Hostspace(s)
5. Following UAT testing, a member of the Business Approvers group can approve the pipeline to continue
6. Update PR Check status to indicate the merge is good, triggering Git to complete the merge of the feature branch into the master branch
7. Generate an extra copy of the deployment artifact (the container image) tagged with the version number with "-PRD" appended to it. While the artifact without the -PRD appended can be overwritten, this -PRD version cannot be overwritten so any attempt to build and deploy new logic with the same version number will fail due to duplicate versioning
8. Using the Helm Chart (or Kustomize overlay files), deploy the application component to the Pre-Production Hostspace(s)
9. Following approval by a member of the Operations Group, the pipeline continues
10. Using the Helm Chart (or Kustomize overlay files), deploy the application component to the Production Hostspace or Hostspaces
11. CI/CD Pipeline is finished

Note that all deployments can be approved for immediate execution or scheduled for a time (and date) in the future.

References

• Semantic Versioning - https://semver.org/