Upgrade Redhat MongoDB (v3.6.x) to Docker based version 4.4.x - V 1

Procedure to upgrade Redhat MongoDB (v3.6.x) to Docker based version 4.4.x

Description

The current mongodb version 3.6.3 is based on redhat based image and recently redhat stopped providing the patch support for mongodb image. The mongodb version 3.6.3 has multiple critical vulnerability issues which are not able to address due to no support from Redhat. To mitigate this issue we have redesigned mongodb catalog using docker hub based image. The existing mongodb deployments which uses redhat based image (v3.6.3) need to be upgraded to use docker hub based image.

Before you upgrade a production environment, use the procedures in this document to upgrade non-production environment that reproduces your production environment, to test your application to make sure they are compatible with the new version.

Prerequisites

You need to be part of ng-ngdc--dba NG to perform database upgrade.
You need to have kubectl (or) oc binary installed on your machine, we will use kubectl or oc binary to perform few database updates part of the upgrade procedure.

Here is the link to download oc tool (This includes kubectl binary). The link to download the file will be at the end of the github page.

https://github.com/openshift/origin/releases/tag/v3.11.0

Upgrade Procedure

IMPORTANT:

Always backup all of your data before upgrading MongoDB. Use the backup pipeline to perform the backup.

The procedure for performing an in-place upgrade (where your existing data and configuration files are preserved) includes the following steps:

Here is the upgrade path for MongoDB version 3.6.3 to 4.4.11

v3.6.3 ==> v4.0.25
v4.0.25 ==> v4.2.18
v4.2.18 ==> v4.4.11

Upgrade to v4.0.x

Create the new branch for database upgrade process. For example we can create the branch named "feature/dbupgrade".
Clone the repo to your local machine and checkout the branch you have created in the previous step. For example "feature/dbupgrade"

Update azure-pipelines.yaml file to add baseImage: mongo:4.0.25 paramater under mongodb section.

extends:
 template: dbaas/mongodb-devexp-stg-2.yml@spaces
 parameters:
   appVersion: 4.1.0
   mongodb:
     podSpec:
       cpuRequests: 160m
     # Uncomment the baseImage parameter if you are upgrading from mongodb version 3.6.3
     baseImage: mongo:4.0.25
     #

Download devexp mongodb boiler-plate repo and make sure you are in release/v4.2 branch.

Here is the link to devexp boiler-plate repo

https://dev.azure.com/netapp-ngdc/devexp/_git/mongodb?version=GBrelease%2Fv4.2

Copy the following files from devexp boiler-plate repo to your database repo.

..\devexp-mongodb\templates\backup-secrets.yaml
..\devexp-mongodb\templates\configmap.yaml
..\devexp-mongodb\templates\init-users-cm.yaml
..\devexp-mongodb\templates\db-init-cm.yaml
..\devexp-mongodb\templates\secrets.yaml
..\devexp-mongodb\templates\deployment-standalone.yaml
..\devexp-mongodb\values.lint.yaml

Update values.yaml file and append the following content at the end of the file.

## New entry for docker hub image change.

## Use the below section to customize startup options.
#
mongodb:
 startupArgs:
   - "--dbpath"
   - "/var/lib/mongodb/data"
   - "--wiredTigerCacheSizeGB"
   - "1"

# configmap:
 # storage:
   # dbPath: /var/lib/mongodb/data
   # journal:
     # enabled: true
   # wiredTiger:
     # engineConfig:
       # cacheSizeGB: 1
 # net:
   # bindIpAll: true
 # security:
   # authorization: enabled
   
initContainers:
 resources: 
   limits:
     cpu: 1
     memory: 1Gi
   requests:
     cpu: 200m
     memory: 512Mi

securityContext:
 enabled: false
 uid: 10000

Or you can also refer the values.yaml file from devexp boiler-plate repo and copy the content below the following comment section
"## New entry for docker hub image change" and paste the copied content to your values.yaml file.

..\devexp-mongodb\values.yaml

If your dataspace is deployed in OpenShift please make sure to enable securityContext and update the current uid in values.yaml

securityContext:
 enabled: true
 uid: 10000

Also update values.workspace.yaml and append the following content at the end.

securityContext:
 enabled: false
 uid: 10000

Run the following command to get the current UID from openshift.

kubectl -n <dataspace name> get pods --selector="db-type=mongodb" --output=jsonpath={.items[].spec.containers[].securityContext.runAsUser}

(or)

oc -n <dataspace name> get pods --selector="db-type=mongodb" --output=jsonpath={.items[].spec.containers[].securityContext.runAsUser}

Commit and push the changes to your repo.
Initiate the pipeline using the branch you have created for upgrade activity.
Wait for pipeline process to complete the database deployment.
If there is no errors in the pipeline process then your database upgrade completed successfuly. Your application database is running in version v4.0.25, but featureCompatibilityVersion is still pointing to v3.6 which we need to update to v4.0

Run the following database command to check the current featureCompatibilityVersion set on database by terminal into database pod.

/usr/bin/mongo -u admin -p"$MONGODB_ADMIN_PASSWORD" --quiet --eval "db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )"

Run the following database command to update the featureCompatibilityVersion to 4.0 by terminal into database pod.

/usr/bin/mongo -u admin -p"$MONGODB_ADMIN_PASSWORD" --quiet --eval "db.adminCommand( { setFeatureCompatibilityVersion: '4.0' } )"

Upgrade to v4.2.x

It's time to upgrade the database to v4.2.x. To start the upgrade, update azure-pipelines.yaml file baseImage to 4.2.18 baseImage: mongo:4.2.18 paramater under mongodb section.

extends:
 template: dbaas/mongodb-devexp-stg-2.yml@spaces
 parameters:
   appVersion: 4.1.0
   mongodb:
     podSpec:
       cpuRequests: 160m
     # Uncomment the baseImage parameter if you are upgrading from mongodb version 3.6.3
     baseImage: mongo:4.2.18
     #

Commit and push the changes to your repo.
Initiate the pipeline using the branch you have created for upgrade activity.
Wait for pipeline process to complete the database deployment.
If there is no errors in the pipeline process then your database upgrade completed successfuly. Your application database is running in version v4.2.18, but featureCompatibilityVersion is still pointing to v4.0 which we need to update to v4.2

Run the following database command to update the featureCompatibilityVersion to 4.2 by terminal into database pod.

/usr/bin/mongo -u admin -p"$MONGODB_ADMIN_PASSWORD" --quiet --eval "db.adminCommand( { setFeatureCompatibilityVersion: '4.2' } )"

Upgrade to v4.4.x

It's time to upgrade the database to v4.4.x. To start the upgrade, update azure-pipelines.yaml file baseImage to 4.4.11 baseImage: mongo:4.4.11 paramater under mongodb section.

extends:
 template: dbaas/mongodb-devexp-stg-2.yml@spaces
 parameters:
   appVersion: 4.1.0
   mongodb:
     podSpec:
       cpuRequests: 160m
     # Uncomment the baseImage parameter if you are upgrading from mongodb version 3.6.3
     baseImage: mongo:4.4.11
     #

Commit and push the changes to your repo.
Initiate the pipeline using the branch you have created for upgrade activity.
Wait for pipeline process to complete the database deployment.
If there is no errors in the pipeline process then your database upgrade completed successfuly. Your application database is running in version v4.4.11, but featureCompatibilityVersion is still pointing to v4.2 which we need to update to v4.4

Run the following database command to update the featureCompatibilityVersion to 4.4 by terminal into database pod.

/usr/bin/mongo -u admin -p"$MONGODB_ADMIN_PASSWORD" --quiet --eval "db.adminCommand( { setFeatureCompatibilityVersion: '4.4' } )"

Troubleshooting

The mongod deamon in the pod logs to the standard output, so the log is available in the pod log. The log can be examined by running:

oc logs <pod>
(or)
kubectl logs <pod>

CONTENTS