Migrate a database cluster to Mission Control

Enterprises with existing database clusters can use the Mission Control Runtime Installer to migrate to an environment running Mission Control. Benefits include:

Zero downtime.
Minimal configuration changes.
Keeping nodes on the cluster in sync during the migration.
Use of Mission Control for next generation cluster management.

Mission Control offers two migration approaches:

In-place migration: Keep your existing hardware and data locations while migrating your cluster.
Non-in-place migration: Move to new hardware or infrastructure while maintaining cluster operations.

With in-place migration, you can manage existing clusters without moving data to new hardware. With non-in-place migration, you gain flexibility to move to new infrastructure.

These instructions work only for migrating in place and require the runtime-based installation.

Migrate an existing database cluster to Mission Control

The migration process ensures zero downtime and preserves all your data and cluster configurations.

The migration process consists of three phases: import init, import add, and import commit.

The import init phase validates the cluster and gathers the necessary configuration data to determine if the migration is possible. It also creates the necessary seed services, creates the project for the cluster, and sets up related metadata so that migrated nodes can continue communicating with non-migrated ones while keeping the cluster healthy during the process. Run the import init phase only once on the first DSE node.
The import add phase runs on every DSE node in the datacenter. The import add command performs the per-node migration, using the configuration fetched in the import init phase. import add supports running in parallel and takes care of serializing the migrations.
The final import commit phase requires that the import add phase be run on every node in the cluster. It creates the necessary Kubernetes configurations. It activates all the migrated nodes to Mission Control and then reconciles them to the desired state. After achieving this state, the cluster and all of its nodes should be managed using Mission Control.

As you step through these phases, here are some useful CLI flags for reference:

--nodetool-path: Where the nodetool command is located. This assumes a complete path including the nodetool executable. cqlsh must reside in the same path.
--nodetool status: Describes status and information about nodes in a datacenter.
--cass-config-dir: Directory where cassandra.yaml is located.
--dse-config-dir: Directory where dse.yaml is located.
--hcd-config-dir: Directory where hcd.yaml is located.
--cassandra-home: Directory where DSE is installed.
--username: DSE admin account name.
--password: DSE admin password.
--jmxusername: Local JMX authentication username.
--jmxpassword: Local JMX authentication password.
--kubeconfig: Path to the kubeconfig file. If not specified, the default kubeconfig file is used.
--namespace/-n: The project/namespace in which the cluster is imported/located. If not specified, the default namespace is used.

`import init` phase

The import init phase verifies that the cluster is ready to be migrated. It must be run from a node in the cluster because it connects to the running DSE node to fetch its state. JMX access credentials are required because this phase uses nodetool. This node’s configuration serves as the base for all other nodes in the managed cluster. Choose a node with a good configuration to use as a generic base.

It is possible to review the configuration (or a failed run), modify the setup, and rerun the import init command. Rerunning import init replaces the values previously written to the cluster.

If the cluster is using authentication and you wish to use existing superuser username/password combination for communication, then enter them using --username and --password parameters. When no credentials are given, Mission Control creates and instates its own superuser to the cluster. Likewise, if the JMX local authentication is enabled, this Mission Control migration tool requires the JMX user account and password.

Run the import init command:

mcctl import init -n dse

`import add` phase

The import add command migrates each node individually, using the configuration from the import init phase. You can run import add in parallel as it handles the migration serialization automatically.

The import add command needs access to modify the data directories that the running DSE instance uses. You must unify ownership and groups of directories before migration. When running under Mission Control, the nodes might use different user IDs or group IDs than the existing deployment, which requires modification during migration.

The import add command supports and requires the same parameters as does the import init command if the installation is using a custom path instead of auto-detected paths.

When jmxpassword/jmxusername/username/password parameters are not available, the values used in the import init phase are reused in the import add phase. If a .deb/.rpm package installation was used, the configuration and tool paths are using the defaults provided by the package manager. In some cases these values might not match values in use, and might require overriding.

To aid in detection, the import add command uses the DSE_HOME environment variable. If either the cassandra-home or the DSE_HOME environment variable is set, Mission Control tries to detect cassandra.yaml, dse.yaml, and nodetool-path from its subdirectories.

Run the import add command:
```
sudo mcctl import add -n dse
```
The import add command must be run on each node individually, one at a time.

For each node, it will:
- Stop the existing DSE process.
- Create a new pod in the Kubernetes cluster.
- Start the node under Mission Control management.
  
  This process should be done sequentially - one node at a time - to ensure a safe migration while keeping the cluster operational.

Verify by running:

nodetool status

and

kubectl get pods -n dse

Result

NAME                     READY   STATUS    RESTARTS   AGE
dse-node-1              1/1     Running   0          2h
dse-node-2              1/1     Running   0          2h
dse-node-3              1/1     Running   0          2h

Reverse the operation

At this point, the operation is reversible. This might be required if certain prerequisites are not met on every node and a particular node requires certain operations. The existing DSE cluster installation remains intact, but you must remove the migrated node before continuing.

Delete the pod which this node created by running:
```
kubectl delete pod/POD_NAME -n dse
```
Replace POD_NAME with the name of the pod.
List and then delete all the PersistentVolumeClaim (PVC)s and PersistentVolume (PV)s that this node mounted:
```
kubectl get pvc -n dse
kubectl delete pvc PVC_NAME -n dse

kubectl get pv |grep "pvc-server-data"
kubectl delete pv PV_NAME
```
Replace the following:
- PVC_NAME: The name of the PVC
- PV_NAME: The name of the PV

Modify the ClusterConfig file:

kubectl edit clusterconfig cluster-migrate-config -n dse

Set the migrated field to false instead of true.
Restart the node with OpsCenter.

`import commit` phase

Before running the final import commit phase, you must run the import add phase on every node in the cluster. The import commit phase creates the necessary Kubernetes configurations, activates all migrated nodes in Mission Control, and reconciles them to the desired state.

Run the import commit command:
```
mcctl import commit -n dse
```
The import commit command creates the MissionControlCluster (mccluster in short) object and finalizes the migration.

Verify by running:

kubectl get mccluster -n dse

Result

NAMESPACE   NAME      AGE
dse         dsetest   73m

At the completion of these three phases the DSE cluster is fully migrated to Mission Control with zero down time. After you complete all three migration phases, Mission Control fully manages your DSE cluster. The migration maintains zero downtime while preserving all your data and cluster configurations.

Before you begin

Before starting the migration process:

Back up your entire DSE cluster.
Document your current cluster configuration.
Plan for the migration during a maintenance window.
Ensure you have sufficient disk space for the Mission Control runtime.

Prerequisites and system requirements

A prepared environment. See Mission Control Runtime Installer.
Each node must have the same storage group rights in the DSE data directories (ownership).
- Group read-write access must be given to DSE directories or the migration cannot continue. Mission Control changes file system groups and write access during migration.
Download mcctl:
- For x86_64 Linux
- For arm64 Linux
User running Mission Control mcctl commands must have access to the DSE’s data directories (running with sudo -u is an acceptable solution).
Linux is the only supported platform.
Only DSE server versions 6.8.26 and later are supported. Check your version with:
```
nodetool version
```

Result

ReleaseVersion: 6.8.26

Migration approaches

Mission Control provides two methods for migrating your DSE cluster to Mission Control. Choose the approach that best fits your infrastructure needs.

In-place migration
Non-in-place migration

Use this approach when you want to maintain your current hardware infrastructure. This method migrates your cluster while keeping the existing hardware and data locations.

Prerequisites

All nodes must have the same storage group rights in the DSE data directories.
Each node must have sufficient disk space for the Mission Control runtime.
The cluster must be healthy and running before migration.

To migrate your DSE cluster to Mission Control, do the following:

Install the Mission Control runtime on the node.
Stop the existing database process:
```
sudo systemctl stop dse
```
Mount the existing data directory as a persistent volume:
```
kubectl apply -f **PV_YAML_FILE.yaml**
```
Replace PV_YAML_FILE.yaml with the path to the persistent volume YAML file.
Start a pod on the local runtime that uses the new persistent volume:
```
kubectl apply -f **POD_YAML_FILE.yaml**
```
Replace POD_YAML_FILE.yaml with the path to the pod YAML file.
Verify the pod is running and the node is healthy:
```
kubectl get pods -n NAMESPACE
```
Replace NAMESPACE with the namespace of the DSE cluster.

Verify the node is healthy:

nodetool status

Result

Datacenter: datacenter1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.0.1       256.67 KB  256          100.0%            aaa1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.0.2       245.32 KB  256          100.0%            bbb1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.0.3       267.89 KB  256          100.0%            ccc1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1

Use this approach when you want to migrate to new hardware or infrastructure. This method creates a new cluster and gradually moves data and traffic to it.

Prerequisites

All keyspaces must use the NetworkTopologyStrategy replication strategy
New hardware must meet Mission Control system requirements
Sufficient network bandwidth between old and new clusters

To migrate your DSE cluster to Mission Control, do the following:

Validate cluster configuration:

# Check keyspace replication strategies
cqlsh -e "SELECT keyspace_name, replication FROM system_schema.keyspaces;"

Result

 keyspace_name | replication
---------------+----------------------------------------------------------
 system_auth   | {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}
 system_schema | {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}
 system        | {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}
 my_keyspace   | {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}

Verify cluster health:

nodetool status

Result

Datacenter: datacenter1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.0.1       256.67 KB  256          100.0%            aaa1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.0.2       245.32 KB  256          100.0%            bbb1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.0.3       267.89 KB  256          100.0%            ccc1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1

Datacenter: new_datacenter
=========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.1.1       0 KB       256          0.0%              ddd1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.1.2       0 KB       256          0.0%              eee1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1
UN  10.0.1.3       0 KB       256          0.0%              fff1b7c1-6049-4a08-a3e0-977baf4cfe4f  rack1

Create a new cluster in Mission Control:
```
mcctl cluster create --name **CLUSTER_NAME** --datacenter **NEW_DC_NAME** --seeds **EXISTING_SEED_NODES** -n **NAMESPACE**
```
Replace the following:
- CLUSTER_NAME: Your existing DSE cluster name
- NEW_DC_NAME: The name for your new datacenter
- EXISTING_SEED_NODES: Comma-separated IP addresses of your existing seed nodes
- NAMESPACE: The name of your Mission Control project, which is also the Kubernetes namespace
Monitor new datacenter bootstrapping:
```
nodetool status
```
Watch for the new nodes to join the cluster and reach the Up Normal UN state.

Verify connectivity between datacenters:

nodetool status

Result

Address         DC          Rack        Status State   Load            Owns                Token
                                                                                         -9213129637601350416
10.0.0.1       datacenter1 rack1       Up     Normal   256.67 KB       33.33%              -9213129637601350416
10.0.0.2       datacenter1 rack1       Up     Normal   245.32 KB       33.33%              -4606564818800675208
10.0.0.3       datacenter1 rack1       Up     Normal   267.89 KB       33.33%              0
10.0.1.1       new_datacenter rack1    Up     Normal   0 KB            0.00%               4606564818800675208
10.0.1.2       new_datacenter rack1    Up     Normal   0 KB            0.00%               9213129637601350416
10.0.1.3       new_datacenter rack1    Up     Normal   0 KB            0.00%               13819694456402035624

Modify the keyspace replication to include new datacenter:
```
cqlsh -e "ALTER KEYSPACE **KEYSPACE_NAME** WITH replication = {'class': 'NetworkTopologyStrategy', '**OLD_DC_NAME**': **REPLICATION_FACTOR**, '**NEW_DC_NAME**': **REPLICATION_FACTOR**};"
```
Replace the following:
- KEYSPACE_NAME: The name of the keyspace you are modifying
- OLD_DC_NAME: Your existing datacenter name
- NEW_DC_NAME: Your new datacenter name
- REPLICATION_FACTOR: The number of replicas (typically three)

Run rebuild on all nodes:

nodetool rebuild **OLD_DC_NAME**

Replace OLD_DC_NAME with your existing datacenter name. Run this command on each new node.

You must run the rebuild command on ALL nodes in the new datacenter.
This process will create significant disk I/O pressure on ALL nodes in the old datacenter as they stream data to the new nodes.
Monitor disk space and I/O performance during this phase.

Monitor streaming progress:

nodetool netstats | grep -v 100%

Result

Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed   Dropped
Large messages                  n/a         0             0         0
Small messages                  n/a         0             0         0
Gossip messages                 n/a         0             0         0

Optional: Watch the command for a live progress report:
```
watch nodetool netstats | grep -v 100%
```
Update application connection strings to point to new datacenter nodes:
```
# Example: **NEW_NODE_IP**:9042
```
Replace NEW_NODE_IP with the IP address of a node in your new datacenter. Update this in your application configuration.
Update the keyspace replication to only use the new datacenter:
```
cqlsh -e "ALTER KEYSPACE **KEYSPACE_NAME** WITH replication = {'class': 'NetworkTopologyStrategy', '**NEW_DC_NAME**': **REPLICATION_FACTOR**};"
```
Replace the following:
- KEYSPACE_NAME: The name of the keyspace you are modifying
- NEW_DC_NAME: Your new datacenter name
- REPLICATION_FACTOR: The number of replicas (typically 3)
Remove old datacenter seed references:
```
mcctl cluster update --name **CLUSTER_NAME** --remove-seeds **OLD_SEED_NODES** -n **NAMESPACE**
```
Replace the following:
- CLUSTER_NAME: Your cluster name
- OLD_SEED_NODES: Comma-separated IP addresses of the old datacenter seed nodes
- NAMESPACE: Your Kubernetes namespace
Stop the old datacenter nodes:
```
sudo systemctl stop SERVICE_NAME
```
Replace SERVICE_NAME with the name of the database service, for example hcd, dse, or cassandra.
Run this command on each node in the old datacenter to stop the database service.

Since all data and traffic has been moved to the new datacenter, you can safely stop the old nodes. No decommissioning is necessary as the cluster is already operating fully on the new datacenter.

Verify cluster health:
```
nodetool status
nodetool info
```
Check that all nodes are in the Up Normal UN state and verify cluster metrics.
Monitor cluster metrics and performance.

Troubleshoot

If you encounter issues during migration, refer to the following common issues and solutions.

Issue Solution

Issue	Solution
`nodetool status` shows nodes as Down	Check if the node’s pod is running: `kubectl get pods -n SERVICE_NAME` Replace `SERVICE_NAME` with the name of the service.
Verify network connectivity between nodes	Check pod logs: `kubectl logs POD_NAME -n SERVICE_NAME` Replace `POD_NAME` with the name of the pod and `SERVICE_NAME` with the name of the service.
Migration fails during `import add`	Verify storage permissions: `ls -la /var/lib/cassandra`
Check if the node has sufficient disk space: `df -h`	Review pod events: `kubectl describe pod POD_NAME -n SERVICE_NAME` Replace `POD_NAME` with the name of the pod and `SERVICE_NAME` with the name of the service.
Cluster health issues after migration	Verify all nodes are in Up Normal state: `nodetool status`
Check cluster metrics in Mission Control dashboard	Review system logs: `kubectl logs POD_NAME -n SERVICE_NAME` Replace `POD_NAME` with the name of the pod and `SERVICE_NAME` with the name of the service.

nodetool status shows nodes as Down

Check if the node’s pod is running: kubectl get pods -n SERVICE_NAME Replace SERVICE_NAME with the name of the service.

Verify network connectivity between nodes

Check pod logs: kubectl logs POD_NAME -n SERVICE_NAME Replace POD_NAME with the name of the pod and SERVICE_NAME with the name of the service.

Migration fails during import add

Verify storage permissions: ls -la /var/lib/cassandra

Check if the node has sufficient disk space: df -h

Review pod events: kubectl describe pod POD_NAME -n SERVICE_NAME Replace POD_NAME with the name of the pod and SERVICE_NAME with the name of the service.

Cluster health issues after migration

Verify all nodes are in Up Normal state: nodetool status

Check cluster metrics in Mission Control dashboard

Review system logs: kubectl logs POD_NAME -n SERVICE_NAME Replace POD_NAME with the name of the pod and SERVICE_NAME with the name of the service.

Migrate a database cluster to Mission Control

Migrate an existing database cluster to Mission Control

`import init` phase

`import add` phase

Reverse the operation

`import commit` phase

Before you begin

Prerequisites and system requirements

Migration approaches

Troubleshoot

Was this helpful?

Give Feedback