ExaCC: Create a VM Cluster Network using OCI CLI

This week I got a new Exadata Cloud@Customer X9-M quarter rack, and I thought it would be a good opportunity to showcase how to create a VM Cluster Network using OCI CLI. I am currently using version 3.25.3.

The first thing to do is to create the JSON template

Rene@eclipsyss-mbp Documents % oci db vm-cluster-network create --generate-full-command-json-input > exarenedbntwrk01.json

Once I have generated the JSON file, I need the compartment, the Exadata OCID, and the compute nodes that we will be assigning an IP to during this process, so I will get these and put them in a variable. For my example, I will be putting them in the following variables: compartment_id and exadata_infrastructure_id

Rene@eclipsyss-mbp Documents % oci iam compartment list --query "data[?\"lifecycle-state\" == 'ACTIVE'].{Name:name,OCID:id,\"Parent OCID\":\"compartment-id\"}" --output table --include-root
+---------------------------+-------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+
| Name                      | OCID                                                                                | Parent OCID                                                                     |
+---------------------------+-------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+
| reneantunezace            | ocid1.tenancy.oc1..aaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb     | None                                                                            |
| reneaceiamblog            | ocid1.compartment.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | ocid1.tenancy.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
| ManagedCompartmentForPaaS | ocid1.compartment.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | ocid1.tenancy.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
+---------------------------+-------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+
Rene@eclipsyss-mbp Documents % export compartment_id=`oci iam compartment list --query "data[?name == 'reneaceiamblog'].{OCID:id}" | jq '.[].OCID' | tr -d '"' `
Rene@eclipsyss-mbp Documents % oci db exadata-infrastructure list --compartment-id $compartment_id --query "data[*].{Name:\"display-name\",OCID:id}" --output table
+--------+-----------------------------------------------------------------------------------------------------------+
| Name   | OCID                                                                                                      |
+--------+-----------------------------------------------------------------------------------------------------------+
| ExaCC1 | ocid1.exadatainfrastructure.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
+--------+-----------------------------------------------------------------------------------------------------------+
Rene@eclipsyss-mbp Documents % export exadata_infrastructure_id=`oci db exadata-infrastructure list --compartment-id $compartment_id --query "data[?\"display-name\" == 'ExaCC1'].{OCID:id}" | jq '.[].OCID' | tr -d '"' `
Rene@eclipsyss-mbp Documents % oci db db-server list --query "data[*].{Name:\"display-name\",OCID:id}" --output table --compartment-id $compartment_id --exadata-infrastructure-id $exadata_infrastructure_id
+------------+----------------------------------------------------------------------------------------------+
| Name       | OCID                                                                                         |
+------------+----------------------------------------------------------------------------------------------+
| dbServer-1 | ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1 |
| dbServer-2 | ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2 |
+------------+----------------------------------------------------------------------------------------------+

Once I have obtained the information above, I fill the JSON template that we got at the beginning of this blog, below is an example as I have masked several pieces of information

Rene@eclipsyss-mbp Documents % cat exarenedbntwrk01.json
{
   "compartmentId": "ocid1.compartment.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "definedTags": {},
   "displayName": "exarenedbntwrk01",
   "dns": [
      "1XX.1XX.1XX.1XX",
      "1XX.1XX.1XX.1XX"
   ],
   "exadataInfrastructureId": "ocid1.exadatainfrastructure.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "freeformTags": {},
   "maxWaitSeconds": 0,
   "ntp": [
      "1XX.1XX.1XX.1XX",
      "1XX.1XX.1XX.1XX"
   ],
   "scans": [
      {
         "hostname": "exarenedb-scan",
         "ips": [
            "1XX.1XX.1XX.1XX",
            "1XX.1XX.1XX.1XX",
            "1XX.1XX.1XX.1XX"
         ],
         "port": 1521,
         "scanListenerPortTcp": 1521,
         "scanListenerPortTcpSsl": null
      }
   ],
   "vmNetworks": [
      {
         "domainName": "reneace.com",
         "gateway": "1XX.1XX.1XX.1XX",
         "netmask": "255.255.255.0",
         "networkType": "BACKUP",
         "nodes": [
            {
               "dbServerId": "ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
               "hostname": "exarenedb1-bk",
               "ip": "1XX.1XX.1XX.1XX",
               "lifecycleState": "CREATING",
               "vip": null,
               "vipHostname": null
            },
            {
               "dbServerId": "ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
               "hostname": "exarenedb2-bk",
               "ip": "1XX.1XX.1XX.1XX",
               "lifecycleState": "CREATING",
               "vip": null,
               "vipHostname": null
            }
         ],
         "vlanId": "101"
      },
      {
         "domainName": "reneace.com",
         "gateway": "1XX.1XX.1XX.1XX",
         "netmask": "255.255.255.0",
         "networkType": "CLIENT",
         "nodes": [
            {
               "dbServerId": "ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", 
               "hostname": "exarenedb1",
               "ip": "1XX.1XX.1XX.1XX",
               "lifecycleState": "CREATING",
               "vip": "1XX.1XX.1XX.1XX",
               "vipHostname": "exarenedb1-vip"
            },
            {
               "dbServerId": "ocid1.dbserver.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
               "hostname": "exarenedb2",
               "ip": "1XX.1XX.1XX.1XX",
               "lifecycleState": "CREATING",
               "vip": "1XX.1XX.1XX.1XX",
               "vipHostname": "exarenedb2-vip"
            }
         ],
         "vlanId": "201"
      }
   ],
   "waitForState": [
      "CREATING"
   ],
   "waitIntervalSeconds": 0
}

Now that I have the JSON filled up, I will now create the VM. The output you see below is redacted, but it does return with an error : get_vm_cluster_network() missing 1 required positional argument: ‘vm_cluster_network_id’. 

But I think it is a bug with OCI CLI, as the network does get created properly in the console.

Rene@eclipsyss-mbp Documents % oci db vm-cluster-network create --from-json "file:///Users/Rene/Documents/exarenedbntwrk01.json"
Action completed. Waiting until the resource has entered state: ('CREATING',)
Encountered error while waiting for resource to enter the specified state. Outputting last known resource state
{
   "data": {
      "compartment-id": "ocid1.compartment.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
      ...
   },
   "etag": "4486beac",
   "opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
TypeError: get_vm_cluster_network() missing 1 required positional argument: 'vm_cluster_network_id'

Now that the VM has been created, I query to see the status and wait until the status is REQUIRES_VALIDATION

Rene@eclipsyss-mbp Documents % oci db vm-cluster-network list --query "data[?\"lifecycle-state\" == 'CREATING'].{name:\"display-name\",OCID:id}" --output table --compartment-id $compartment_id --exadata-infrastructure-id $exadata_infrastructure_id --all
+------------------------------------------------------------------------------------------------------+------------------+
| OCID                                                                                                 | name             |
+------------------------------------------------------------------------------------------------------+------------------+
| ocid1.vmclusternetwork.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | exarenedbntwrk01 |
+------------------------------------------------------------------------------------------------------+------------------+
Rene@eclipsyss-mbp Documents % oci db vm-cluster-network list --query "data[?\"lifecycle-state\" == 'REQUIRES_VALIDATION'].{name:\"display-name\",OCID:id}" --output table --compartment-id $compartment_id --exadata-infrastructure-id $exadata_infrastructure_id --all
+------------------------------------------------------------------------------------------------------+------------------+
| OCID                                                                                                 | name             |
+------------------------------------------------------------------------------------------------------+------------------+
| ocid1.vmclusternetwork.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | exarenedbntwrk01 |
+------------------------------------------------------------------------------------------------------+------------------+

I will now run the validation of the VM, I will see the same error as when creating the network, but still, the network gets validated.

Rene@eclipsyss-mbp Documents % oci db vm-cluster-network validate --exadata-infrastructure-id $exadata_infrastructure_id \
--vm-cluster-network-id $vm_cluster_network_id \
--wait-for-state VALIDATED --wait-for-state VALIDATION_FAILED
Action completed. Waiting until the resource has entered state: ('VALIDATED', 'VALIDATION_FAILED')
Encountered error while waiting for resource to enter the specified state. Outputting last known resource state
{
   "data": {
      "compartment-id": "ocid1.compartment.oc1..aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
... "etag": "90947197",
   "opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
TypeError: get_vm_cluster_network() missing 1 required positional argument: 'vm_cluster_network_id'
Rene@eclipsyss-mbp Documents % oci db vm-cluster-network list --query "data[?\"lifecycle-state\" == 'VALIDATED'].{name:\"display-name\",OCID:id}" --output table --compartment-id $compartment_id --exadata-infrastructure-id $exadata_infrastructure_id --all
+------------------------------------------------------------------------------------------------------+------------------+
| OCID                                                                                                 | name             |
+------------------------------------------------------------------------------------------------------+------------------+
| ocid1.vmclusternetwork.oc1.ca-toronto-1.aaaaaaaaXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | exarenedbntwrk01 |
+------------------------------------------------------------------------------------------------------+------------------+

Hope this small blog post helps you on creating a VM Cluster Network and gives you ideas on how to automate this process.