For more clarity, I am posting an extract of Doc ID 2488326.1 below.
This grabbed my attention when I saw the word ‘production ready’ because RAC on Docker can be heavily used in the future when it was only certified for non-production purposes. Therefore, I thought of getting my hands dirty on deploying Oracle 21c RAC in Docker.
When it comes to Oracle deployment on Docker, mainly we have two deployment scenarios.
- Option 1: Deploy Oracle RAC Database containers on a Single Docker Host
- Option 2: Deploy Oracle RAC Database containers on Multiple Docker Hosts
Both options can be further subdivided based on how you are going to distribute Docker containers. However, I will be considering only option 1 at this moment. Below is a brief explanation of how the architecture of option 1 is going to work.
Deploy Oracle RAC Database on a Single Docker Host:
RAC database nodes are deployed on multiple Docker containers within the single Docker host. Even though this model is easy to deploy, it is vulnerable to a single point of failure if the underline docker host (Bare Metal or VM Server) is hit by any hardware or software failure. Therefore, this deployment model should not be used for production database workloads on Docker.
The following diagram illustrates Oracle RAC deployment on a single Docker host.
In this deployment architecture, client connections can be made to the database in two ways.
If the docker network is configured using Macvlan driver (or using any alternative driver), Docker container IPs can be exposed to the physical network so that the client/application can communicate with database SCAN IPs directly.
If the deployment is on a host with a container-only IP address (Docker is running on a standard bridge network) you can use Oracle Connection Manager (another docker container) to access the RAC database from the client/application. In this scenario, Oracle Connection Manager acts as a database proxy server.
In this demo, I will be using Oracle-provided scripts on GitHub with some changes to them.
I will be deploying an Oracle-provided lightweight DNS server for RAC, two Oracle RAC nodes, and Oracle Connection Manager as Docker containers in a single host. Therefore, we would require a considerable amount of server resources on the Docker host. I’ll be using a VM server hosted in Oracle Cloud Infrastructure with the below configuration.
Hardware requirement:
| CPU | 4 |
| Memory | 60 GB |
| Boot Volume (OS file system) | 150 GB |
| Block Volume1 (/dev/asm_disk1) | 60 GB |
| Block Volume2 (/dev/asm_disk2) | 50 GB |
Software requirement:
- Oracle Grid Infrastructure Release 21c (21.3 or later release updates)
- Oracle Database Release 21c (21.3 or later)
- Oracle Database Client 19c (19.3)
- Oracle Container Runtime for Docker Release 18.09 or later
- Oracle Linux 7-slim Docker image (oraclelinux:7-slim)
- Oracle Linux for Docker host on Oracle Linux 7.4 (Linux-x86-64) or later updates.
Unbreakable Enterprise Kernel Release 5 (UEKR5), Unbreakable Enterprise Kernel 6 (UEKR6), and their updates. Refer to Oracle Linux Oracle Container Runtime for Docker User’s Guide for the supported kernel versions. (https://docs.oracle.com/en/operating-systems/oracle-linux/docker/docker-install.html)
In this example, I use Oracle Linux 7.9 (Linux-x86-64) with the Unbreakable Enterprise Kernel 5.4.17-2136.301.1.4.el7uek.x86_64.
We need to plan private and public networks for RAC containers before starting the installation. I will be using the 10.0.20.0/24 network segment for the public network and named it rac_pub_nw1. For private interconnect, I will be using the 192.168.10.0/24 network under the name rac_priv_nw1.
Preparing the Docker Host:
Prerequisite: Oracle Linux 7.9 is already installed. Two disks are attached for ASM. In my server /dev/sdb and /dev/sdc will be used as Oracle ASM disks.
I will be starting by installing Oracle Container Runtime for Docker as follows.
[root@oradockerhost2 ~]# yum update -y [root@oradockerhost2 ~]# yum install -y docker-engine docker-cli [root@oradockerhost2 ~]# systemctl enable --now docker [root@oradockerhost2 ~]# systemctl status docker [root@oradockerhost2 ~]# docker info
After installing Docker you must enable Open Container Initiative (OCI) runtime capabilities in the Docker daemon and container. This is because Oracle RAC needs to run certain processes in real-time mode. This is how you do it.
Log in as root, and using your preferred editor /usr/lib/systemd/system/docker.service and append the following parameter setting for the line starting with entry ExecStart= under
the [Service] section:
–cpu-rt-runtime=950000
After appending the parameter, the ExecStart value should appear similar to the following example:
Run the following commands to reload the daemon and container with the changes:
[root@oradockerhost2 ~]# systemctl daemon-reload [root@oradockerhost2 ~]# systemctl stop docker [root@oradockerhost2 ~]# systemctl start docker [root@oradockerhost2 ~]# systemctl status docker
Setting up Docker Network:
The below steps show how to create rac_pub_nw1 for the public network (10.0.20.0/24) and create rac_priv_nw1 (192.168.10.0/24) for the private network. You can use any network subnet that is suitable for your deployment.
[root@oradockerhost2 ~]# docker network ls [root@oradockerhost2 ~]# docker network create --driver=bridge --subnet=10.0.20.0/24 rac_pub_nw1 [root@oradockerhost2 ~]# docker network create --driver=bridge --subnet=192.168.10.0/24 rac_priv_nw1
(Optional) If you plan to use the Macvlan driver here are the equivalent commands.
docker network create -d macvlan --subnet=10.0.20.0/24 --gateway=10.0.20.1 -o parent=ens5 rac_pub_nw1 docker network create -d macvlan --subnet=192.168.10.0/24 --gateway=192.168.10.1 -o parent=ens6 rac_priv_nw1
Setting up Docker for Oracle RAC is almost done. Now let’s do some additional configuration on the docker host.
This is a non-production deployment so I’m going to disable the internal Linux firewall for easiness.
[root@oradockerhost2 ~]# systemctl stop firewalld.service [root@oradockerhost2 ~]# systemctl disable firewalld.service Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
(Optional) If your server does not have enough swap space increase as below. It will increase the swap by 8GB in my case.
[root@oradockerhost2 ~]# dd if=/dev/zero of=/swapfile1 bs=1024 count=8388608 [root@oradockerhost2 ~]# chown root:root /swapfile1; chmod 0600 /swapfile1; mkswap /swapfile1; swapon /swapfile1 Make it persistent throughout reboot by adding to fstab. [root@oradockerhost2 ~]# vim /etc/fstab /swapfile1 none swap sw 0 0
To ensure that your kernel resource allocation is adequate you must set the following parameters at the host level in /etc/sysctl.conf
[root@oradockerhost2 ~]# vim /etc/sysctl.conf fs.aio-max-nr=1048576 fs.file-max = 6815744 net.core.rmem_max = 4194304 net.core.rmem_default = 262144 net.core.wmem_max = 1048576 net.core.wmem_default = 262144 net.core.rmem_default = 262144 vm.nr_hugepages=16384 [root@oradockerhost2 ~]# sysctl -a ; sysctl -p
Install git and clone Oracle repository:
We will be using Oracle-provided scripts for docker deployment. Hence, we must also install Git and clone the Oracle repository for dockers.
[root@oradockerhost2 ~]# yum install -y git [root@oradockerhost2 ~]# mkdir /scratch ; cd /scratch [root@oradockerhost2 ~]# git clone https://github.com/oracle/docker-images.git
Now you should have a cloned repository like the one below.
Configure DNS server for RAC:
Now I’m going to do a few modifications to the DNS server configuration scripts according to my network setup.
[root@oradockerhost2 ~]# cd /scratch/docker-images/OracleDatabase/RAC/OracleDNSServer/dockerfiles/latest
Back up the existing zone file and create a new file as below.
[root@oradockerhost2 latest]# cat zonefile $TTL 86400 @ IN SOA ###DOMAIN_NAME###. root ( 2014090401 ; serial 3600 ; refresh 1800 ; retry 604800 ; expire 86400 ) ; minimum ; Name server's IN NS ###DOMAIN_NAME###. ; Name server hostname to IP resolve. IN A ###RAC_DNS_SERVER_IP### ; Hosts in this Domain ###HOSTNAME### IN A ###RAC_DNS_SERVER_IP### ###RAC_NODE_NAME_PREFIX###1 IN A ###RAC_PUBLIC_SUBNET###.151 ###RAC_NODE_NAME_PREFIX###2 IN A ###RAC_PUBLIC_SUBNET###.152 ###RAC_NODE_NAME_PREFIX###3 IN A ###RAC_PUBLIC_SUBNET###.153 ###RAC_NODE_NAME_PREFIX###4 IN A ###RAC_PUBLIC_SUBNET###.154 ###RAC_NODE_NAME_PREFIX###1-vip IN A ###RAC_PUBLIC_SUBNET###.161 ###RAC_NODE_NAME_PREFIX###2-vip IN A ###RAC_PUBLIC_SUBNET###.162 ###RAC_NODE_NAME_PREFIX###3-vip IN A ###RAC_PUBLIC_SUBNET###.163 ###RAC_NODE_NAME_PREFIX###4-vip IN A ###RAC_PUBLIC_SUBNET###.164 ###RAC_NODE_NAME_PREFIX###-scan IN A ###RAC_PUBLIC_SUBNET###.171 ###RAC_NODE_NAME_PREFIX###-scan IN A ###RAC_PUBLIC_SUBNET###.172 ###RAC_NODE_NAME_PREFIX###-scan IN A ###RAC_PUBLIC_SUBNET###.173 ###RAC_NODE_NAME_PREFIX###-gns1 IN A ###RAC_PUBLIC_SUBNET###.175 ###RAC_NODE_NAME_PREFIX###-gns2 IN A ###RAC_PUBLIC_SUBNET###.176 ; CMAN Server Entry ###RAC_NODE_NAME_PREFIX###-cman IN A ###RAC_PUBLIC_SUBNET###.3
Once the zone file is done, back up the existing reverse zone file and create a new reverse zone file as below.
[root@oradockerhost2 latest]# cat reversezonefile $TTL 86400 @ IN SOA ###DOMAIN_NAME###. root.###DOMAIN_NAME###. ( 2014090402 ; serial 3600 ; refresh 1800 ; retry 604800 ; expire 86400 ) ; minimum ; Name server's ###HOSTNAME_IP_LAST_DIGITS### IN NS ###DOMAIN_NAME###. ; Name server hostname to IP resolve. IN PTR ###HOSTNAME###.###DOMAIN_NAME###. ; Second RAC Cluster on Same Subnet on Docker 151 IN PTR ###RAC_NODE_NAME_PREFIX###1.###DOMAIN_NAME###. 152 IN PTR ###RAC_NODE_NAME_PREFIX###2.###DOMAIN_NAME###. 153 IN PTR ###RAC_NODE_NAME_PREFIX###3.###DOMAIN_NAME###. 154 IN PTR ###RAC_NODE_NAME_PREFIX###4.###DOMAIN_NAME###. 161 IN PTR ###RAC_NODE_NAME_PREFIX###1-vip.###DOMAIN_NAME###. 162 IN PTR ###RAC_NODE_NAME_PREFIX###2-vip.###DOMAIN_NAME###. 163 IN PTR ###RAC_NODE_NAME_PREFIX###3-vip.###DOMAIN_NAME###. 164 IN PTR ###RAC_NODE_NAME_PREFIX###4-vip.###DOMAIN_NAME###. 171 IN PTR ###RAC_NODE_NAME_PREFIX###-scan.###DOMAIN_NAME###. 172 IN PTR ###RAC_NODE_NAME_PREFIX###-scan.###DOMAIN_NAME###. 173 IN PTR ###RAC_NODE_NAME_PREFIX###-scan.###DOMAIN_NAME###. 175 IN PTR ###RAC_NODE_NAME_PREFIX###-gns1.###DOMAIN_NAME###. 176 IN PTR ###RAC_NODE_NAME_PREFIX###-gns2.###DOMAIN_NAME###. ; CMAN Server Entry 3 IN PTR ###RAC_NODE_NAME_PREFIX###-cman.###DOMAIN_NAME###.
Now it’s time to build our DNS server. First, it’s a good idea to reboot the server before starting the Docker configuration. Once the server is up, follow the below steps.
[root@oradockerhost2 ~]# cd /scratch/docker-images/OracleDatabase/RAC/OracleDNSServer/dockerfiles [root@oradockerhost2 dockerfiles]# ./buildContainerImage.sh -v latest
You should get an output like the below example.
Once completed execute the Docker run command.
[root@oradockerhost2 ~]# docker run -d --name racnode-dns \ --hostname racnode-dns \ --dns-search="example.com" \ --cap-add=SYS_ADMIN \ --network rac_pub_nw1 \ --ip 10.0.20.2 \ --sysctl net.ipv6.conf.all.disable_ipv6=1 \ --env SETUP_DNS_CONFIG_FILES="setup_true" \ --env DOMAIN_NAME="example.com" \ --env RAC_NODE_NAME_PREFIX="racnode" \ oracle/rac-dnsserver:latest
You can check the logs below.
[root@oradockerhost2 ~]# docker logs -f racnode-dns
In my case, I got the below output.
Login to the DNS server and check it correctly resolves our IP/hostnames.
[root@oradockerhost2 ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d69507706e2 oracle/rac-dnsserver:latest "/bin/sh -c 'exec $S…" 26 hours ago Up 19 hours racnode-dns [root@oradockerhost2 ~]# docker exec -it racnode-dns /bin/bash [orcladmin@racnode-dns ~]$ nslookup racnode1 Server: 10.0.20.2 Address: 10.0.20.2#53 Name: racnode1.example.com Address: 10.0.20.151
Configure Oracle Connection Manager (optional):
Copy Oracle database client 19c software to the below location
[root@oradockerhost2 ~]# cd /scratch/docker-images/OracleDatabase/RAC/OracleConnectionManager/dockerfiles/19.3.0 [root@oradockerhost2 ~]# cp /home/opc/LINUX.X64_193000_client.zip . [root@oradockerhost2 19.3.0]# cd .. [root@oradockerhost2 dockerfiles]# pwd /scratch/docker-images/OracleDatabase/RAC/OracleConnectionManager/dockerfiles [root@oradockerhost2 ~]# ./buildDockerImage.sh -v 19.3.0
Your output should be similar to this.
Now execute the below Docker run command to
[root@oradockerhost2 ~]# docker run -d --hostname racnode-cman \ --dns-search=example.com \ --dns=10.0.20.2 \ --network=rac_pub_nw1 \ --ip=10.0.20.3 \ -e DNS_SERVERS=10.0.20.2 \ -e DOMAIN=example.com \ -e PUBLIC_IP=10.0.20.3 \ -e PUBLIC_HOSTNAME=racnode-cman \ -e SCAN_NAME=racnode-scan \ -e SCAN_IP=10.0.20.171 \ --privileged=false \ -p 1521:1521 \ --name racnode-cman \ oracle/client-cman:19.3.0
Always check the logs
[root@oradockerhost2 ~]# docker logs -f racnode-cman
Configure Oracle RAC Database Container node 1:
Now it’s time to build our very first RAC node. Use the following steps to do it.
[root@oradockerhost2 ~]# cd /scratch/docker-images/OracleDatabase/RAC/OracleRealApplicationClusters/dockerfiles/21.3.0/ [root@oradockerhost2 21.3.0]# cp /home/opc/LINUX.X64_213000_* . [root@oradockerhost2 21.3.0]# cd .. [root@oradockerhost2 dockerfiles]# pwd /scratch/docker-images/OracleDatabase/RAC/OracleRealApplicationClusters/dockerfiles [root@oradockerhost2 dockerfiles]# ./buildContainerImage.sh -v 21.3.0
The outcome of the steps above should be similar to the output below.
Once you successfully built the RAC docker image, now it’s time to do some mandatory configuration at the Docker host level so that we can bring up the container without any issues.
A shared host file must be available to all containers. We will be creating it as below.
[root@oradockerhost2 ~]# mkdir /opt/containers; touch /opt/containers/rac_host_file
We can set up grid/oracle and database passwords during node creation. For that, we need to use an encrypted password file in the docker host. This location should be accessible by all containers.
[root@oradockerhost2 ~]# mkdir /opt/.secrets/ [root@oradockerhost2 ~]# openssl rand -hex 64 -out /opt/.secrets/pwd.key
Create a file called /opt/.secrets/common_os_pwdfile and seed the password for grid/oracle and database.
In this demonstration, I will be using Welcome1 as a common password for all accounts.
[root@oradockerhost2 ~]# echo "Welcome1" > /opt/.secrets/common_os_pwdfile [root@oradockerhost2 ~]# openssl enc -aes-256-cbc -salt -in /opt/.secrets/common_os_pwdfile -out /opt/.secrets/common_os_pwdfile.enc -pass file:/opt/.secrets/pwd.key [root@oradockerhost2 ~]# rm -f /opt/.secrets/common_os_pwdfile [root@oradockerhost2 ~]# chmod 400 /opt/.secrets/common_os_pwdfile.enc; chmod 400 /opt/.secrets/pwd.key
If you intend to use a distinct password for all accounts, you may create separate files for each and encrypt them under /opt/.secrets. However, make sure to refer to them using environment variables ORACLE_PWD_FILE, GRID_PWD_FILE, and DB_PWD_FILE for Oracle, Grid, and Database Account respectively
Now it’s time to create a Docker container with the below configuration.
[root@oradockerhost2 dockerfiles]# docker create -t -i \ --hostname racnode1 \ --volume /boot:/boot:ro \ --volume /dev/shm \ --tmpfs /dev/shm:rw,exec,size=4G \ --volume /opt/containers/rac_host_file:/etc/hosts \ --volume /opt/.secrets:/run/secrets:ro \ --volume /etc/localtime:/etc/localtime:ro \ --cpuset-cpus 0-3 \ --memory 16G \ --memory-swap 32G \ --sysctl kernel.shmall=2097152 \ --sysctl "kernel.sem=250 32000 100 128" \ --sysctl kernel.shmmax=8589934592 \ --sysctl kernel.shmmni=4096 \ --dns-search=example.com \ --dns=10.0.20.2 \ --device=/dev/sdb:/dev/asm_disk1 \ --device=/dev/sdc:/dev/asm_disk2 \ --privileged=false \ --cap-add=SYS_NICE \ --cap-add=SYS_RESOURCE \ --cap-add=NET_ADMIN \ -e DNS_SERVERS=10.0.20.2 \ -e NODE_VIP=10.0.20.161 \ -e VIP_HOSTNAME=racnode1-vip \ -e PRIV_IP=192.168.10.151 \ -e PRIV_HOSTNAME=racnode1-priv \ -e PUBLIC_IP=10.0.20.151 \ -e PUBLIC_HOSTNAME=racnode1 \ -e SCAN_NAME=racnode-scan \ -e SCAN_IP=10.0.20.171 \ -e OP_TYPE=INSTALL \ -e DOMAIN=example.com \ -e ASM_DEVICE_LIST=/dev/asm_disk1,/dev/asm_disk2 \ -e ASM_DISCOVERY_DIR=/dev \ -e CMAN_HOSTNAME=racnode-cman \ -e CMAN_IP=10.0.20.3 \ -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \ -e PWD_KEY=pwd.key \ --restart=always --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \ --cpu-rt-runtime=95000 --ulimit rtprio=99 \ --name racnode1 \ oracle/database-rac:21.3.0
Once the Docker container is created, we can assign networks to the container by executing the following commands:
[root@oradockerhost2 dockerfiles]# docker network disconnect bridge racnode1 [root@oradockerhost2 dockerfiles]# docker network connect rac_pub_nw1 --ip 10.0.20.151 racnode1 [root@oradockerhost2 dockerfiles]# docker network connect rac_priv_nw1 --ip 192.168.10.151 racnode1
Now it’s time to start the racnode1 container. Use the below command.
[root@oradockerhost2 dockerfiles]# docker start racnode1 [root@oradockerhost2 dockerfiles]# docker logs -f racnode1
Finally, you should see something like this.
Login to racnode1 and check the cluster resource as below.
[root@oradockerhost2 dockerfiles]# docker exec -it racnode1 /bin/bash [grid@racnode1 ~]$ [grid@racnode1 ~]$ crsctl stat res -t
Configure Oracle RAC Database Container node 2:
If your racnode1 is up and running it’s time to add a second RAC container node. Execute the below steps.
[root@oradockerhost2 dockerfiles]# docker create -t -i \ --hostname racnode2 \ --volume /boot:/boot:ro \ --volume /dev/shm \ --tmpfs /dev/shm:rw,exec,size=4G \ --volume /opt/containers/rac_host_file:/etc/hosts \ --volume /opt/.secrets:/run/secrets:ro \ --volume /etc/localtime:/etc/localtime:ro \ --cpuset-cpus 4-7 \ --memory 16G \ --memory-swap 32G \ --sysctl kernel.shmall=2097152 \ --sysctl "kernel.sem=250 32000 100 128" \ --sysctl kernel.shmmax=8589934592 \ --sysctl kernel.shmmni=4096 \ --dns-search=example.com \ --dns=10.0.20.2 \ --device=/dev/sdb:/dev/asm_disk1 \ --device=/dev/sdc:/dev/asm_disk2 \ --privileged=false \ --cap-add=SYS_NICE \ --cap-add=SYS_RESOURCE \ --cap-add=NET_ADMIN \ -e EXISTING_CLS_NODES=racnode1 \ -e DNS_SERVERS=10.0.20.2 \ -e NODE_VIP=10.0.20.162 \ -e VIP_HOSTNAME=racnode2-vip \ -e PRIV_IP=192.168.10.152 \ -e PRIV_HOSTNAME=racnode2-priv \ -e PUBLIC_IP=10.0.20.152 \ -e PUBLIC_HOSTNAME=racnode2 \ -e SCAN_NAME=racnode-scan \ -e SCAN_IP=10.0.20.171 \ -e OP_TYPE=ADDNODE \ -e DOMAIN=example.com \ -e ASM_DEVICE_LIST=/dev/asm_disk1,/dev/asm_disk2 \ -e ASM_DISCOVERY_DIR=/dev \ -e ORACLE_SID=ORCLCDB \ -e CMAN_HOSTNAME=racnode-cman \ -e CMAN_IP=10.0.20.3 \ -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \ -e PWD_KEY=pwd.key \ --restart=always --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \ --cpu-rt-runtime=95000 --ulimit rtprio=99 \ --name racnode2 \ oracle/database-rac:21.3.0
As we did for racnode1, we need to assign a network for racnode2 as well.
[root@oradockerhost2 dockerfiles]# docker network disconnect bridge racnode2 [root@oradockerhost2 dockerfiles]# docker network connect rac_pub_nw1 --ip 10.0.20.152 racnode2 [root@oradockerhost2 dockerfiles]# docker network connect rac_priv_nw1 --ip 192.168.10.152 racnode2
Now we can start the racnode2 container as below.
[root@oradockerhost2 dockerfiles]# docker start racnode2
Check the container logs and you should observe something similar to the below.
[root@oradockerhost2 dockerfiles]# docker logs racnode2 -f
Test your RAC database on Docker Containers:
Now it’s the moment of truth! Login to the connection manager container (it has Oracle client installed in it) and try to connect to the RAC database as follows.
[root@oradockerhost2 ~]# docker exec -it racnode-cman /bin/bash [oracle@racnode-cman ~]$ sqlplus sys/Welcome1@//racnode-scan.example.com:1521/ORCLCDB as sysdba
Further, you can access your RAC database from outside of the containers. This is possible via the Docker host itself because we’ve done port mapping from the Docker host to the connection manager container on port 1521.
Install the Oracle client on the docker host and follow the below steps.
[oracle@oradockerhost2 ~]$ sqlplus system/Welcome1@//localhost:1521/ORCLCDB SQL*Plus: Release 19.0.0.0.0 - Production on Mon Dec 20 17:05:58 2021 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Mon Dec 20 2021 16:49:10 +00:00 Connected to: Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.3.0.0.0 SQL>
Finally, we can outline that one of the easiest and fastest ways to get your Oracle RAC up and running is to deploy them on Docker containers. This increases agility, therefore considerably reducing the delivery time. Furthermore, it will be simpler when it comes to database maintenance, patching, upgrades, and migrations. In the future, we could expect an increase in the use of Oracle Databases on Docker containers.