Oracle M7 Supercluster Configurations

I wanted to share the current Oracle M7 Supercluster configurations. Most of this information is on the oracle M7 Supercluster datasheets however I present it in a slightly different way that may make it easier for you to understand.

 

Exadata Exafusion!

We are looking into using the Exafusion feature on an Exadata X5 full rack and will start to test in non-production first.

This is well documented in MOS Oracle support note: EXAFUSION - FAQ (Doc ID 2037900.1)

Exafusion is Direct-to-Wire protocol allows database processes to read and send Oracle Real Applications Cluster (Oracle RAC) messages directly over the Infiniband network bypassing the overhead of entering the OS kernel, and running the normal networking software stack. This improves the response time and scalability of the Oracle RAC environment on Oracle Exadata Database Machine.  Data is transferred directly from user space to the Infiniband network, leading to reduced CPU utilization and better scale-out performance. Exafusion is especially useful for OLTP applications because per message overhead is particularly apparent in small OLTP messages.

Exafusion helps small messages by bypassing OS network layer overhead and is ideal for this purpose.


Requirements to use Exafusion -

Oracle Exadata Storage Server Software release 12.1.2.1.1 or later
Oracle Database software Release 12.1.0.2.0 BP6 or later
Mellanox ConnectX-2 and ConnectX-3 Host Channel Adapters (HCAs) are required.
Oracle Enterprise Kernel 2 Quarterly Update 5 (UEK2QU5) kernels (2.6.39-400.2nn) or later are required.

Enabling Exafusion -

Exafusion is disabled by default.
To enable Exafusion, set the EXAFUSION_ENABLED initialization parameter to 1.
To disable Exafusion, set the EXAFUSION_ENABLED initialization parameter to 0.
This parameter cannot be set dynamically. It must be set before instance startup.
All of the instances in an Oracle RAC cluster must enable this parameter, or all of the instances in an Oracle RAC cluster must disable the parameter.

12c Dataguard broker setup error ORA-16698

The Oracle Data Guard broker is a great management framework that automates and centralizes the creation, maintenance, and monitoring of Oracle Data Guard configurations.

Recently for a Primary Oracle RAC database I was in the process of setting up a RAC Dataguard broker in Oracle 12c - 12.1.0.2 on the latest Exadata X5-2 and encountered an error ORA-16698. The setup steps and error details and solution/workaround I used is below.

On the Primary RAC database ensure the broker file is on ASM and turn on the broker.

-- Primary

alter system set dg_broker_config_file1 = '+DATA/DROID/DATAFILE/dg1_DROID.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATA/DROID/DATAFILE/dg2_DROID.dat' scope=both sid='*';

alter system set dg_broker_start=true scope=both sid='*';

On the Standby RAC database ensure the broker file is on ASM as well and turn on the broker.

-- Standby

alter system set dg_broker_config_file1 = '+DATA/DROIDDG/DATAFILE/dg1_DROIDDG.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATA/DROIDDG/DATAFILE/dg2_DROIDDG.dat' scope=both sid='*';

alter system set dg_broker_start=true scope=both sid='*';

 

On the primary I invoked the dataguard broker and encountered the error when setting up the configuration for the standby database.

-- Back to Primary


DGMGRL> connect sys
Password:
Connected as SYSDG.
DGMGRL>  CREATE CONFIGURATION 'DROIDDR' AS PRIMARY DATABASE IS 'DROID' CONNECT IDENTIFIER IS DROID;
Configuration "DROIDDR" created with primary database "DROID"
DGMGRL> ADD DATABASE 'DROIDDG' AS CONNECT IDENTIFIER IS DROIDDG;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.

Save the log archive destination settings from both the Primary and Standby databases then remove the configuration.

 
-- PRIMARY log_archive_dest_2 setting
log_archive_dest_2='SERVICE=DROIDDG ARCH VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DROIDDG'

-- STANDBY log_archive_dest_2 setting
log_archive_dest_2='SERVICE=DROID ARCH VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DROID'

-- Remove the Dataguard configuration

DGMGRL> remove configuration;
Removed configuration


 Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

alter system set log_archive_dest_2='' scope=both sid='*';
  

Disable then Enable the broker parameter on both the Primary and Standby databases.

-- Primary
alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';

-- Standby
alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';


On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.


[oracle@okx1pdbadm06 ~]$ dgmgrl
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected as SYSDG.
DGMGRL>
DGMGRL>  CREATE CONFIGURATION 'DROIDDR' AS PRIMARY DATABASE IS 'DROID' CONNECT IDENTIFIER IS DROID;
Configuration "DROIDDR" created with primary database "DROID"
DGMGRL> ADD DATABASE 'DROIDDG' AS CONNECT IDENTIFIER IS DROIDDG;

 Revert the original settings back for the log_archive_dest_2 settings from both Primary and Standby databases.


-- PRIMARY log_archive_dest_2
alter system set log_archive_dest_2='SERVICE=DROIDDG ARCH VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DROIDDG' sid='*' scope=both;

-- STANDBY log_archive_dest_2
alter system set log_archive_dest_2='SERVICE=DROID ARCH VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DROID' sid='*' scope=both;

  

Enable the broker configuration and we now have a successful 12c RAC dataguard broker configuration enabled on Exadata!


[oracle@okx1pdbadm06 ~]$ dgmgrl
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected as SYSDG.
DGMGRL>

DGMGRL> enable configuration;
Enabled.
DGMGRL> show configuration;

Configuration - DROIDDR

  Protection Mode: MaxPerformance
  Members:
  DROID   - Primary database
    DROIDDG - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 315 seconds ago)




    

ORA-00304: requested INSTANCE_NUMBER is busy

I was working on verifying that all 12c Dataguard instances were running on an Exadata X5 full rack system and I came across one instance that was not running. I noticed this when I saw the /etc/oratab entry and did not see the corresponding instance running on the database compute server.

I manually tried to bring up the instance from node 3 and encountered the error below.

node3-DR> sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Dec 16 18:30:15 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;

ORA-00304: requested INSTANCE_NUMBER is busy

SQL> exit

Disconnected

I then looked at the instance alert log and saw the following:

...

Wed Dec 16 18:30:22 2015

ASMB started with pid=39, OS id=59731

Starting background process MMON

Starting background process MMNL

Wed Dec 16 18:30:22 2015

MMON started with pid=40, OS id=59733

Wed Dec 16 18:30:22 2015

MMNL started with pid=41, OS id=59735

Wed Dec 16 18:30:22 2015

NOTE: ASMB registering with ASM instance as Standard client 0xffffffffffffffff (reg:1438891928) (new connection)

NOTE: ASMB connected to ASM instance +ASM3 osid: 59737 (Flex mode; client id 0xffffffffffffffff)

NOTE: initiating MARK startup

Starting background process MARK

Wed Dec 16 18:30:22 2015

MARK started with pid=42, OS id=59741

Wed Dec 16 18:30:22 2015

NOTE: MARK has subscribed

High Throughput Write functionality enabled

Wed Dec 16 18:30:34 2015

USER (ospid: 59191): terminating the instance due to error 304

Wed Dec 16 18:30:35 2015

Instance terminated by USER, pid = 59191

 The error number was 304 that was in the output from sqlplus and from the alert log as well, then the next thing I did was naturally check google and oracle support to see if I could find something that matched my scenario and could not find something exact. 

Then the next thing that came to mind was to check gv$instance to see where the Primary RAC and Standby RAC standby database were running from.

# Primary
  INST_ID INSTANCE_NUMBER INSTANCE_NAME     HOST_NAME      STATUS
---------- --------------- ---------------- ------------ ------------
         1               1     PROD1            node01        OPEN
         3               3     PROD3            node08        OPEN
         2               2     PROD2            node07        OPEN

# Standby
   INST_ID INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME      STATUS
---------- --------------- ---------------- ------------ ------------
         1               1 DG1               node01        MOUNTED
         3               3 DG3               node08        MOUNTED
         2               2 DG2               node07        MOUNTED

I then realized from node 3 that the oratab entry for instance 3 was not correct it was already running from node node08!

Furthermore I also verified the RAC configuration via srvctl.

node3-DR>srvctl config database -d DG3

Database unique name: DG

Database name: PROD

Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1

Oracle user: oracle

Spfile: +RECOC1/DG/PARAMETERFILE/spfile.4297.894386377

Password file:

Domain:

Start options: mount

Stop options: immediate

Database role: PHYSICAL_STANDBY

Management policy: AUTOMATIC

Server pools:

Disk Groups: DATAC1,RECOC1

Mount point paths:

Services:

Type: RAC

Start concurrency:

Stop concurrency:

OSDBA group: dba

OSOPER group: dba

Database instances: DG1,DG2,DG3

Configured nodes: node01,node07,node08

Database is administrator managed

The configured nodes for the Dataguard Standby database is supposed to be running from node01, node07 and node08 it is not configured to run on node03.

Once I confirmed this I simply removed the oratab entry to prevent any future further confusion. Moral of the story please only leave oratab entries intact for real instances that need to run from the node. ☺