Oracle ASR snmp notification solution for Database Compute Nodes

Oracle ASR - Automatic Service Request is a secure automatic service request generation and priority service request handling for hardware faults on Oracle engineered systems such as Exadata and the ZFS storage appliance. When a hardware fault is detected, Oracle Auto Service Request opens a service request with Oracle automatically and transports electronic fault telemetry to help 

expedite the diagnostic process. Oracle Support is notified promptly and parts are dispatched upon receipt of the service request if required.

I was working on setting up ASR on a couple of Exadata systems for a client and noticed the asrexachk script did not get any snmp notification from the Database compute nodes. The asrexachk script is designed to check and test ASR configurations to make sure that the Engineered Systems can communicate to the ASR Manager server.

Oracle Support suggested that the correct route on the Exadata compute nodes may not be taken by snmp to send a message via the UDP protocol on port 162 and that a manual static route may need to be added to the Exadata database compute nodes,

The following commands were given as a solution to run on all database compute nodes.

First you need to get the gateway IP address from the server.

$ grep -i gateway /etc/sysconfig/network-scripts/ifcfg-eth0
GATEWAY=<GATEWAY IP Address>

Then you can add the static route.

$ /sbin/route add -host <ASR Manager Server IP Address> gw <GATEWAY IP Address>


Run the tcpdump command from the ASR Manager server to see what notifications are sent in. You will see after the above route changes snmp trap notifications will start to come in from the DB nodes when an test notification is generated from the DB compute nodes.

[root@ASRMANAGER ~]# tcpdump -ni eth1 udp port 162

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes

22:06:28.712136 IP <DB node IP address>.33020 > <ASR Manager Server IP Address>.snmptrap:  Trap(24)  .1.3.6.1.6.3 10.43.47.133 coldStart 0   <--- SNMP TRAP DB NODE

22:08:36.931920 IP <DB node IP address>.12984 > <ASR Manager Server IP Address>.snmptrap:  V2Trap(286)  .1.3.6.1.2.1.1.3.0=206723517 [|snmp] <--- DB NODE ASR SNMP TRAP

From the Database compute nodes you can run the snmptrap command to send to the ASR Manager server and you will see the above message.

[root@EXANODE1 ~]# snmptrap -v 1 -c public <ASR Manager Server IP Address> .1.3.6.1.6.3 "" 0 0 coldStart.0

From the Database compute nodes you can run the exadata_mon_hw_asr.pl command to send to the ASR Manager server and you will see the above message.

[root@EXANODE1 ~]# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -validate_snmp_subscriber -type asr

Sending test trap from <DB NODE 1 IP ADDRESS> to destination - <ASR Manager Server IP Address>:162

Then an email is sent as follows and it comes from no.reply@oracle.com and is with the subject - Oracle ASR: *Test* Service Request.

Serial#: XXXXXXXXX

Hostname: <DB NODE>
Service Request test-create was successful.

The Oracle Auto Service Request documentation can be accessed on http://oracle.com/asr.
Please use My Oracle Support https://support.oracle.com for assistance. 

The below table is a reference guide for the ports requirements for the ASR Manager server and also the source, destination, protocol and description information.

Enterprise Manager 12c No Special Characters for SYSMAN account

While doing an upgrade of Oracle Enterprise Manager 12c from 12.1.0.1 to 12.1.0.4 I encountered an error which caused the upgrade to fail. Please keep in mind you cannot do a direct path upgrade from EM 12.1.0.1 to 12.1.0.4 you will first need to apply Bundle Patch (BP1) to the 12.1.0.1 binaries and then you can upgrade from 12.1.0.1 BP1 --> 12.1.0.2 or 12.1.0.3 and then finally you can upgrade to 12.1.0.4.

From the upgrade logs the error stack is below.


INFO: oracle.sysman.top.oms:MDS Schema Creation status: FAILURE 
SEVERE: oracle.sysman.top.oms:MDS Schema Creation is failed 
INFO: oracle.sysman.top.oms:_______________________________________________________________________ 
INFO: oracle.sysman.top.oms:Starting opss Schema Creation. 
INFO: oracle.sysman.top.oms:Failed to create OPSS schema 
INFO: oracle.sysman.top.oms:_______________________________________________________________________ 
INFO: oracle.sysman.top.oms:Starting APM Schema Creation. 
INFO: oracle.sysman.top.oms:APM Component is already created 
INFO: oracle.sysman.top.oms:The plug-in MDS Schema Configuration has failed its perform method 

Cannot change password for sysman_mds java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 

ERROR:MDS java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 

java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 



The Upgrade basically failed due to special character(s) in the SYSMAN password. Please make sure you do not have any special characters in the SYSMAN password such as *,#,!, etc Just use a good combination of alpha and number characters.

Since the upgrade failed I had to fallback and do the following to restart the upgrade all over again for this issue.

1.) Restore the database, inventory and OMS Middleware Home from backup.
2.) Startup database and listener
3.) Start the OMS
4.) Change the SYSMAN password that doesn't use special character(s), procedure below.


<OMS_HOME>/bin/emctl config oms -change_repos_pwd

Please note changing the SYSMAN password will change it on both the OMS and on the repository database.

Enterprise Manager 12c Patching changes

Starting with Enterprise Manager 12.1.0.4.3 (PS3) all OMS bundle patches and plugins will be released as a single OMS-side "System" patch.  The patch will be installed using the OPatchauto installation method.  The EM 12.1.0.4.0 Agent and Agent-side Plug-in Bundle patches will continue to use Opatch.

More details at MOS: Enterprise Manager 12.1.0.4.0 (PS3) Master Bundle Patch List for Agent and Plug-ins (OMS and Agent side) (Doc ID 1900943.1)

Bash security alert on Exadata/Oracle Linux

There is a new security issue named "Bash Shell Shock" also documented as CVE-2014-6271/CVE-2014-7169.

Shellshock is the nickname for a flaw in Bourne Shell, or Bash, which is commonly used on Oracle Enterprise Linux (OEL). The Bash Shell Shock vulnerability allows remote attackers to write to files or possibly have unknown other impact.

On database nodes, obtain and update updated bash package using the following version, or later:

bash-3.2-33.el5_11.4.x86_64

This package may be obtained via public-yum.oracle.com.

To install this package on DB nodes, the exadata-sun-computenode-exact RPM must be removed first. If using Exadata DB server image version 11.2.3.3.0 or later, first run this command: rpm -e exadata-sun-computenode-exact

Then, use this command on all releases to install the updated rpm.

rpm -Uvh <new bash rpm>

To install this package on storage cells (supported as an exception for this CVE only), install using "rpm -Uvh --nodeps <path to bash rpm>"

Stay up to date by regularly check My Oracle Support(MOS) Note for the latest security finding.

Responses to common Exadata security scan findings (Doc ID 1405320.1)

The following command can be used to see if you are affected by the issue, if the command returns vulnerable then you are at risk.

[root@localhost ~]# env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; e                                                     cho vulnerable' bash -c "echo this is a test"

vulnerable

bash: BASH_FUNC_x(): line 0: syntax error near unexpected token `)'

bash: BASH_FUNC_x(): line 0: `BASH_FUNC_x() () { :;}; echo vulnerable'

bash: error importing function definition for `BASH_FUNC_x'

this is a test

Below I have included the before and after results of removing the Shell shock issue from a OEL server.

[root@localhost ~]# yum update bash

Loaded plugins: downloadonly, security

public_ol6_UEKR3_latest                                                                                       | 1.2 kB     00:00

public_ol6_latest                                                                                             | 1.4 kB     00:00

public_ol6_latest/primary                                                                                     |  41 MB     00:37

public_ol6_latest                                                                                                        26205/26205

Setting up Update Process

Resolving Dependencies

--> Running transaction check

---> Package bash.x86_64 0:4.1.2-15.el6_4 will be updated

---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be an update

--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================

 Package                 Arch                      Version                                Repository                            Size

=====================================================================================================================================

Updating:

 bash                    x86_64                    4.1.2-15.el6_5.2                       public_ol6_latest                    905 k

Transaction Summary

=====================================================================================================================================

Upgrade       1 Package(s)

Total download size: 905 k

Is this ok [y/N]: y

Downloading Packages:

bash-4.1.2-15.el6_5.2.x86_64.rpm                                                                              | 905 kB     00:00

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Updating   : bash-4.1.2-15.el6_5.2.x86_64                                                                                      1/2

  Cleanup    : bash-4.1.2-15.el6_4.x86_64                                                                                        2/2

  Verifying  : bash-4.1.2-15.el6_5.2.x86_64                                                                                      1/2

  Verifying  : bash-4.1.2-15.el6_4.x86_64                                                                                        2/2

Updated:

  bash.x86_64 0:4.1.2-15.el6_5.2

Complete!

[root@localhost ~]# env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo this is a test"

bash: warning: x: ignoring function definition attempt

bash: error importing function definition for `BASH_FUNC_x'

this is a test

[root@localhost ~]#