Oracle ASR snmp notification solution for Database Compute Nodes

Oracle ASR - Automatic Service Request is a secure automatic service request generation and priority service request handling for hardware faults on Oracle engineered systems such as Exadata and the ZFS storage appliance. When a hardware fault is detected, Oracle Auto Service Request opens a service request with Oracle automatically and transports electronic fault telemetry to help 

expedite the diagnostic process. Oracle Support is notified promptly and parts are dispatched upon receipt of the service request if required.

I was working on setting up ASR on a couple of Exadata systems for a client and noticed the asrexachk script did not get any snmp notification from the Database compute nodes. The asrexachk script is designed to check and test ASR configurations to make sure that the Engineered Systems can communicate to the ASR Manager server.

Oracle Support suggested that the correct route on the Exadata compute nodes may not be taken by snmp to send a message via the UDP protocol on port 162 and that a manual static route may need to be added to the Exadata database compute nodes,

The following commands were given as a solution to run on all database compute nodes.

First you need to get the gateway IP address from the server.

$ grep -i gateway /etc/sysconfig/network-scripts/ifcfg-eth0
GATEWAY=<GATEWAY IP Address>

Then you can add the static route.

$ /sbin/route add -host <ASR Manager Server IP Address> gw <GATEWAY IP Address>


Run the tcpdump command from the ASR Manager server to see what notifications are sent in. You will see after the above route changes snmp trap notifications will start to come in from the DB nodes when an test notification is generated from the DB compute nodes.

[root@ASRMANAGER ~]# tcpdump -ni eth1 udp port 162

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes

22:06:28.712136 IP <DB node IP address>.33020 > <ASR Manager Server IP Address>.snmptrap:  Trap(24)  .1.3.6.1.6.3 10.43.47.133 coldStart 0   <--- SNMP TRAP DB NODE

22:08:36.931920 IP <DB node IP address>.12984 > <ASR Manager Server IP Address>.snmptrap:  V2Trap(286)  .1.3.6.1.2.1.1.3.0=206723517 [|snmp] <--- DB NODE ASR SNMP TRAP

From the Database compute nodes you can run the snmptrap command to send to the ASR Manager server and you will see the above message.

[root@EXANODE1 ~]# snmptrap -v 1 -c public <ASR Manager Server IP Address> .1.3.6.1.6.3 "" 0 0 coldStart.0

From the Database compute nodes you can run the exadata_mon_hw_asr.pl command to send to the ASR Manager server and you will see the above message.

[root@EXANODE1 ~]# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -validate_snmp_subscriber -type asr

Sending test trap from <DB NODE 1 IP ADDRESS> to destination - <ASR Manager Server IP Address>:162

Then an email is sent as follows and it comes from no.reply@oracle.com and is with the subject - Oracle ASR: *Test* Service Request.

Serial#: XXXXXXXXX

Hostname: <DB NODE>
Service Request test-create was successful.

The Oracle Auto Service Request documentation can be accessed on http://oracle.com/asr.
Please use My Oracle Support https://support.oracle.com for assistance. 

The below table is a reference guide for the ports requirements for the ASR Manager server and also the source, destination, protocol and description information.