Oracle Enterprise Manager Cloud Control 12.1 – Agent Installation, Issues and Solutions

Currently busy for a client to install and configure Oracle Enterprise Manager 12c for database and more administration across the globe. These environments were configured and setup by a different 3rd party so not always follow our wishes and administration guidelines. You can imagine that such environments are also a neat environments regarding learning curves while attempting (and today still successful) implementation of a “administer everything via Oracle Enterprise Manager 12c” setup.

Besides some “getting used to” and “Marco, read the manual first, for once in your live time” – issues, I encountered multiple issues (problems) trying to install the Oracle OEM 12c Agent software. Your first step to success… I also found some answers that I hereby want to share, so they will soften your upgrade and installation pain (if there is any).

First of all, it is probably a very good idea to install the agent software in a separate OS user, with the proper “oinstall” and “dba” OS group privileges. I still haven’t done it, but I guess that it would have made my issues / learning curve less painful. Installing the agent software via such a predetermined setup OS user would have guaranteed a existing predefined standard basis to start with. Especially in my globally scattered multiple database environments with different setups, Optimal Flexible Architecture (OFA) implementation ideas and OS usernames, home directories and group naming, this was really asking for a lot of extra effort and problems. Also in such an environment, it is very difficult to create a standard installation (manual description).

The following is my kind of installation reference and issues & solutions I came up with, so far…

Keep in my these items are reflecting MY environment (Sun Sparc 64b machines) and might not reflect your problems AND MOST IMPORTANTLY, these are possible guidelines that MIGHT help you but maybe will not…

OEM 12c Agent Software – Installation

The following defaults for Oracle 12c Agent software install on local and remote hosts apply.

Basics

Use the following values and/or applied rules.

•  Always use Fully Qualified Domain Names (FQDN) naming regarding hosts entries in local and remote sites.
• Double check naming for local and remote names in local and remote /etc/hosts file are EXACTLY the same, including TCP/IP numbers and properly resolved via /etc/hosts in the following (mandatory) format:

 {TCP/IP Number} {Fully Qualified Domain Name} {Hostname} [Aliases]

Most issues occur due to mismatched FQDN in remote and local hosts files.

A host file based name resolving setup must be used on remote and local sites.

Firewall

Check on firewall issues beforehand and be aware that, by default port 4900 will be used for http(s) traffic from the OEM Agent software to the Oracle Management Server environment and port 3872 from the OMS environment to its Oracle Agents.

Sometimes you will also see this when the OMS/OEM server node is not added in the target/remote server node in the /etc/hosts file.

If the Fully Qualified Domain Name is added and will be properly resolved, then the next step is to add and deploy the agent software on this new node.

Directories

It is also a good idea to beforehand check the read and write privileges on your remote site for the directories accessed, so Agent install directory but also the settings on the $ORACLE_BASE / $HOME directories. You will get a warning if you don’t have the proper settings during install, but it is better, of course, to check this beforehand.

Steps To Add a Target

The following section provides you with steps needed to add a target, that is deploy the minimal Oracle Enterprise Manager Agent software to be able to add databases, listeners etc, afterwards in Oracle Enterprise Manager Cloud control.

Manually Adding a Target

Be also aware that the settings here are discussed in our group regarding “How to setup OFA naming” or regarding security implementation settings.  This setting might not reflect your environment, but they are described here so you have a context regarding the solutions and issues sections further down the line. I don’t say the are good implementation settings; I don’t say they are bad implementation settings, BUT this is where we internally agreed on and fit our purposes. We have implementations with multiple Oracle software owners: oracle_dev, oracle_prd etc., but agreed to refer back, standardize on the general used “oracle” user.

1. Click [Setup] in the menu
2. Click [Add Targets Manually]
3. Add Targets Manually by marking [Add Host Targets] and click on button [Add Host…]
4. Add the new server (use a FQDN!) via the [ + Add] button and select the proper OS agent platform / software
5. When done click the [Next] button
6. Enter the following values for the blank fields/columns:

• Use /oracle/product/12.1.0/agent as the value for “Installation Base Directory for the OMS/Agent 12.1.0.1.0 software.

Be aware that:

• /oracle must exists, owned by the “Named Credential” and writable
• the root, first directory that is defined in $ORACLE_BASE must also be owned and writable by the “Name Credential” user, so for example this would be /u01 in /u01/app/oracle (=$ORACLE_BASE) or for instance the /oracle_prd directory of /oracle_prd/product (=$ORACLE_BASE)
• Keep the automated filled in “Instance Directory” value as is
• Use the proper “Named Credential” value for the remote software Oracle install owner
  • Alter the “Privileged Delegation Setting” accordingly, if “sudo” is in place on the remote site
  • Alter the value into /usr/local/bin/sudo –u %RUNAS% %COMMAND% for a Sun OS, instead of the default /usr/bin/sudo
  • If “sudo” is NOT place on the remote site: blank out all values (keep it empty). You will be asked to run the root.sh script afterwards manually via the root account.
  • If possible, especially when jumping network ranges and/or when firewall settings are in place, try to keep the default port setting of 3872. If this can not be done, empty out the 3872 value, so it will use a range between 1830 and 1849

7. When done: Click the [Next] button.
8. After values been reviews regarding correctness: Click the [Deploy Agent] button.

Issues and Possible Solutions…

Possible solutions to issues that might occur during the installation of Oracle Enterprise Management 12.1.0.1 Agent software.

Issue #01

“Banner Check Failed”

Solution

Remove the statements that create the banner (or extra print text call) in the .bash_profile/.profile etc. Adding the banner to the ignoremessages.txt file in the OMS environment doesn’t always help because banners can/will be autogenerated based on the environment. So in this case it is easier to (temporarily) remove the scripting that creates the banner during shell login.

Issue #02

“Agent Configuration Failed SEVERE: Agent port passed by user is busy and cannot proceed with the configuration. Pass a free port and retry the configuration”

The command “netstat -na|grep 3872” clearly shows the port is not being used.

Solution

Possible solutions are

• Port 3872 is being blocked by routers, security, firewall (iptable(s)) settings
• TCP/IP numbers for hosts in local /etc/hosts and remote /etc/hosts don’t match
• Hostnames for hosts in local /etc/hosts and remote /etc/hosts don’t match exactly

See also on http://support.oracle.com:

• Bug 13652664: AGENT DEPLOY FAILS WITH AGENT PORT PASSED BY USER IS BUSY
• Bug 13943336: AGENT INSTALL FAILS IN PORT ERROR INSPITE OF PORT AVAILABLE
• Bug 14170013: INSTALLING AGENT FOR OEM CLOUD FAILS WITH BUSY PORT BUT PORT IS NOT BUSY.

Issue #03

“EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)”

This might be due to an incomplete OMS/Agent verification handshake. In other words, the OMS_DOWN bit does not apply.

If so, you might see something like (mind the OMS version (unknown) status):

[]/oracle/product/12.1.0/agent/agent_inst/bin/emctl status agent verify

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
verifykey: Successfully Completed communication with agent
[]/oracle/product/12.1.0/agent/agent_inst/bin/emctl status agent

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : (unknown)
Protocol Version  : 12.1.0.1.0
Agent Home        : /oracle/product/12.1.0/agent/agent_inst
Agent Binaries    : /oracle/product/12.1.0/agent/core/12.1.0.1.0
Agent Process ID  : 15360
Parent Process ID : 15214
Agent URL         : https://remote_dev01:2872/emd/main/
Repository URL    : https://oms12c:4900/empbs/upload
Started at        : 2012-06-07 16:04:32
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 3
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 89.79%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2012-06-07 16:16:45
Last successful heartbeat to OMS             : (none)
---------------------------------------------------------------
Agent is Running and Ready

Solution

This might be solved by

1. Stop the Oracle Agent on the remote host via:

/oracle/product/12.1.0/agent/agent_inst/bin/emctl stop agent

2. Clean up all not send XML messaging data

/oracle/product/12.1.0/agent/agent_inst/bin/emctl clearstate agent

3. Secure the agent (OMS/Agent SSL handshake/negotiation) via

/oracle/product/12.1.0/agent/agent_inst/bin/emctl secure agent

4. Start the Oracle Agent software via

/oracle/product/12.1.0/agent/agent_inst/bin/emctl start agent

5. Check the OMS Version status via

/oracle/product/12.1.0/agent/agent_inst/bin/emctl status agent

The output should now show something like the following:

[]/oracle/product/12.1.0/agent/agent_inst/bin/emctl status agent verify

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /oracle/product/12.1.0/agent/agent_inst
Agent Binaries    : /oracle/product/12.1.0/agent/core/12.1.0.1.0
Agent Process ID  : 12895
Parent Process ID : 12805
Agent URL         : https://remote_dev01:2872/emd/main/
Repository URL    : https://oms12c:4900/empbs/upload
Started at        : 2012-06-07 16:25:32
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2012-06-07 16:25:41
Last attempted upload                        : 2012-06-07 16:25:41
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 2
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 90.61%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2012-06-07 16:25:40
Last successful heartbeat to OMS             : 2012-06-07 16:25:40
---------------------------------------------------------------
Agent is Running and Ready

Issue #04

The following error might occur:

“Error Message: Agent Configuration Failed SEVERE: Agent port passed by user is busy and cannot proceed with the configuration. Pass a free port and retry the configuration. Exit Code :1”

None of the earlier mentioned issues are valid, that is:

• Hostnames and TCP/IP numbers resolve
• Ports local (4900, OMS) and remote (3872) are reachable
• No firewall restrictions for the mentioned ports

Solution

Check if the port is busy as recommended by the installer.

netstat -na | grep 3872
      *.3872               *.*                0      0 128000      0 LISTEN
      *.3872               *.*                0      0 128000      0 LISTEN

If the port is indeed already in use, stop the or kill the process. On solaris this might be done via (kudos to Lubos Kosco, Oracle for providing a solution for Sun environments):

bash-3.2$ cat getport.sh

#!/bin/bash
# Original script from Lubos Kosco, Oracle, via
# https://blogs.oracle.com/taz/entry/get_application_pid_listening_on
#
# Get the process which listens on port

# $1 is the port we are looking for

if [ $# -lt 1 ]
then
echo "Please provide a port number parameter for this script"
echo "e.g. $0 22"
exit
fi

echo "Greping for your port, please be patient (CTRL+C breaks) ... "

for i in `ls /proc`
do
pfiles $i | grep AF_INET | grep $1
if [ $? -eq 0 ]
then
echo Is owned by pid $i
fi
done

bash-3.2$ ./getport.sh
Please provide a port number parameter for this script
e.g. ./getport.sh 22

bash-3.2$ ./getport.sh 3872
Greping for your port, please be patient (CTRL+C breaks) ...
        sockname: AF_INET6 ::  port: 3872
Is owned by pid 1091

bash-3.2$ ps -ef|grep 1091
 ora_dev  1091   874   0 10:30:48 ?           0:25 /oradata/oracle/product/12.1.0/agent/core/12.1.0.1.0/jdk/bin/sparcv9/java -Xmx1
 ora_dev 21636  2946   0 12:03:21 pts/3       0:00 grep 1091

This wasn’t the case at that time, in my situation, but after a complete reinstall issue number #05 occurred with the recommendation to execute:

• /oracle/product/12.1.0/agent/agent_inst/bin/emctl secure agent
• /oracle/product/12.1.0/agent/agent_inst/bin/emctl start agent
• /oracle/product/12.1.0/agent/agent_inst/bin/emctl config agent addinternaltargets

This might have been an indication regarding a SSL certification mismatch (so once again hostname/FQDN related).

Issue #05

“Execution of command /oracle/product/12.1.0/agent/agent_inst/bin/emctl start agent on host … failed”

Solution

Follow the recommendation to execute:

• /oracle/product/12.1.0/agent/agent_inst/bin/emctl secure agent
• /oracle/product/12.1.0/agent/agent_inst/bin/emctl start agent
• /oracle/product/12.1.0/agent/agent_inst/bin/emctl config agent addinternaltargets

If once again there are some issues, double check if issue number 3 might apply.

Issue #06

“The Agent Oracle Home or Install Base Directory, is already registered with the inventory”

A typical issue endured after connection timeout errors or others where the installation of the software succeeded and/or other issues broke the installation.

Solution

This might be caused due to inconstancy between the actual install status and the oraInventory bookkeeping in $ORACLE_BASE/oraInventory/ContentsXML file “inventory.xml”. This inventory.xml file can also be used to find the used ORACLE_HOME and ORACLE_HOME_NAME of the OEM Agent values used during the silent install.

Also see for more info: http://docs.oracle.com/cd/B28359_01/em.111/b31207/oui2_manage_oracle_homes.htm#CJAEBIAH

For example via values found in “inventory.xml” you could now detach the offending oracle homes and/or manually or automatically remove the software as well.

/oracle_dev/product/10.2.0.4/oui/bin/runInstaller -silent -detachHome -invPtrLoc /oracle_dev/oraInventory/oraInst.loc ORACLE_HOME="/oracle/product/12.1.0/agent/core/12.1.0.1.0" ORACLE_HOME_NAME="agent12g1"
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /oracle_dev/oraInventory/oraInst.loc
The inventory is located at /oracle_dev/oraInventory

'DetachHome' was successful

/oracle_dev/product/10.2.0.4/oui/bin/runInstaller -silent -detachHome -invPtrLoc /oracle_dev/oraInventory/oraInst.loc ORACLE_HOME="/oracle/product/12.1.0/agent/sbin" ORACLE_HOME_NAME="sbin12g1"
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /oracle_dev/oraInventory/oraInst.loc
The inventory is located at /oracle_dev/oraInventory

'DetachHome' was successful.

Issue #07

Encountered another one with the following “Initialization Issue”…

Connection to the SSH daemon (sshd) on the target host failed with the following error: “Algorithm negotiation fail”

Solution

This might be caused by not recognizing the cipher method via the sshd daemon. The following might be logged in the system log file /var/adm/messages:

  Jul  4 14:45:56 [hostname] sshd[29575]: [ID 800047 auth.crit] fatal: Client and 
  server could not agree on a common cipher: client "aes128-cbc,3des-cbc,
  blowfish-cbc", server "aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,
  arcfour". The server cipher list can be controlled using the "Ciphers" option, 
  see sshd_config(4) for more information.

Therefore adding the cipher method to the /etc/ssh/ssh_config should solve it. This can be done via adding a “Cipher”/”Ciphers” parameter specifying the needed method. A standard, vanilla, ssh_config file might look like this

cat /etc/ssh/ssh_config

# $OpenBSD: ssh_config,v 1.21 2005/12/06 22:38:27 reyk Exp $
# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.
# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.
# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.
# Host *
#   ForwardAgent no
#   ForwardX11 no
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
Host *
        GSSAPIAuthentication yes
# If this option is set to yes then remote X11 clients will have full access
# to the original X11 display. As virtually no X11 client supports the untrusted
# mode correctly we set this to yes.
        ForwardX11Trusted yes
# Send locale-related environment variables
        SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
        SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
        SendEnv LC_IDENTIFICATION LC_ALL

The Oracle Linux OEM environment makes an attempt with the following Cipher methods, so add all or one of these to the sshd configuration file.

  client "aes128-cbc,3des-cbc,blowfish-cbc"

In short

Most issues can be avoided by being very precise regarding configuration and checks when dealing with resolving the Fully Qualified Domain Names of the Oracle Enterprise Manager server and the remote to be administered node/server machine.

Be aware of firewall issues and privileges. Currently we temporarily avoided some security problems by importing the public ssh key of the OEM software owner in the authorized_keys file of the to be managed remote software owner. This is of course not the most secure solution due to the fact that the “OEM owner” can open any connection to remote machines without being asked for a password verification. One of the reasons why we temporarily used this to install the software as long as we were not yet officially in charge of those machines (we are now as of the 1st of July).

When I encounter more issues, I will add them here, but be very aware that those issues and solutions described here might not work or even damage your environment, if you are not certain about what you are to do. Most pre-conditions are described in the Oracle Enterprise Manager 12c administration manual and should be studied carefully.

There are more ways of installing the Oracle Agent software, like via a RPM, like described by Martin Bach (“Installing OEM 12c agents in RPM format”) but I hope this manual install will help some people.

HTH