In the first part of this blog post series I introduced the concept behind the Azure Monitoring Gateway (AMGW). In the second part of this blog post series I discussed the architecture and the pricing for AMGW's in a variety of configurations. The third part of this blog series provided details on how to install and configure AMGW servers.
This blog post will focus on steps to install and configure Operations Manager agents to report to AMGW servers and discuss security changes required on the management group.
Agent Installation & Configuration
For the AMGW's in this configuration we are working from the assumption that only port 5723 outbound from the agent will be available for communication. This means that agents will not be pushed from the Operations Manager console due to these port restrictions. As a result of this approach these will be treated as manually installed agents for each system monitored by the AMGW. This applies regardless of which of the three configurations (introduced the first part of this blog series) the AMGW is being used for. While in some of these configurations additional traffic beyond 5723 may be allowed, certificates will still need to be deployed to each agent so push from the Operations Manager console is not a viable solution. The diagram below shows how this configuration was tested:
Agents require name resolution or the FQDN of the management server(s) or gateway server(s) that they will be communicating with. One of the benefits to the AMGW approach is that name resolution is done via publicly accessible DNS servers. As a result, there are no specific steps required for name resolution though it is a good idea to do a telnet test from the agent to validate that it can resolve the name for each AMGW and can communicate outbound with it on port 5723. An example of this is shown below:
From my experiences with Operations Manager certificates are frequently the most complex piece of working with Operations Manager in untrusted environments. From a high level there needs to be a valid certificate loaded to the personal certificates with a validate certificate path as shown below. This is unique to the agent based upon the FQDN of the agent. The example below shows a valid certificate for my agent added to the Computer Account in the personal certificates area (not there are no errors on the certification path).
The root certificate should be in the Trusted Root Certification Authority as shown below (the trusted root certificate is part of the Certification Path for the agent certificate as shown in the screenshot above).
After using MOMCertImport this certificate should also appear in the Operations Manager certificates section (Note: it is ok that the certificate path is showing as critical – this is normal behavior within the Operations Manager section of the Certificates snap-in based on my experiences).
QuickTrick: Are you looking for a quick way to find out if the certificate that you added is configured correct and being used by Operations Manager? Stop the Microsoft Monitoring Agent service (in Operations Manager 2012 R2) or the System Center Monitoring service (In Operations Manager 2012). Clear the Operations Manager event log and then restart the service. Filter the Operations Manager event log to look for event number 20053 (the event source = "OpsMgr Connector"). This informational alert indicates that the certificate was accepted and used by Operations Manager on the system as shown below.
After validating that the certificate is working, the next event I look for is an informational event indicating that a management pack has successfully loaded. In most cases, after the first management pack loads on the agent it's not long until that agent will appear as monitored in Operations Manager.
Operations Manager Agents and Gateways have a built-in failover functionality where the agent/gateway can report to a primary and a failover system. The process to configure an agent to report to multiple management servers or gateway servers is discussed at http://blogs.technet.com/b/jimmyharper/archive/2010/07/23/powershell-commands-to-configure-gateway-server-agent-failover.aspx.
Troubleshooting the agent:
During my testing I ran into a situation where the time on an agent was not synchronizing correctly. Time synchronization is still important in Operations Manager even when using certificates. If you have an agent which appears to be installed correctly with the correct certificate but it's not changing state to monitored check the time on it to make sure it's correct.
Management Server configuration
Since the agents will be treated as manually installed, the settings for the management group need to be configured to allow this. This setting is configured on the administration pane under settings, security as shown below. The default configuration setting is "Reject new manual agent installations". The recommended configuration is "Review new manual agent installation in pending management view" as this will allow manual agent installation but will not automatically approve them.
Approving the Agents:
Once this setting is in place, new agents which report to the AMGW will appear in the administration pane under Device Management, Pending Management.
Next Steps for AMGW Architecture:
One of the next big steps for usage of AMGW's will be the automation of agent installation and certificate generation. The following are the steps identified for automation in an Operations Manager agent reporting to an AMGW: (If you have automated this process or have steps of this automated let me know – this would make a great follow-up blog post!)
1. Download installation media and extract the files
2. Add the root certificate
3. Create a certificate request
4. Approve the certificate request
5. Install the certificate, and use the MOMCertImport
6. Install the Operations Manager agent
I started the process to automate the agent installation and certificate creation but did not get it completed. My initial automation steps are documented below:
#1 Download installation media and extract the files
# File download not automated, manually placed in c:\temp
# Create temporary installation directories
rmdir -force -recurse c:\temp\64bit
# Extract required files
$shell = new-object -com shell.application
$zip = $shell.NameSpace("C:\temp\64bit.zip")
foreach($item in $zip.items())
#2 Add the root certificate
# From elevated cmd prompt in the unpacked directory type certutil –addstore Root 2012root.cer
certutil –addstore Root 2012root.cer
#3 Create a certificate request
# Change the custom value in 2012newreq.txt.
if ($env:userdnsdomain -eq $null)
(gc c:\temp\64bit\2012newreq.txt) -replace '<Machine FQDN>', $comp > c:\temp\64bit\2012newreq.inf
# Request the cert
certreq –New –f c:\temp\64bit\2012newreq.inf c:\temp\newcert.req
#4 Approve the certificate request
# Process not automated yet
#5 Install the certificate, and use the MOMCertImport
# Process not automated yet
#6 Install the Operations Manager agent
# Process not automated yet
Blog post series links:
The additional blog posts in the series include:
PowerShell and failover for agent and gateways: http://blogs.technet.com/b/jimmyharper/archive/2010/07/23/powershell-commands-to-configure-gateway-server-agent-failover.aspx
Obtaining certificates for Operations Manager via command line or script: http://blogs.technet.com/b/momteam/archive/2008/06/02/obtaining-certificates-for-ops-mgr.aspx?Redirected=true
Certificate enrollment for the Operations Manager agent: http://en-us.sysadmins.lv/Lists/Posts/Post.aspx?List=332991f0%2Dbfed%2D4143%2D9eea%2Df521167d287c&ID=5#prepare_certtmpl
Certificate errors and what they mean: http://www.systemcentercentral.com/wiki/operations-manager-wiki/operations-manager-authentication-event-reference/