Revision as of 18:44, 21 November 2007

Nagios Integration with COSMOS

Change History

Workload Estimation

Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term regarding how it is used in this document:

Introduction

The COSMOS vision is entailed in the definition of what COSMOS is - "The world or universe regarded as an orderly, harmonious system". The intention of the project is to apply an orderly and harmonious behavior to the world of system management. Complementing standards such as CMDBf, SML, and Web2.0 technologies are making this vision a reality. The overall COSMOS vision is to provide an extensible framework, based on a set of acceptable standards, to simplify the task of building an ecosystem of existing system management tooling.

Inline with that vision, Nagios can help to not only mature the COSMOS framework but also provide out-of-the-box value to COSMOS users. This two-folded advantage has many positive implications:

1. It is a step forward to evolving an open source code base to a framework that is usable in a production environment
2. Makes the COSMOS project an attractable solution that provides value by its own
3. Simplifies integration of proprietary solutions with Nagios, and
4. Demonstrates a working example of a well-established system management application in COSMOS framework

The next section provides more detail about Nagios.

What is Nagios?

Nagios is a system and network monitoring application that is capable of detecting and notifying abnormal behavior. The definition and monitoring behavior is defined by administrators using a set of flat-file configurations. The files indicate what and how things should be monitored. There are three primary atomic entities in Nagios:

An administrator is required to define hosts, services, and commands to effectively monitor a set of resources. The actual monitoring of a host/service is not done by Nagios. It is instead done by add-on plug-ins that are defined as individual commands. This architecture provides the capability to virtually monitor any aspect of a system that can be automated. There are already many available plug-ins for monitoring common hosts/services in a typical networking environment. Where limited, administrators can write their own plug-in to accomplish the monitoring of an uncommon host/service.

Nagios itself runs on Linux but it is capable of monitoring desktops running Windows via its plug-in architecture. As part of its monitoring solution, Nagios also provides an alerting mechanism that broadcasts a problem to sets of contacts or contact groups. A notification handler can also be registered to take certain actions based on incoming events (e.g. storing status information in an RDBMS). The diagram below, extracted from Nagios documentation, pictorially depicts the components of Nagios:

There is also a web-based UI included that provides reporting and limited administration capabilities. A screen shot of the Nagios web-based UI is included below. The next section describes the scope and the value of this enhancement.

See Nagios user guide to find out more about its capabilities.

Purpose

The purpose of this document is to outline the initial effort in bringing Nagios closer to COSMOS. The integration points and their related value to the Nagios and COSMOS user base will be covered by subsequent sections.

Scope

There are a number of areas where COSMOS can add value to Nagios. The areas can be summarized into three categories:

1. Data Manager Integration
2. Administration Capabilities
3. Reporting

Data Manager Integration

The task of defining the required objects in Nagios is cumbersome, time-consuming, and error-prone. It's usually the case that information required to define objects is stored in other data stores. For example, a subset of configuration items stored in a CMDB can typically serve as the hosts that an administrator may want to monitor. It could also be the case where host information is stored under an asset database.

COSMOS can significantly ease the task of defining objects by providing integration points between data managers and Nagios. The ability of defining objects can be as simple as dragging and dropping a set of queried items from a data manager into the Nagios data collector. There are 10 different object types defined in Nagios:

1. Hosts
2. Host Groups
3. Services
4. Service Groups
5. Contacts
6. Contact Groups
7. Commands
8. Time Periods
9. Notification Escalations
10. Notification and Execution Dependencies

It is often the case where hosts, services, and contacts are defined in other data stores. COSMOS can use CMDBf to define a seamless integration between where the objects are stored and the Nagios monitoring framework. See use cases and implementation detail for more information.

Another area where Nagios can be integrated with COSMOS in the context of data manager integration is via the Common Base Event (CBE) database. COSMOS currently includes a CBE data manager that stores logging events in the form of CBEs. Nagios keeps track of host/service status, alerts, and notifications by logging events. The events are rotationally logged in flat files. COSMOS can contribute a Nagios plug-in that will log events to the CBE data manager for persistent record keeping. The list below is a number of advantages for providing this integration:

1. Events are logged in a common format
2. Exiting reporting capabilities can be reused
3. Other monitoring tools that have similar characteristics can also log events to a CBE data manager
4. Events are persisted in a database instead of flat files

Administration Capabilities

This area of integration ties very closely to the previous one. Once the user selects a set of hosts, services, and/or contacts from other data stores, the Nagios configuration needs to be changed to reflect the new resources being monitored. There is a set of administration tasks that need to be provided for a system administrator to effectively work with the Nagios framework. At a minimum the following administration tasks will be provided:

• Defining objects (hosts, services, contacts, time periods, etc...). The definition of objects may originate from a data store.
• Enable/disable host checks
• Enable/disable service checks
• Restarting the Nagios process

Future releases of COSMOS may consider the following administration tasks:

• Automatic deployment of agents to machines
• Controlling notifications

Where possible, common administration tasks need to be reusable by other monitoring tools.

Reporting

The Nagios web-based UI is far from sleek or modern. It provides a few primitive reporting capabilities:

• Trends
• Availability
• Alert Histogram
• Alert History
• Alert Summary
• Notifications
• Event Log

A BIRT integration with Nagios under the COSMOS framework can significantly add value by providing an extensible mechanism for generating customized reports. Consumers can benefit by tunning reports to target a specific audience set. The reports contributed by COSMOS will use the CBE database as its source but this does not prevent anyone from contributing reports that use Nagios logging information as its data source. Consumers can also redirect Nagios events to a proprietary database registered as a data manager with customized reporting.

COSMOS will need to provide a set of report templates as exemplars of the reporting capability that can be added on top of Nagios-based data. See implementation detail for more information.

Requirements

The following is a list of requirements that falls in the scope of the Nagios/COSMOS integration:

1. The ability to write resource information to the SML repository via the client
2. Viewing Nagios as a data manager in COSMOS framework
3. Initiate and control the monitoring of resources via the client
4. Generating reports on Nagios based events
5. Viewing Nagios resources and their status
6. Storing Nagios events in a CBE database
7. An employee database registered as a data manager
8. A configuration database registered as a data manager

Use Cases

The following use cases outline some of the typical tasks that COSMOS users will perform to accomplish an objective.

Use Case 1: Adding a machine to the asset database

Assumption: The COSMOS framework is successfully installed with the asset repository MDR

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the asset database and selects 'Define Object'. A form is displayed under the details pane for the user to populate the required fields. The form can contain multiple pages that cleanly break down the flow of user actions.
3. The fields are populated and the 'Finish' button is pressed.
4. Client will indicate that it is writing the data to the database. The user is either prompted with an error message or a confirmation message to indicate success. In case of user error, the form is returned to be corrected.

Use Case 2: Monitoring of a host in Nagios

Assumption: The COSMOS framework is successfully installed with the asset repository MDR and the Nagios data collector.

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the Nagios item displayed under the data manager navigator and selects 'Define Object'. A form is displayed under the details pane for the user to populate the required fields. The form can contain multiple pages that cleanly break down the flow of user actions.
3. User selects the type of object that is to be defined, which in this case happens to be a host. The form is populated by all items from registered data managers that are candidates of the object type selected. In this case all resources that are candidates of being monitored are queried from the asset database and displayed in the details pane.
4. Where possible items of corresponding data stores are displayed to make it easier for a field to be populated. For example, the items of an employee database can be displayed for the contact/contact group fields of a host definition.
5. The user either has the option of selecting a discovered host or populate the fields manually. In the case where a discovered host is selected, the fields of the form are populated based on the selected host.
6. User clicks 'Finish' to finalize the process of initiating the monitoring process of the host

Use Case 3: Monitoring of a service defined with Nagios

Assumption: The COSMOS framework is successfully installed with the asset repository MDR and the Nagios data collector.

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the Nagios item displayed under the data manager navigator and selects 'Define Object'. A form is displayed under the details pane for the user to populate the required fields. The form can contain multiple pages that cleanly break down the flow of user actions.
3. User selects the type of object that is to be defined, which in this case happens to be a service. The form is populated by all available services on the associated host. For example, a configuration database can be queried by the client to retrieve all available services on an associated host.
4. Where possible items of corresponding data stores are displayed to make it easier for a field to be populated. For example, the items of an employee database can be displayed for the contact/contact group fields of a host definition.
5. The user either has the option of selecting a discovered service or populate the fields manually. In the case where a discovered service is selected, the fields of the form are populated based on the selected service.
6. User clicks 'Finish' to finalize the process of initiating the monitoring process of the service

Use Case 4: Viewing the status of hosts/services being monitored

Assumption: The COSMOS framework is successfully installed with the Nagios data collector. It's assumed that one or more host/service is configured for monitoring.

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the Nagios item displayed under the data manager navigator and selects 'Display Monitoring Resources'. A tree of hosts and services are displayed in the details pane with corresponding icons that indicate the last status check of a host/service. See use case 5 for details about finding more information about a failed host/service.

Use Case 5: Determining the problem associating with a host/service

Assumption: The COSMOS framework is successfully installed with the Nagios data collector. It's assumed that one or more host/service is configured for monitoring and at least one host/service is down. The context of the use case is the status navigation tree.

1. User right clicks a host/service that is indicated to be down and selects 'Display Information'
2. A BIRT report is generated to display the events of the selected host/service that led to its downtime.

Use Case 6: Generating reports based on host availability

Assumption: The COSMOS framework is successfully installed with the Nagios data collector.

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the Nagios item displayed under the data manager navigator and selects 'Generate Report > Availability'. A BIRT report is generated and displayed to show the general availability of hosts and services being monitored.

Use Case 7: Removing a host being monitored

Assumption: The COSMOS framework is successfully installed with the Nagios data collector. It's assumed that one or more host/service is configured for monitoring.

1. User opens a browser and points to the URL of the COSMOS client
2. User right clicks the Nagios item displayed under the data manager navigator and selects 'Display Monitoring Resources'. A tree of hosts and services are displayed in the details pane.
3. User right clicks a host and selects 'Remove'. The user is prompted with a message that indicates the implication of removing a host.
4. The user clicks OK to proceed. The host is removed from Nagios and the status navigation tree is updated to reflect the user action.

Implementation Detail

Integrating Nagios will span features across three sub-projects: Data Collection, Resource Modeling, and Data Visualization. The features can be categorized into three different areas:

Nagios Data Manager Package
Resource Modeling MDRs
Nagios Client

The diagram below displays the interaction between the COSMOS and Nagios components:

Nagios Data Manager Package

A package will need to be included to register the Nagios server as a data manager with COSMOS framework. The package will need to:

1. Discover the management domain and register itself as a data manager with the advertised brokers
2. Discover the CBE data manager to determine its end point. This maybe code that needs to run periodically until the CBE repository registers itself with the management domain. Keep in mind that there is no ordering of how data managers are registered. There is always the possibility of the CBE data manager registering itself after the Nagios data manager.
3. After discovering the CBE data manager, activate a Nagios plug-in that will redirect all events to the CBE repository. This step will require Nagios to be reconfigured and restarted. The user will need to be prompted before restarting the Nagios process.

The Nagios packaging code base is expected to be checked into the Data Collection subproject.

Resource Modeling MDRs

As part of illustrating the seamless integration of multiple MDRs with a system monitoring application, there will be two additional MDRs added to the Resource Modeling subproject. The asset repository will also need to be modified to ensure a smooth integration. The two new MDRs will be:

1. Configuration MDR - contains configuration detail about what is stored and running on a host
2. Employee MDR - contains information about staff members

The first database will be used to discover services that can be monitored on a specific host and the second database will be used to display a list of employees that can be included in the contact list of a host or service definition. Both MDRs are expected to be implemented on top of the SML repository which already includes a CMDBf query capability. The SML repository code will need to be refactored to extract out any code that is specific to the asset repository.

Nagios Client

The data visualization subproject is expected to contribute the following functionalities:

1. The ability to define hosts and services
2. The ability to generate reports on Nagios events, notifications, alerts, and etc...
3. The ability to write objects to the asset, configuration, and employee MDRs
4. Nagios specific views to visualize the status of the monitored objects

Task Breakdown

The following section breaks down each individual task based on each subproject. Symbols are used to indicate the enhancement that each work item falls under.

Resource Modeling

Φ Refactor any code necessary to provide write capability to the asset repository
Φ Refactor the data center SML model to make it fit better with the resource model that Nagios uses
Φ Refactor the CMDBf query code for the asset based repository to provide any additional queries that the client will need
Φ Provide a model mapping from the asset model to the Nagios model
Ψ Refactor the SML repository code to provide a common plug-in that multiple SML based repositories can use
Ψ Provide an employee based model using SML
Ψ Extend the SML repository code to provide an employee database
Ψ Provide a CMDBf query implementation for the employee database
Ψ Provide a model mapping from the employee model to the Nagios model
Ω Provide a configuration based model using SML
Ω Extend the SML repository code to provide a configuration database
Ω Provide a CMDBf query implementation for the configuration database
Ω Provide a model mapping from the configuration model to the Nagios model
β Use the programming model to plug-in the employee MDR into COSMOS framework
β Use the programming model to plug-in the configuration MDR into COSMOS framework

Enhancements:
Φ [Nagios]Generalize the asset repository and the data center model
Ψ [Nagios]Provide an employee MDR based on the SML repository
Ω [Nagios]Provide a configuration MDR based on the SML repository
β [Nagios]Add the employee and configuration MDRs to the COSMOS framework

Data Collection

Φ Define a mapping between Nagios events and CBE events
Φ Provide a Nagios plug-in to forward events to the CBE data manager
β Provide a mechanism to register a Nagios server as a data manager
β Provide administrative capabilities that the client can invoke

Enhancements:
Φ [Nagios]Provide a Nagios plug-in to log events as CBEs to the CBE data manager
β [Nagios]Register a Nagios monitoring server as a data manager

Data Visualization

Φ Provide actions to write to the asset MDR
Φ Provide the forms necessary to write data to the asset MDR
Φ Provide actions to write to the employee MDR
Φ Provide the forms necessary to write data to the employee MDR
Φ Provide actions to write to the configuration MDR
Φ Provide the forms necessary to write data to the configuration MDR
Ψ Provide actions to define objects on the Nagios data manager
Ψ Provide the forms necessary to define the Nagios objects
Ψ Provide actions to perform administrative tasks on the Nagios data manager
Ψ Define a framework that allows for an MDR to be replaced/added as part of defining objects for Nagios
Ω Provide a navigator that displays the status of hosts and services monitored on Nagios
β Provide reporting capabilities for viewing host/service events
β Provide two general reporting capabilities on Nagios events (e.g. availability and alert history)

Enhancements:
Φ [Nagios]Provide write actions and forms for the asset, configuration, and employee MDR
Ψ [Nagios]Provide actions and forms for defining Nagios objects
Ω [Nagios] Provide a status navigator for the Nagios data manager
β [Nagios] Provide reporting capabilities for the Nagios events

Open Issues/Questions

All reviewer feedback should go in the talk page for Nagios integration with COSMOS.