Reliable Response

David Rudder

August 22, 2005

CLUE - North

Monitoring, Responding and Disaster Recovery

Network Maintenance with Nagios

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Quick Introduction

David Rudder

President of Reliable Response, LLC

Makers of Reliable Response Notification

Veronica Proksch

Sales for Reliable Response, LLC

Contact Info on Last Slide

Reliable Response

David Rudder

August 22, 2005

CLUE - North

The Role of Monitoring

Alert you before the client does

Provide information on what caused the outage

Also known as “Root Cause Analysis”

Allow quick recovery

Provide forensics

Provide accountability

Reliable Response

David Rudder

August 22, 2005

CLUE - North

The Types of Monitoring

2 Flavors

System Monitoring

Application Monitoring

Reliable Response

David Rudder

August 22, 2005

CLUE - North

System Monitoring

Allows you to know if a machine is up

Quick, non-intrusive, repeatable

Uses SNMP, Ping, WMI, etc

Gives only basic information. “Host down”

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Application Monitoring

Runs a whole transaction

Typically runs less frequently

Uses the application's protocol

HTTP, WAP, SOAP, RPC

Much more taxing on the server

Much better root cause analysis

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Advanced Monitoring Features

Network Discovery

Enterprise scope

GUI Capabilities

Notification and Escalation

Integrations

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Discussion

Who maintains complex networks?

Who uses a monitoring application?

Which ones?

Discuss some of the good experiences and bad

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Nagios – Open Source Monitoring

Descendant of NetSaint

Active Development, Active Participation

C based core

Perl and Shell based monitors

Plugins for System and Application Monitoring

Not Enterprise class, but good enough for most

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Enterprise Monitoring

What Nagios is Lacking

Can not monitor thousands of nodes

Network Discovery

Reconfiguration without Restart

Deep integration with Enterprise products

LDAP integration

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Enterprise Monitoring

What Nagios Has

Custering, fail-over, distributed operation

SNMP Support

Deep integrations with some products

Dashboards

Reporting

Notification and Escalation

Dependencies

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Discussion

What monitoring features are most important to you?

How many computers do you monitor?

Are you in a geographically-disperse company?

How important is uptime to you?

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Notification and Response

Monitoring is nice, but useless without some way to tell the admins that something is wrong.

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Full Disclosure

Reliable Response makes a Notification server. We have strong opinions in this area.

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Notification Methods

Most simple: Send an email

Add in pagers, IM, SMS

Add group notification

Add escalation

Add auditing, reporting, and display

Add post-emergency updates to management and clients

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Response Methods

Most simple: No response is possible

Add acknowledgment via webpage

Add comments via webpage

Add transmission of acknowledgments and comments to other interested parties

Add an IVR, email input or other OOB inputs

Add responses and check status directly from mobile devices

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Discussion

Is notification and response important to you?

Do you have problems with non-response to emergencies?

Do you have problems with multiple response to emergencies?

Horror stories?

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Recovery

Recovery means

Allowing the business to continue during the emergency

Fixing the problem

Restoring any lost data

Making up for lost time

Studying what went wrong

Studying how to improve response in the future

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Business Continuity

Most outages do not cause the business to halt.

Some cause a partial outage of business functions

Some cause major outages

In the worst case, business functions are halted completely

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Business Continuity

Monitoring improves the chances that more business functions can continue, because the problem is found and solved quicker.

Notification allows IT to warn non-IT business units about problems, allowing them to come up with continuity plans

But, all businesses should have a BC plan in place

Reliable Response

David Rudder

August 22, 2005

CLUE - North

BC Plans

How to function without

Telephone service

Data/Internet

Data access

Power

Transportation

Reliable Response

David Rudder

August 22, 2005

CLUE - North

BC Plan Examples

Include cell phone number on business cards

Provide a local cache of mission critical data in remote offices

Provide a line of credit that can be used by stranded sales employees

Emergency power backup

Bridge lines dedicated to emergencies

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Discussion

Who has BC plans?

Who has needed BC plans?

Has 9/11 changed your BC plans?

Has new technology changed your BC plans?

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Emergency Recovery

Fixin' It

Monitoring helps fix problems by

Providing quick alerting

Root cause analysis

Cross-team alerting

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Emergency Recovery

Fixin' It

Notification helps fix problems by

Aiding the alerting by reliable notification

Manual cross-team alerting. “Get me a DBA!”

Tracking of responders, preventing duplication

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Recovering Data and Time

Monitoring tells you what systems were affected.

Some errors may be hidden. After the emergency, finding what broke is important.

Often, cron jobs will be past, or deadlines missed.

Recovering from these problems are generally a manual problem.

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Forensics with Nagios

Monitoring and notification can tell you what went wrong and how to fix it.

Nagios reports

Nagios logs

State Stalking

Performance data

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Forensics with Notification

Unfortunately, Nagios doesn't log notification responses.

Reliable Response Notification will log the notifications sent and responded to.

Allows forensics to track response times. Cross-check the notification times with the state stalking and performance data

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Discussion

Experiences doing forensics

Has your process ever improved due to forensics?

Does “the real world” intrude?

Reliable Response

David Rudder

August 22, 2005

CLUE - North

Contact Info

Reliable Response, LLC

1600 Broadway

Suite 2400

Denver, CO 80202

David Rudder

Technical and President

303-542-1990

david@reliableresponse.net

Veronica Proksch

Sales

303-542-1989

veronica@reliableresponse.net

2005-08-06T15:01:47