Monitoring Basics 103 | Building a NOC | Portland, Oregon NOC designs

The "No Network is 100% Secure" series
- Monitoring Basics 103 -
A White Paper

All rights reserved - may not be copied without permission
Easyrider LAN Pro, NOC Design Consultants

Contact Us

Background Information
Monitoring Basics 101
Monitoring Basics 102
NOC Design Primer
Types of monitoring
The pitfalls of self-monitoring
Monitoring Benefits
Remote Monitoring History

So you've decided to invest in a NOC: Building and maintaining a NOC requires a substantial and sustained emotional and financial commitment. If you are not prepared to provide both, it may be wise to revisit your plans before writing any checks.

The road to failure: In previous white papers, we have described the process that most failed NOC deployments follow. If you've ever managed a NOC project that failed, you know the pitfalls and mistakes as well as we do.

The road to success: Designing, building and maintaining a NOC is in many ways similar to the typical software development process. There are differences, of course. But following standard best practices would be a good first start in ensuring that your NOC project doesn't jump the tracks. This includes tasks such as developing a requirements statement, selecting products that support those requirements, developing a design specification and then tracking your project's progress. Many projects that have failed fail because there was never a collective understanding and agreement on what would be built and how the NOC will operate (use case). Others failed because purchasing the monitoring software was done before the NOC design requirements were thought about or understood. And lastly, even if the above steps are taken, your NOC should be built by someone who has a LOT of experience building comprehensive monitoring environments. Assigning this task to an already overworked admin to "save money" makes absolutely no sense to us. Nor does hiring an inexpensive rent-a-tech. The old axiom certainly holds true here: the poor quality of your NOC and it's inability to see even major outage events will be remembered long after you've forgotten about the money you saved by having it built by someone who didn't know what they were doing.

Fully half of the NOC projects we've worked on were started and botched by someone else first. And in most cases, the money spent on those false starts is money (and wasted time) that went right down the drain.

What is a NOC?: Network Operations Centers (NOC) provide 24x7 visibility with Remote Monitoring and Management capability for routers, switches, Microsoft Windows servers, Linux/Solaris/HPUX, *NIX/etc Servers including Applications. This allows you to proactively manage computing service delivery for businesses that come under purview of regulatory compliances such as HIPAA, SOX, and so on.

Why build a NOC?: A NOC will improve the productivity of your IT staff because a well designed and operated NOC will either correct or intelligently escalate the information it receives from the monitoring environment. Network and Systems Administrators can focus on reported Critical events, problem analysis, steps for resolution and long term planning.

NOCs provide a competitive advantage allowing you to commit to a higher level of accountability than your NOC-less competition. Your computing services can be delivered under stringent SLAs which may not be available to Smaller, less proactive Businesses.

A NOC will improve the quality of life for you and your staff by having less disruptions over night, during weekends and over the holidays. No more 2am false alarm pager alerts for you!!!!

Things to consider if you are planning on building a NOC: The most important consideration is your budget. Over my career I have talked to many IT managers who had Cadillac tastes and a Chevrolet budget. Costs will certainly vary depending on the level of coverage and monitoring comprehensiveness you have in mind. But as a general rule I would say that if you don't have an IT budget of at least $1 million bucks per year, building a NOC may not be for you. Managers considering building a NOC are most likely responsible for between 50 and 250 servers and associated networking gear. Data centers with more than 250 servers most likely already have NOC monitoring in place. If not, you are definitely behind the curve.

A very basic, entry level NOC is going to cost in the neighborhood of $150,000 to $250,000 to do the initial deployment. This assumes that there is already a NOC room available for use. If not, you would need to add that facility expense along with the cost of NOC furniture, workstations and so on.

A good planning purpose estimate would be $125,000 for the monitoring software platform (including a server to run it on) plus $1,000 x # servers to be monitored. Again, note that these estimates are to build a VERY basic monitoring environment with very few features, capabilities, bells and whistles. Managers can easily spend $500k to $1 MM in just software alone, depending on what types of capabilities they want. Proactive monitoring is more expensive to build than reactive monitoring. Predictive monitoring is more expensive still. Add to that trouble ticket software, asset management capabilities, capacity planning features, applications monitoring plug-ins and you are talking real money.

We won't go into great detail regarding staffing costs except to include it as a consideration. The cost to staff your NOC will depend a lot on whether your NOC will operate reactively or proactively. And whether the Techs will be "first responders" who will try to fix problems that they see or whether they will just observe and report. And also the number and degree of specialization each NOC tech will have. Clearly, hiring Techs who will do little more than make a phone call if an icon changes color will be less expensive than staffing with Administrator-class Engineers. However, it is our opinion that managers are wise to leverage their NOC investment by staffing it with folks who can actually investigate issues, troubleshoot symptoms and who will ultimately correct a large percentage of the problems that they see. Otherwise, the substantial financial investment to build a NOC may not be worthwhile.

Understand your NOC design objectives: Do you want to deploy a NOC in support of increasingly aggressive SLAs? Do you need a NOC because you are experiencing too many undetected service outages? Do you need a NOC to protect your revenue stream or to make regulators happy? All laudable goals but it's important to understand what these objectives are when the NOC is being designed and built. As an example, if a Government agency that regulates your industry (let's say you are a power generation facility) decrees that you must not have any outages that go undetected for longer than 15 minutes, this will have a great influence on the monitoring software selection process. It will also effect how that software and the alarms that it sends are configured and what kinds of information are presented to your NOC Techs.

And we haven't event talked about security monitoring which will add additional layers of cost and complexity to your NOC design plans.

Your NOC operational plan: What process will be employed to patch and upgrade monitoring server software, agent software, plug-ins, templates and so on? How will new equipment be added to the monitoring environment and how will retired/legacy equipment be removed? What is your disaster recovery plan? How will new features and capabilities be added to your monitoring environment? How will mistakes be corrected? What's the upgrade plan for when the monitoring server hardware needs to be replaced? What fault tolerance or fail-over provisions need to be put in place in case something fails? How scalable do you want your NOC to be? What will your plan be for performing maintenance in a 7x24x365 environment?

There are no "right answers" to these questions. But the issues need to be thought about and processes put in place, ideally before the NOC is even built. Easyrider LAN Pro can help identify options and is experienced in negotiating sensible compromises.

Easyrider LAN Pro has decades of experience guiding these discussions and processes. Hopefully by now the astute reader has come to understand that, contrary to monitoring software salespeople's hyperbole, the tasks of designing, building, operating and maintaining a NOC is a lot more involved than simply spinning up a CDROM and typing ./setup

A sensible, well thought out plan is critical to the success of any NOC deployment. If you are going to spend this kind of money building a NOC, the likely non-technical executive staff who authorized the expenditure are going to expect the monitoring capabilities to work wonderfully and "as advertised". You certainly don't want to disappoint them by purchasing software that won't do the job or by deploying the software in a manner that does not utilize the maximum product capabilities, right?

Engaging the services of experienced NOC designers like Easyrider LAN Pro will pretty much guarantee the success of your project. Of course if failure is an acceptable option, feel free to save a few bucks and hand over your high risk, high visibility project to the kid who mows your lawn or to someone else who's never built a NOC before. If the people you are currently talking to haven't been building monitoring environments for at least 30 years you may want to at least get yourself a second opinion.

Next in the security white paper series:

How Cyber Criminals will mature over the next ten years
Are you vulnerable to drive-by exploits?
High value sites recent hacks
IT employment challenges of the 21st century
Employment reference checking white paper
Competency Certifications White Paper
Firewall White Paper
Virus White Paper
GhostNet White Paper
Password White Paper
Digital Identification Certificates White Paper
Cryptography White Paper
OpenID White Paper
Intrusion Detection Systems IDS White Paper
Rootkit White Paper
Unnecessary Windows XP Services White Paper
Scareware White Paper
Exaflood Internet Brownout White Paper
Cloud Computing White Paper
Proxy Server White Paper
Personal Computer PC Security White Paper
Phishing White Paper
DNS Poisoning White Paper
Conficker White Paper
SPAM White Paper
Best Practices White Paper
Denial of Service DoS White Paper
Trojan Virus Attacks White Paper
Port Scanning White Paper
Monitoring Basics 101 White Paper
Monitoring Basics 102 White Paper
Virtual Machine Security White Paper
Aurora vulnerability White Paper
Shelfware White Paper
Outsourced IT White Paper

Easyrider LAN Pro Consulting services:

Network Security Audit and PC Tune-up service

- Proxy server installation and configuration
- Enterprise security consultations
- Disaster recovery planning
- Disaster recovery services
- Capacity, migration and upgrade planning
- Build and deploy central syslog server
- Build trouble ticket systems
- Design and build monitoring environments
- Design and build Network Operations Centers (NOC)
- HP Openview, BMC Patrol consulting