The Gotchas of Monitoring Infrastructure in a Dual Data Center Environment

Lets assume for the remainder of the article
  that an organization hosts dual data centers that are geographically separate in order to support business continuity in the event that one data center goes down. I have creatively called them Data Center 1 and Data Center 2.



Lets also assume that these data centers service corporate and regional offices around the country.



From a network perspective the visibility between Data center 1 and regional offices does not ensure visibility between Data center 2 and the regional offices. In other words, just because you can reach a regional office from one data center doesn't mean you can reach it from the other.



This is a problem of perspective. If your monitoring tool is located in data center 1 then your availability worldview is relative to that location. But what would happen if data center 1 dropped off the air?



Data center 2 may have well taken up the slack by servicing users, but from a monitoring perspective it appears that the entire network is down because the monitoring tool, sitting in the broken data center 1, cannot connect to anything.



The consequence of that problem can be expensive. If that monitoring station is collecting availability statistics for service level agreements or similar contractual arrangements then it will incorrectly report an enterprise wide service failure that may result in a breach of service agreements that in turn invoke penalty clauses.



The opposite of the aforementioned problem is also true. Another inaccurate measurement that a single monitoring station perspective can produce is reporting data center 2 as available when it is not.



Let me explain, if data center 2 loses connection to the regional offices but maintains a connection to data center 1 then everything looks ok from the monitoring tools' perspective. This is because the monitoring station (located in data center 1) can see the infrastructure in data center 2 and thus reports service availability. But in reality the organization has lost its data center redundancy because the connection from data center 2 to regional offices has been severed.



One monitoring station per data center addresses these problems. Each monitoring station provides its unique perspective on the reachability of regional infrastructure relative to its own geographical location. But the collation of that data for examination by network monitoring staff becomes more complex.



The reason it is more complex is because each regional site is reported on twice. Once from the datacenter 1 perspective and once from datacenter 2 perspective. This problem can also be solved using a hierarchy monitoring model where each regional site is a group and that group contains an input from both locations.