A couple of weeks ago we posted an interview DLT’s Chief Cloud Technologist David Blankenhorn did with fedScoop. The interview focused on open source software for government and included a question on how DLT supports it. To answer the question, David mentions our long time partnership with Red Hat.

Recently, fedscoop posted an interview with Red Hat Public Sector’s Chief Technology Strategist Gunnar Hellekson. In this interview Hellekson provides his own insights on the importance of open source for government. In addition, Hellekson talks about Red Hat’s history in open source and gives a recommendation for those interested in exploring open source options.

If you have questions about open source in general or are looking for detailed information on how DLT can support your open source initiatives, drops us a line in the comments section or on our information request page.

Some OpenStorage technologies,  support out-of-band replication in which the contents of the disk storage are replicated between devices. In these instances, NetBackup has no knowledge of the data, which complicates recovery in a NetBackup protected environment because the replicated data can only be accessed after recreating the NetBackup catalog. Replication is done by importing the entire contents of the disk storage at the remote site using the bpimport command. The nbcatsync utility, introduced in NetBackup 6.5.6 and 7.0.1, can address this challenge as well, but it relies on being able to restore the catalog from a catalog backup and then post-processing it to reconcile the disk device mappings, resulting in a very time consuming process.

Fresh AIR

Recently Symantec introduced Auto Image Replication (AIR) in NetBackup 7.1. The use of Storage Lifecycle Policies (SLP) and optimized duplication between disk pools allows disk-based backups to be replicated between devices under NetBackup control. Prior to 7.1, this process had been limited to the NetBackup master server from which they originated. With NetBackup 7.1, site-to-site disaster recovery –whether the site is two rooms in the same data center or separate data centers across the country — can now be done using two separate masters with an automatic process once the SLP has been configured.

AIR in Action

In order to use AIR, suitable disk storage devices must be configured in the source and target domains. In the NetBackup 7.1 release, the AIR feature only supports the Media Server Deduplication Pool (MSDP) to duplicate backups between domains. When storage vendors update their plug-ins to the Open Storage API, this capability will extend to those appliances.

AIR works by duplicating backups from a disk pool in the source domain to a disk pool in the target domain. The replication operation requires two SLPs, one in the source domain and one in the target domain, of which both must have the same name. The SLP in the source domain is associated with the backup policy and controls the writing of backup and the subsequent duplication to the target domain. The SLP in the target domain is not associated with a backup policy but is invoked by an alerting mechanism when a new image –duplicated from the source domain — is detected. This SLP runs the process to add the information about the backup to the target domain and can also be configured to duplicate the backup to other storage locations in the target domain.

An AIR backup image is different from a normal NetBackup backup image. Once the backup has completed, the associated image database information (the part of the NetBackup catalog that lists what the backup actually contains) is appended to the end of the backup image before it is duplicated to the target domain. When a new backup is detected in the target domain, this information is read from the backup and used to populate the target domain’s NetBackup catalog. This information only exists in the source domain copies and the initial copy in the target domain and is not included in any subsequent duplicate copies created in the target domain.

To understand this process, reference the diagram below:

The figure shows the sequence of events in an AIR backup with the duplication operation broken down into four discrete steps:

• Step 1: The backup is written to disk storage in the source domain using a backup policy with an SLP configured for AIR. When the backup completes the catalog, the data it generates is appended to the end of the backup.
• Step 2: The backup is duplicated to the target domain across the WAN (or LAN).
• Step 3: The storage device in the target domain alerts the target master server to the fact that a backup has been duplicated to it. This triggers the receiving SLP to run a “fast import” operation in which the catalog data transferred from the source domain is added to the target domain’s catalog.
• Step 4:  The receiving SLP in the target domain can now duplicate the received backup to any desired location for storage – such as creating a tape for long term retention.

Photos courtesy of TreeHugger.com and 42U.com

For more information on how DLT can support your open source initiatives, please visit our here.

An agency’s computer system is under constant cybersecurity threats from several factors.  While many of them are intentional, such as fraud and theft, there are also the unintentional errors and omissions that threaten a systems security. Let’s take a closer look at some examples.

The Intentionally Malicious

Information technology is increasingly used to commit fraud and theft.  Computer systems are exploited in numerous ways, both by automating traditional methods of fraud and by using new methods.

Unfortunately, insiders who are authorized users of a system perpetrate the majority of the fraud uncovered on computer systems.  Since insiders not only have access to, but are also familiar with the victim computer system (including what resources it controls and where the flaws are),  authorized system users are in a better position to commit crimes.  Former employees may also pose threats, particularly if their access is not terminated promptly.

The Good, the Bad and the Unintentional

End users, data entry clerks, administrators and programmers frequently make unintentional errors that contribute to security problems.  Sometimes the error is the threat, such as a data entry error or a programming error that crashes a system.  In other cases, installation and maintenance errors can create vulnerabilities, which are weaknesses that allow an attacker to reduce a system’s information assurance.  Not learning from past mistakes, leaving a bug in the software, browsing harmful sites on the Internet and poor password management can all lead to security threats.

Implementing a Policy

To set up a proper security policy, you have to determine the level of threat to protect against, what risks are acceptable and how vulnerable your system will be as a result.  Risk is the possibility that an intruder may be successful in trying to access your computer, the possibility of an error or a malicious program entering your system and causing direct or indirect damage to your business and business processes.

Because of threats like these, an agency should address some important questions like “what is at stake if someone breaks into the system” and “how much time would it take to retrieve or recreate and data that was lost?” It is very important to develop a consistent, simple and generic policy for your system that users can easily understand and follow.  It will have to protect the data you are trying to safeguard.  The policy will have to state who has access to the system, who is allowed to install software, who owns what data and so on.

Photo courtesy of SecurityAffairs.co

• Bring the power of Google search to your enterprise data – behind the firewall.
• Promote secure collaboration with Google Apps: web-based email, calendar, documents, spreadsheets, presentations, and more that are accessible via cloud anywhere, anytime.
• Develop and visualize a common operating picture with Google mapping tools – easily deployed even in remote and mobile environments.

Date: Wednesday, January 25, 2012
Time: 8:00am – 2:00pm
Location: Cheyenne Mountain Resort
3225 Broadmoor Valley Road
Colorado Springs, Colorado 80906

To register and for a detailed agenda visit: Google Tech Day at Cheyenne Mountain Resort

Well, let me dig through my notes and see what I did about it…
Ah, yes! I remember when a customer ran into a Status Code 2 error when running an SQL database backup. In my investigation to help them fix the problem, I stumbled across a Symantec support article specifically focused on this issue.

After reading this tech note, it became crystal clear the problem that my customer had was that the credentials did not have administrative privileges, therefore it was failing. You see, Netbackup requires that you use an account with admin rights in two places, but my customer only had them in one.

The Symantec article provides a detailed solution to the problem. But in a nutshell, remember to include administrative rights in two places: NetBackup Client Service Properties and Set Database Login Perimeters Properties. If both are not entered correctly, you surely get that ol’ Status Code 2.

‘Til next time!

Operations management aims to keep failures to a minimum while increasing efficiency.  The systems we manage are complex chains of interconnected processes that sometimes fail.  The measure of how often these failures occur is known as the mean time between failures (MTBF). In a complex chained system, MTBF is combinatory; the mean for each component decreases the availability of the overall system.  Parallel systems, unlike chained systems, have higher availability if the system is considered available when one of the two parallel members is up.  Users want their MP3’s to play, their photos to print, their updates to reach their friends and families.  The “nines” expresses availability as an amount of allowable downtime over a time period; how long users will put up with no lights or dial tone.  “Two nines” would be 99% or about 4 days and “five nines” would be 99.999% or about 5 and half minutes a year.

Regardless of the system reliability math, the “belt and suspenders” mentality that pervades operations is based on the driving theme of uptime. Just about everything we do is to mitigate, prevent, manage or discover failures or failure-inducing conditions. Standard operating environments reduce the variables that could cause issues in practice or troubleshooting.  Automation reduces human interactions that can introduce drift.  Monitoring alerts us to changes from the expected system behaviors.  Written methods of procedure ensure a clear understanding of actions taken during a maintenance window.  Backups provide recovery from catastrophic failures.  High availability clustering provides parallel environments to increase our reliability.

Recognize that failures are inevitable. The only thing we can control is our response.

Changing our vantage point

I’d like to offer up a challenge: where we’ve been architecting around reliability, we should be building for recovery.  Mean time to recover (MTTR) is the measure of restoring service, which quite frankly, is more important than eliminating potential failure points.  Service Level Agreements (SLAs) and, perhaps more importantly, the corresponding penalties are based on downtime metrics, not uptime. This makes recovery time our keystone for designs, not reliability. A simple system with fewer components that fails once a month and takes five minutes to restore service has the same uptime ‘score’ as a highly complex fail-over architecture that accumulates twenty 30-second interruptions over the same period.

Taking the Conversation Off Road

Changing to a service resiliency model instead of a failure survivability model impacts the choices we make in architecture and our tooling.  Let’s look outside the software arena and examine two prominent auto manufacturers who approach this idea in different ways: Jeep and Rolls Royce.

Designed for the US military, Jeep was mainly built using off-the-shelf automotive components as an all-terrain recon vehicle.  The design allowed for quick modification and repair so that a modern Jeep can be disassembled and reassembled by an Army drill team in under four minutes. Showy perhaps, but it gets to the core of the design. Given the use case, part failure is inevitable, so Jeeps need to be easily recovered.

On the other hand, the iconic, British luxury car Rolls Royce is designed around long duty cycles, for less harsh conditions. Specialized electronics, engine and interior components are built around increasing lifecycle reliability.  While this can result in long and expensive repairs, Rolls Royce has earned a brand reputation for the highest quality of the parts.

Software Engineering Tools

From the design table to the shop floor, it’s clear these two automotive icons have major differences.  For example, tolerances on parts and assembly, material choices, methods and tools used during assembly are widely varied based on the guiding choice of recovery versus reliability.  The automotive engineering analogy holds true in software engineering as well.  We care about different things if a single component failure in the chain doesn’t bring our application to a grinding halt.  These design choices also change as new technologies emerge. For instance, virtualization provides new options for recovery and resiliency while cloud computing offers a similar, but distinct set of options and challenges.

Many of the tools remain the same, but their application to our design choices will change.  Monitoring will alert us to problem components that need to be investigated and returned to the pool of available resources.  Automated configuration management system can correct drift without human intervention.  Increases in load can be handled elastically in response to load balancers within pools of available resources.

There are other factors to be sure; inefficient processes for getting the right people involved probably wastes the most time in an outage.  But a recoverable system that allows techs to work around and repair the failed component without impact to service availability will tack on that extra 9 much faster.

Sounds like the cloud, neh?

What is validation?
The SCAP Program is responsible for maintaining established standards and ensuring that validated products comply. Validation is achieved through proving that the testing performed by the laboratory has been carried out correctly.

Who does independent testing?
Test results for validation are accepted from laboratories that are accredited by the National Voluntary Laboratory Accreditation Program (NVLAP). This accreditation is earned after full review of the laboratories’ Quality Management System (QMS) and passing of the technical proficiency tests.

Who needs to validate their products under SCAP?
Validation is required for vendors of security configuration management, vulnerability testing, and other security auditing tools who wish to sell products in the U.S. Government market under the Federal Information Security Management Act (FISMA) requirements or to commercial customers who have adopted the standard’s requirements.

Why do vendors need independent testing?
Independent, third-party testing assures the agency that the product meets the NIST specifications.  The SCAP standards can be complex and several configurations must be tested for each component and capability to ensure that the product meets the requirements.  An accredited, third-party lab provides assurance that the product has been thoroughly tested and found to meet all of the requirements.  In essence, a third party confirms a vendor’s claims of SCAP-compliance, providing the vendor credibility in an agency’s opinion.

Are there any fees or licensing restrictions associated with SCAP checklists and Test Procedures?
There are no licensing fees or restrictions associated with the SCAP content hosted through the National Vulnerability Database (NVD). Vendors, government agencies, and other organizations are encouraged to use this SCAP content for whatever purposes they envision, including as a source for SCAP-capable tools. Note that SCAP enumeration data is derived from open standards.

Have all vendors who advertise “SCAP-compliant” for their product implemented the SCAP standard in an identical manner?
Buyers are encouraged to research “SCAP compatible” products and services thoroughly before investing in them. NOTE: not all products have fully implemented every SCAP standard. NVD provides a list of SCAP validated products. Now that the standard is in force, neither SCAP-Compliant nor SCAP-Compatible will meet FISMA requirements. Instead, the product must be NIST validated for the components and capabilities that you need.

How long does it take to get an SCAP-compliant product validated?
The time it takes to complete testing and validation depends on several factors. Assuming there is a completed product that conforms to the specifications, laboratory testing time can still vary from as little as two weeks up to several months (or more).

Once all testing is complete and the report is submitted to NIST, the validation can be issued in a few weeks.

What are the requirements for validation from the vendor’s side?
Typically, independent SCAP validation laboratories need all of the following to complete the mandatory tests given in the derived test requirements:

• A contract and Non-disclosure Agreement
• A list of the SCAP test requirements given in the Derived Test Requirements of the SCAP standard.
• Access to the product to be tested, and its documentation

How much does it cost?
The cost of conformance testing and validation varies with the readiness of the product, the nature of the product, previous analysis, evaluation of versions of the product, and the requested timeline.

As a NIST-sponsored effort for both automated provisioning and continuous monitoring, SCAP implementation can save time and money by measuring and automating detection using open standards, finding vulnerabilities, and then offering methods to score those findings. Next month, we will review the six underlying standards of SCAP.

As agencies begin looking at cloud initiatives, the challenge is implementing a continuous monitoring program that reduces risk and ensures compliance with NIST and other relevant guidance in an environment of decreased control. The solution begins with knowing where compliance ends and risk begins.

The Game of Risk

Risk is an operational prerogative– the level of risk an agency is willing to take within a given situation or even as an operational baseline is subjective. For all of their complexity, the NIST SP 800-X series documents only provide guidelines by outlining control families, processes and reporting procedures for proving due care and diligence. SP 800-60 V1 & V2 outline processes for determining information type and the security category (low, med, hi) for systems, but we all know that risk is a trade-off between availability, integrity and confidentiality. Despite all of the guidelines, in the end, there will still be some level of risk remaining.  Therefore, continuous monitoring should serve to provide agencies with a dashboard of information that lets them know if something has changed to increase their actual risk from what their initial assessment of the risk was. So the question remains: How is this activity impacted by moving parts of your systems to the cloud?

Leap of Faith/Loss of Control

Moving to the cloud involves taking a leap of faith, given that the point of moving to the cloud is to transfer responsibility for the system in question. It can be argued that risk should not be transferred, but that really isn’t consistent with reality. While agencies can’t transfer accountability, they can and should, most certainly, transfer the responsibility.

If an agency puts a platform in the cloud, the security requirements of that platform, which they are accountable for, have not changed. And if they have followed their certification and accreditation procedures properly, they know exactly which controls–in the 16 families of NIST controls–apply to that platform. But there’s the rub. An agency no longer has complete control over that platform once it’s in the cloud. They then have to reassess those controls with respect to the Service Level Agreement (SLA) with the cloud provider. For things that are external to the platform (such as physical and network controls), agencies have to rely on their provider to be compliant.  The SLA had better include these newly reassessed controls as well as a description of how the provider is going to prove compliance through continuous monitoring and reporting of those controls. When agencies move something to the cloud, they are moving a significant portion of control over to the provider and the only protection they have is the SLA.

Going Public

Of course, we have to remember that there is no such thing as “zero risk”.  To this end, when something goes wrong, how do we remediate the issue?  This question brings up a point that agencies are not typically used to dealing with since data has to have value to be compensated for its loss. A provider can patch vulnerability or otherwise remediate the risk, but if there’s an incident, then the presumption is that there’s been a loss.  Therefore, agencies can potentially open their remediation issues to the public scrutiny because enforcing an SLA requires legal action. Ultimately, the bottom line in considering whether or not an agency should move its platform to the cloud is understanding that the associated risk could mean issuing a public explanation and apology.

When having what seems to be a complex network connectivity problem, sometimes the best method to tackle the issue is to implement the KISS Method. Look for the simple stuff first and keep these tips in mind when troubleshooting network connectivity.

Top 10 Tips for Troubleshooting Network Connectivity

 

1. Use a methodical approach.
2. Work from the bottom upward.
3. Divide the problem into pieces and then solve them one at a time.
4. Keep a record of your problem solving tasks.
5. Keep an open mind about what may be causing the problem.
6. Be aware of security barriers.
7. Pay attention to error messages.
8. Try to duplicate the problem and solve it in a known environment.
9. Stick to a few troubleshooting tools that you understand thoroughly.
10. Don’t neglect the obvious.

First, gather detailed information about exactly what is happening (or not happening).  When a user reports a problem, discuss it with them.  Next, get the details. Find out which application failed. It’s also important to get the remote host’s name, IP address and the user’s host name and address.  Then ask the user what error message was displayed?  If possible, verify the problem by having the user run the application while you talk him or her through it or duplicate the problem on your own system.

Other key issues you should consider include whether the problem occurs in other applications on the user’s host or is only one application having trouble?  If only one application is involved, the application may be misconfigured or disabled on the remove host.  Because of security concerns, many systems disable some services. 

After reviewing the problem on the application level, it’s now time to turn your attention to the remote host. Does the problem occur with only one remote host all remote hosts, or only certain “groups” of hosts?  If only one remote host is involved, the problem could easily be with that host.  If all remote hosts are involved, the problem is probably with the user’s system –particularly if no other hosts on your local network are experiencing the same problem.  If only hosts on certain subnets or external networks are involved, the problem may be related to routing.

Once you know the symptoms of the problem, visualize each protocol and device that handles the data.  Visualizing the problem will help you avoid over simplifications and keep you from assuming you know the cause even before you start testing. Problem solving isn’t always easy. When troubleshooting, take it one step at a time, remember these tips and keep it simple!