Safe to Deploy? What’s That Mean?
June 24th, 2010
Over the past several months, I have spent many days in the lab performing Safety to Deploy (STD) tests on our customers’ products. Let me tell you, this has been an enlightening and immensely educational experience. If you’re not familiar, the Safety to Deploy test is the first in our trio of tests designed to improve product development. STD is the initial pass with a high level security approach focused on a single typical-use case of the product. Further tests (Fit for Purpose is second in the series and the Destructive Test is the third) are designed to push the security limits and dive deeper into validating product claims.
Many of our customers are in the security industry so we’re often asked to focus on their security tools or products. The TSC tests are not quality assurance tests that look at what the product engineers designed the product to actually do, although that can be a part. The Safety to Deploy test looks at the product from the perspective of a potential end user or systems administrator and considers whether, under typical use, it introduces any security or business risk into the environment. This is almost an easy answer: Unless the product is replacing an existing tool and making things more uniform or simpler, it naturally adds risk – it’s another vector for vulnerabilities to be introduced; it adds more complexity into the user’s world. We all know that these things introduce risk. But these aren’t the kinds of things our Safety to Deploy test evaluates; instead, we assess whether the product unintentionally adds more risk than the benefits it provides. In other words – is it something TSC would recommend be deployed in a network or on a system? If not, what needs fixing and how should it be fixed?
The TSC team thinks about a wide variety of concerns while going through this exercise. My colleagues and I brainstorm all the ways that things can break and how malicious attacks or unintentional mistakes may cause either system or configuration failures. Will I be able to deploy it? Is it easy? Could I accidentally leave holes in the security when I do so? Can someone get to the configuration and deliberately or accidentally change it? There’s so many of these questions that I could take pages to enumerate them. Many of them are dependent on the kind of product it is and the complexity. Fundamentally, though, remember that this isn’t a hacking exercise. I’m using the team’s collective experience to understand where failures can be introduced and looking to see whether, indeed, they have been introduced.
I’ve tested products where everything breaks. Believe me, this is not a day to be in my office. Even I don’t want to be there. It’s hard to keep going when you get frustrated trying to get the product to run! Fortunately, there are the opposite days, too, where you get a really sweet product that works the way you expect and has a great user interface and clearly had smart people working on the design. Strangely enough, I’ve seen products at either end of this spectrum–from the same development teams. Our goal is to help our customers consistently turn out great products every time! [Or maybe just so I don’t have those nasty days in my office! ;-> ]
The end result of our Safety to Deploy testing is a packaged document that considers the product, the typical environment, and the business and security risks that can be introduced in a standard deployment; we also provide recommendations on how to improve the development processes to make them more reliable. I think the TSC customers definitely get their money’s worth. The recommendations that come from the team are sound, based on strong business acumen and years of security, operations, and development experience. I haven’t had a dissatisfied customer yet…