Matasano Chargen - Latest Comments in More on Pen Testing

Re: More on Pen Testing

Doug — Wed, 14 Mar 2007 10:34:06 -0000

Seems to me that the purpose of any comprehensive security test is to validate that the organization achieved what it set out to achieve. Ideally, the organization set out to limit its risks to some residual level. That limitation comes about through the three components of governance and technology (1) design, (2) implementation, and (3) operation.

Isn't it best to design appropriately then gain assurance that the design is appropriate, and implemented and operated according to the design?

If so, how well do modern pen tests identify the root causes of flaws, and isolate and categorize issues between enterprise design, implementation, and operation? And how well do modern pen tests provide continual assurance of appropriate operation?

Re: More on Pen Testing

PaulM — Wed, 14 Mar 2007 09:33:54 -0000

So far everybody has failed to mention the best real-world business driver for a third-party pen-test: making an effort to demonstrate to business partners, customers, and auditors that your security doesn't suck.

Seriously, here's some spending on security where you can put the deliverable RIGHT IN THE SALES CYCLE.

Of course, this means that 1) your security rocks and you come up clean on the first run, 2) your third-party auditor sucks (but it's almost impossible for your customers to tell #1 from #2, so tell them it's #1), or 3) you have to go through several test-fix cycles to finally come up clean, which costs more money.

I know I sound jaded on the topic, and I probably am a little bit, but it's true - you can use a pen-test deliverable just like a SAS or ISO audit report to help inspire confidence in your organization. ...and maybe improve security along the way.

Re: More on Pen Testing

LonerVamp — Wed, 14 Mar 2007 01:02:54 -0000

Of course, any discussion on pen-testing should start with defining what you mean by pen-testing, especially in relation to vuln testing. I think Bruce is talking more about vuln testing, which means his conclusion is fairly accurate.

I think Marcus has a great point, though he maybe didn't make it obvious enough. Penetration testing should not just come in and prove vulnerability. Like Bruce said, he can "prove" that in just a few words. Pen-testers should give something back. No one likes someone who just points out the bad things then sticks their nose up in the air and walks away. Rather, point them out and give plenty of constructive criticism and feedback.

Maybe because that can be an added service that costs extra beyond what some companies want to pay (consulting?) they don't get that level, but *I* certainly would want that level. Even if it just means having the IT/security team sit down informally over dinner with the testers and pick their brains and share stories.

In addition, there are way too many "bad" assessments being done, and this is not always the assessor's fault. My last company had 4 assessments of varying degress and while us techs in the trenches were praying for plenty of holes to dog mgmt to give us resources or just plain managerial backing, the assessments were just done because clients required them. And they were met with dancing, smoke, mirrors, and utilizing ignorant middle managers as the liaisons to the assessors. Lip service and a cheap company... Just like Marcus said, companies with poor security tend to know it, some just deny that they know it while others don't deny it but wiggle and dance so they don't have to do anything about it.

Re: More on Pen Testing

Chris_B — Tue, 13 Mar 2007 20:42:43 -0000

Wasnt it Deming who talked about the "PDCA model" of management? PDCA or Six Sigma type ideas should mandate good design and verification of implementation.

There sure do seem to be lots of Chris's round lately!

Re: More on Pen Testing

Nate — Tue, 13 Mar 2007 17:49:06 -0000

ChrisW, agree with your conclusion.

Also, I think we should stop calling vulnerability reporters "security researchers" and instead call it what it is, "security QA". Developing a new class of attack (i.e. format strings) is security research. Finding the Nth instance of it in a product? Security QA.

Security QA is very important, and it should get its due. But when you look at it as just another type of QA, you can see why it is plagued with the problems it has. Vendors file reports in their normal QA database, fixes later show up in software updates, issues get dropped if you're not some major customer, just like every other type of QA.

Re: More on Pen Testing

Chris Rohlf — Tue, 13 Mar 2007 16:17:46 -0000

Bruce makes the best point. A lot of the time IT staff knows their networks are vulnerable. But they need an outside, 3rd party/independent piece of paper to prove it to management in order to get the budget.

Internal audits don't always work and sometimes its out of your control (i.e. you legally MUST have a third party come in and look depending on the nature of your organization).

And as a side note, I think the more and more 'security testing' and 'QandA' converge the more security pro's are going to distance themselves from the term or try to reinvent their purpose.

Re: More on Pen Testing

Chris W. — Tue, 13 Mar 2007 16:06:30 -0000

Penetration testing can be especially effective when you DON'T have a good security design. I know it won't come as a shock to people that not everyone has a good security design or that many people don't want to pay for a good security design.

From where I sit many applications are composite. They have legacy code, outsourced code, shared internal code, binary only components, and open source code. This is done for a practical reason. It is faster to reuse than to create from scratch. Where is the design spec for the binary only component or the open source component or the legacy code, etc...?

Security testing has the advantage of being able to find problems you didn't know was in the code because you either didn't have access to the source or the design documentation or the designers.

Without security testing how are you going to find problems like a leftover debug interface in the code or a insecure side effect of a binary component such as a predictable temp file name? What about a DWR interface that is missing its authorization or session check? A secure design can mitigate these vulnerabilities but not eliminate them.

It's not use a secure design OR do security testing. It's do both! If your development process makes assessing the design difficult or impossible then security testing is all the more important.

No wonder quality assurance professionals are the poor step children of the software development community compared to developers and architects. I see the same biases being played out here in the security space as security professionals who review software design pooh pooh security testers.

-Chris

Re: More on Pen Testing

yoshi — Tue, 13 Mar 2007 15:50:36 -0000

sometimes (ok all of the time) I think marcus lives in some fantasy world that I just don't get.

in the world i inhabit at least, applications are frequently installed or designed without someone knowledgeable about the risks they are introducing -or- you don't have full control over an implementation (for whatever reason). security assessments and pen tests are useful tools to understand and to document that risk.