Matasano Chargen - Latest Comments in What Common Criteria Certification Means

Re: What Common Criteria Certification Means

sargon — Mon, 25 Sep 2006 20:15:14 -0000

Just to keep the old blog current, this just in (from NIAP website):

Due to fiscal constraints, beginning on October 1, 2006, for FY07, the NIAP CCEVS will only accept Medium and High Robustness PP compliant products in support of National Security customers. Product submissions meeting the above criteria will be queued and validation resources will be allocated as they become available. As a condition of acceptance, detailed letters of intent that identify the intended DoD or IC customer (containing POC name, organizations, email, phone number) will be required.

CCEVS will continue to provide updates on the status of the program via this website. Please direct questions or concerns to NIAP at (410) 854-4458.

So not only are CC evaluations crap, you can't even do them anymore :)

Re: What Common Criteria Certification Means

Jeff Jones — Fri, 23 Jun 2006 19:59:21 -0000

Ray - Hadn't found your blog until (former Colleague) Window reach out to me this past week to see how things were going, but I like the topics so far.

I recently also wrote a bit about the Common Criteria process at (http://blogs.technet.com/security/articles/4300... Importance of the “Evaluated Configuration” in Common Criteria Evaluations.

I'm pretty critical of the process too, but there are some benefits it has brought - largely to get governments away from large custom (arbitrarily spec'd) systems and to provide impetus for driving research forward in the industry well before security became "hot".

The other key problem is that there is a lot of variation in the scope of existing process by the different evaluators AND oversite bodies worldwide, which basically enables a "race to the bottom", or lowest common denominator. That is part of why the mutual recognition agreement ends at EAL4 - USGov wouldn't trust anyone who says they evaluated something at EAL7 unless NSA folks reviewed it themselves.

Re: What Common Criteria Certification Means

Jon — Fri, 23 Jun 2006 14:09:33 -0000

While the original post makes valid points, it also shows a lack of understanding of Common Criteria and the complexities of assurance methodologies. I think Mr. Potter addressed most of these so I will not rehash that.

A few points I would like to add.

There is a difference between the common criteria methodology and how that methodology is applied. I personally feel that much of the problems with CC today are due to problems with government oversight. They are the ones that have to power to raise the bar with penetration testing etc. That being said without over site the process will quickly degrade in quality as vendors will generally not choose a lab that will break their product and fail them. Much of the documentation CC asks for is just good software engineering; my experience is that companies that do not have design documentation generally do not have very good products.

As a consumer of a product it is also difficult to have faith in a “certification” performed by a “high end” independent lab. How would one compare these labs? Is matasano better than @stake? Also labs generally have core competencies but will happy to do evaluations for all product types (example I would not trust a hardware assessment by @stake but I may if Ross Andersons group does it). Additionally within each company individuals skills differ. Has a vulnerability in the product been found but due to NDA not been made public or in fact fixed (yes this DOES happen).

I have seen a number of vulnerabilities found in CC evaluations and have seen vendors change to a lower EAL level because they could not easily address them and the lab/scheme lost some faith in the product. That being said these generally tended to be hardware based higher assurance systems with the evaluations performed by specialized labs.

So yes CC needs to be revamped. BUT you can be sure that if labs/schemes start failing evaluations there will be a lot of political ramifications.

Re: What Common Criteria Certification Means

Ray Potter — Thu, 22 Jun 2006 22:17:24 -0000

Thomas-

>So the obvious question is: how many
>vulnerabilities can you point to that
>were discovered by the CC process?

From my experience at Cisco: none.
From my experience as a consultant: none.

Remember, EAL2-EAL4 evaluation isn't meant to be a "find the vulnerability" exercise. It's a level of assurance on the implementation of a product’s security functionality. That's it.

So the next obvious question is: is that valuable?

My take (in brief) is maybe, maybe not. It’s certainly not a panacea. As far as the other evaluations (and the work you do at Matasano), that's all very good and useful and needed stuff. I certainly think it yields more tangible, measurable, and quantifiable benefits/results. Personally, I'd like to see some increased requirements for these methods to complement Common Criteria or FIPS 140. Those types of activities are usually conducted as part of a systems-level certification & accreditation process. Products will undergo pen testing as prescribed by various requirements of policies and systems owners, and this testing is conducted on the product(s) in their deployed configuration.

One thing is certainly true: Whether it's disparate customer requirements, difference in evaluation strategies, or variance of involvement of engineering, everyone has their own experience, insights and emotions regarding Common Criteria and FIPS 140. Mileage certainly varies.

I look forward to continued off-line discussions.

Re: What Common Criteria Certification Means

Thomas Ptacek — Thu, 22 Jun 2006 10:27:47 -0000

Ray, thank you for writing. You are making interesting points. I'd love to be able to offer you some form of equal-time defense of the program beyond blog comments.

That said: I just disagree with you. I don't know you personally but I have every reason to believe that you're a dedicated and competant professional. I think that's probably true of the majority of the people who work at CCTLs. But that doesn't mean CC evaluation is a serious process for vendors.

I'm speaking from knowledge of several vendors, including the successfuly-certified vendor who coached me through the CC process when I was managing the product Arbor got evaluated. In each case these vendors saw the CC process as largely irrelevant to the functionality of their product, meaning engineering resources (beyond tech writing) were not committed to it.

There are other processes --- a Neohapsis evaluation, an @stake assessment, a customer pen-test, even Cisco AVVID certification --- that I have been a party or witness to. And I see a night-and-day difference between those and CC evaluations.

Here's another way of making the same point:

At Matasano we've done many, many customer application and product pen tests. We do ours under insanely tight schedules --- one week is probably a good median allowance. I can think of only one instance in which we did not come back with shattering vulnerabilities (that was a storage HBA, which had firmware built on a synthesized CPU, with minimal "everything", for which we had just a few days testing time).

So the obvious question is: how many vulnerabilities can you point to that were discovered by the CC process?

I'm saying this to offer you a chance to quantifiably stick up for the program, not just to make a rhetorical point.

Re: What Common Criteria Certification Means

AnAAGReader — Thu, 22 Jun 2006 06:13:59 -0000

Re: 'That CC testing addresses modern threats, such as would be addressed in a Bugtraq post. No. CC testing confirms the existence of features such as a password prompt, unless you don’t have one, in which case you can just document that anyone can log in and still get certified.'

I don't think there is intent for this to hold true with CC 3.0. I believe there is intent to ensure that external evaluators will be urged to check for vulnerabilities, i.e., CERTs on the product being evaluated and verify whether or not they impact the TOE.

Also re: 'That CC testing is overseen in good faith by NIST, the NSA, or the DOD. No. CC testing is conducted by “CCTLs”, which are often tiny divisions of the IT practices of companies like Lockheed and SAIC, like every other commercial enterprise that seeks to do business with the government.'

I have to at least partially agree with Ray's dispute on this. I'm confident that either people directly representing NSA/NIAP are involved or that people who have received "validator training" from NSA/NIAP are involved in the evaluation process, particularly at EAL4 and above. At EAL5 and above TOEs are pen-tested by teams affiliated with NSA. I think it would be accurate to say that direct government involvement is limited at low assurance levels; but that involvement increases with respect to the level. The team that did the EAL7 "data link diode" would be in a great position to answer that.

Re: 'That vendors get CC testing so they can get an independent assessment of their security. No. Vendors get CC tested so they can close product sales to the government, particularly the DoD, where lack of CC status can delay or kill a deal.'

I think Johnathan Shapiro of EROS/CoyoteOS put this best when he said (re the Windows evaluation): 'Security experts have been saying for years that the security of the Windows family of products is hopelessly inadequate. Now there is a rigorous government certification confirming this.'
(http://eros.cs.jhu.edu/~shap/NT-EAL4.html) CC is (in general) reflection that the TOE does what you says it does; and a measure of the steps you've taken to provde it.

Re: What Common Criteria Certification Means

Ray Potter — Wed, 21 Jun 2006 22:25:59 -0000

I'd like to restate my previous comment of "Otherwise, I think your other points are spot-on." By this I was referring to your other stated misconceptions.

>What it does correlate with is the
>amount of money you’ll spend with
>Lockheed or SAIC to purchase a better
>grade for your product than your
>competitors have.

While it's true that evaluation costs typically increase as the assurance level increases, but I don't think that's really the point you're trying to make here. Anyone who has been through Common Criteria evaluation knows that a product vendor doesn't just cut a check for a certificate. CC evaluation requires involvement from senior engineering resources (and yes, their tech writers) to support the effort - trust me... I ran the program for one of the most active vendors in the CC community. I certainly wished it was as simple as you make it sound to be.

Don't get me wrong... I see a lot of issues and a lot of room for improvement and expansion to add value. But folks need to know and remember the focus of CC, why it's there, and what it does and doesn't do. And while it may be a checkbox exercise for procurement, it's certainly not a checkbox exercise for product vendors, (most) consultants, or evaluation labs.

Re: What Common Criteria Certification Means

Thomas Ptacek — Tue, 20 Jun 2006 20:08:29 -0000

Ray, I hear what you're saying about how hard these things are to quantify. The problem is, they aren't hard to quantify. We have years of data now about the histories of popular software.

CC certification hasn't correlated with stronger products.

This doesn't surprise people who have been through the process before, because most of them share the experience of having the overwhelming majority of the work done by their tech writers, not their engineers.

Re: What Common Criteria Certification Means

The Other Tyler Durden — Tue, 20 Jun 2006 10:29:37 -0000

I wanted to destroy something beautiful.

Re: What Common Criteria Certification Means

Ray Potter — Mon, 19 Jun 2006 23:52:03 -0000

I think there is another misconception: that a product evaluated against the Common Criteria is a secure product. Are products that are evaluated hacked less often? Does evaluation yield fewer (or even no) vulnerabilities? These metrics are hard to quantify, and this is a rather dicey issue in the Common Criteria community.

That said, the purpose of Common Criteria is to provide assurance that a product's specified security features work as the vendor claims. And there can be some benefits to the development process.

Regarding your misconception #2, there actually is Government review of testing. You're correct in that the CCTLs conduct the testing, but there is a review conducted by a government Validator (or Certifier in other global evaluation schemes). Each Validator checks that the CCTL evaluation of evidence documentation is thorough according to the Common Evaluation Methodology, and at the end of the evaluation, the Validator will write a Validation Report.

Otherwise, I think your other points are spot-on.

Re: What Common Criteria Certification Means

Tyler Durden — Mon, 19 Jun 2006 21:44:52 -0000

The first rule of the CC certification process is: you do not talk about the CC certification process.
The second rule of the CC certification process is: you DO NOT TALK ABOUT THE CC CERTIFICATION PROCESS.
Since this whole deal is so similar to a mob business, you should watch out for the EAL hitmen.
Seriously, thanks for debunking this extortion racket in such brutally honest terms.

Re: What Common Criteria Certification Means

ScummyVendor — Mon, 19 Jun 2006 19:37:26 -0000

The CC process is nothing more than yet another "full employment act" for government contractors. At the levels that DoD requires to sell into govt organizations (EAL2) it pretty much just certifies that your product documentation describes what the product actually does. The ability of the CCTLs to do any credible level of testing on network security technologies like a hardware-based IPS engine is laughable when the vendors themselves have a hard time coming up with a complete test matrix.

CC is nothing more than a government mandated ~$250k barrier to entry for startups and other small vendors. To say it is worthless is overstating its value.

Re: What Common Criteria Certification Means

Thomas Ptacek — Mon, 19 Jun 2006 17:44:34 -0000

Are you trying to say something nice about AIX or mean about AIX? =)

Re: What Common Criteria Certification Means

IBMer — Mon, 19 Jun 2006 17:27:42 -0000

You forgot to mention that AIX has been evaluated at EAL4+ against LSPP and CAPP.