-
Website
http://www.matasano.com/log -
Original page
http://www.matasano.com/log/1093/patch-your-non-djbdns-server-now-dan-was-right-i-was-wrong/ -
Subscribe
All Comments -
Community
-
-
Top Commenters
-
Popular Threads
Is it possible for the average person to comprehend these design flaws without an advanced engineering degree?
Could someone of as humble a vocation as "security researcher" get their mind around this, or would it require, say a trained engineer such as yourself?
Nah, I've got it on good authority that if you aren't lost on the pass dns cache poisoning attacks and maybe familiar with the workings of the protocol, recursive lookups, etc. Then I'm sure Dan's coming talk will be suitably enlightening.
By the by, Tom's initial but retracted post was close to my idea of the problem, and I wasn't all that interested till someone with some insight on the matter, combined with Dan's interview with Rich, disavowed me of my foolhardy assessment of its seriousness.
just to be clear, the discussions may or may not have had were very vague and I'm happy to wait for the offical word like everyone else.
Somebody decided that a random source port per query was the best (and only) way to solve this issue and since I don't know what the issue is I can't assess if that is the right solution for me.
Besides DNSSEC Here is what Vixie proposed (yeah looks like an April 1st RFC)
Here is the XQID proposal (which I think is a much better way to address the issue than random source ports) and the prompt NX response from the "powers that be" on the basis of favoring a "political solution" to the "political problem".
Simple theory behind this: It would be more easy(in time) to spoof dns, if we wouldn't need to race against real dns server!
So what Dan found, is how to make legitimate DNS server ignore victims request, so that attacker could have more time sending spoofed responses.
Did you think why remedy is to randomize source port? Because, if we know victims source port (dst port we already know it will be 53), we can spoof UDP request to legitimate DNS server and make that legitimate DNS server reject victims request!
I hope to meet a few of you guys working for Matasano.
(this is why people laugh at both you & him)
"The easier by far solution is to implement DNSSEC."
Awesome stuff.
I'm sure it's not that simple here, but I'd put a small wager on a bad random number generation technique for those IDs whereby knowing a set of responses from the target DNS server shrinks the keyspace massively...
ok, ok, significantly more interesting than QID space, I'll buy that for now (since now in addition to Dan, you and Dino say so too) but the question that I still don't have an answer for is:
Is randomizing source port the right (and only) way to address this new significantly more interesting thing?
What about implementing the XQID proposal (or a variation of it)? Would that address the problem?
The reason I want to know this is because deploying the current patch may have several side effects with real security implications and I want to understand if I should do it and assume its cost because it the best technical solution or if I am been forced to do it due to somebody's political leverage or preference. I do not think these are unfair questions to ask. We could avoid all this if we had more details but alas we don't...Also, I know of at least one OS software vendor with subject matter expertise that is in the same situation whose only choice is to fall in line and implement the same solution or be perceived as not being diligent in addressing the issue.
oh btw, the XQID thing did address downgrade attacks it simply wasnt immediately evident to the reader.
Why do something simple and non-invasive like XQID when you can do something massively pointless and overengineered like DNSSEC?
Lets talk at BH about XQID.
16 of us got in a room, and there was not a whiff of politics. Lots of -- wow, we screwed up -- but no politics.
Oh yes.
For the record, I believe you have a legitimate exploit. At issue here is that we're creeping back to a mid-90's security mindset that says patch because we say so. The way DNS is deployed for some people is changing in a major way - and they aren't being told why.
I fully expect better ideas after August 6th. If I was going out with a huge bug, and no particular date in mind, or a date way off in the future -- alright. But there's a population that would like advance notice, and there's a population that ain't doing anything without technical details. If nothing else, both populations are getting very close to what they want.
I'm totally open to a debate, after my talk, about whether this was the right path to take. But so far, "every vendor in the world patching on the same day and not whining about it" is about as distant from a Mid-90's security mindset as you can get.
Because he was right.
Corporate security folks out there - how are you approaching actually applying this patch? Whats your timeframe? How reticent are you to apply this without better understanding of why its changing assumptions you may have made about how DNS is deployed in your company? Its entirely possible I'm wrong, and I misinterpret how our clients would react to patching a critical part of their infrastructure based on limited information. And changing their firewall rules too.
(Wouldn't it be nice if there was a way to apply a rule like this in a single place, and have it pushed out to all your firewalls simultaneously? And roll it back if all hell breaks loose? matasanoplaybook)
It's also worth pointing out that I don't really have any specific DNS or anti-Dan agenda here. I just feel very strongly about fostering and encouraging open security discussion, as I think its improved things immensely. Had I not been asked NOT to discuss this stuff, I probably would have continued my blog coma.
So, you think I should have said nothing until Black Hat, and gotten the vendors to wait until then too?
Because, you know, that was an option. Would your clients have preferred that?
I use OpenBSD for DNS servers in my company, and the fact that they have chosen NOT to use the BIND solution, and instead implement their own, tells me that the ISC solution is sub-optimal.
Since the limited information available infers that random source ports mitigate the issue, I've enabled NATing on the external interfaces of my recursive DNS servers, since pf uses random per-connection source ports for NATing. If this avenue wasn't available, I would have migrated the company to djbdns (and still may).
As a corporate network and security guy, rather than a researcher, I frequently feel insulted by vulnerability releases like this. Apparently, researchers and vendors assume that everyone else to too dumb to make informed choices, or apply their own mitigation strategies. I have no problem with big behind-the-scenes coordination for SPs (like the TCP reset bug, where ISPs got the fix first), but after the initial wave, I find little value in waiting to release enough detail for admins to make their own decisions.
I expect the other talks at Black Hat will be spacious as everyone crams into Dan's talk to see what the big fuss is about.
When patch Tuesday comes around, people look at whats been patched. If you hadn't tried to stifle useful discussion, maybe a Tom would have done enough research on his own to gain a better understanding of the issue at hand. In turn, the corporate security + IT types could get a cogent discussion of the issue laid out for them, and decide for themselves how to prioritize patching, or apply a different mitigation strategy - because like Steve above, they're intelligent enough to make their own decisions.
I'm not saying full disclosure is the only way - just that thoughtful technical discourse on security issues is key to making the world a safer place. I think we'll just have to agree to disagree on whether or not that's the right attitude to have, in this case - but I promise you, whatever you release at Black Hat will not make me any less upset that you succeeded in sidetracking the Tom's (and Ivan's, and Steve's) of the world from discussing the technical issue.
Asking people to accept Dan's (and Tom and Dino's) words on blind faith is asking a lot. I think the position a lot of people are taking is "details first, patch later" as opposed to the "patch first, ask questions later" cowboy mentality.
My take on what Dan is going to say: he is going to show that when the attacker and target DNS client are on a GbE network (think of a company network), and when the source port of DNS requests is known and constant, the attacker can bruteforce the 16-bit TXID in 34ms average or 67ms seconds worst case (assuming he sends 128-byte DNS replies -- (128*65536*8)/1e9 -- packets can be made ever shorter by removing unnecessary fields, probably 60-90 bytes).
Which means that DNS spoofing is DAMN EASY and it has always been the case for quite a while (basically since GbE became widespread). Most people just don't know it takes only 67ms. I think this is what shocked Thomas :-)
Then Dan is going to explain that randomizing the source port adds ~32-64k (depending on your UDP settings) values to bruteforce, transforming the attacker's pb to a 31-32 bruteforcing pb (making it impractical again).
Am I right, or am I right ? :-)
Plus Tom probably whiteboarded that Tuesday morning in his jammies.
Also, the patched named (9.5.0-P1) uses the entire pool of non-privileged ports for its randomization purpose without an option for users to configure it, you can verify this by reading get_randomport() in lib/dns/dispatch.c. that is what OpenBSD developers may consider an ugly hack developed in haste. Your NAT approach may not work entirely well because unless pf understands the protocol there is no easy way (except a timeout) to remove entries from the NAT state table. I'm interested to hear more about how that option is working out for you.
I'm confused here. Maybe I'm just too old school for my own good, but, what happened to the fundamental tenet that EVERYONE in the know use to subscribe to: "more information is good thing"? Remember when smart people would disclose what they knew to the rest of the world so that other smart people could discuss and come up with other viewpoints and angles? Kind of like an "open community" (not my term).
I like and respect Dan. Back In The Day when I was cool, we did some dirt together. But I'm just not convinced this is the right way to go about this. Amiwrong or does it kind of feel like of media exploitation? I mean, honestly, if we are going to jump ship to the other camp, wouldn't the Internet and the World be safer if there was no disclosure at all? I don't think so, but I do think the disclosure and discussion needs to come at the time of remediation so that organizations are free to determine what is the best path to a solution and they are free to weigh the second order effects from patching vs. not patching vs. .
I'm probably not the best test case, since my DNS servers only serve my enterprise of ~2000 systems, performing about 5-10 queries per second. At that level, the NAT rule seems to be averaging about 100 - 120 states at any given time, and most are less than a minute old. For my purposes, it seems to be accomplishing the goal of source port randomization, anyway.
But having said that, he's still going to get on stage in a couple weeks with a really, really nice finding. So I'm not going to go toe to toe with him on this, because I'll lose. It's not the best handling of disclosure ever, but it's really good research work.
However a lot of people are running around right now trying to get a complete understanding of this bug and how it impacts them, and in addition to that how the fixes impact their current deployments. And at this point I don't believe they have enough information to fully answer those questions.
Can you elaborate on what you would like to see done differently?
Dan says, "I have a patch for a serious vulnerability. If you believe me, install it. If you don't believe me, which is perfectly understandable, give those who do believe me just 30 days to get a chance to brace themselves and I will explain everything you want to know."
What is wrong with this procedure? Would the right approach be to feed the black hats the knowledge while the whole world is still vulnerable? Would you rather see the affected systems breached than wait 30 days?
It seems to me you are criticizing, but not in fact offering any alternatives.
If the goal is to prevent systems from being compromised, and it really isn't possible to reverse engineer this flaw from the patches, what's accomplished by talking about it at Blackhat? Do you think 30 days is enough time to patch and secure the entire internet?
On top of that, this is not a run-of-the-mill SQL injecting bug in some barely known web application, it is related to a DNS design flaw whose solution has been surrounded by controversy and political bias for over a decade. It was likely to stir more controversy that normal if the details and the mechanics of how it was reported, discussed and disclosed were not entirely clear.
I have no idea, but yes - with a patch backed by leading vendors in the industry, I do imagine 30 days is sufficient to secure the most critical systems on the Internet.
The purpose of talking about it at Black Hat is exactly the reason that you and/or others have given when asking Dan to disclose full details: peer review and disseminating the specifics about this vulnerability to system administrators so even the most skeptical ones have a chance to assess its severity and determine whether they should patch or not.
With regards to not discussing, I assume you refer to the last bit of Dan's initial blog post. First of all, I read that as a very humble (he admitted it was unusual and maybe unreasonable in the post itself) request, rooted in the same (reasonable) safety concerns as described above - not a command or instruction as some people is trying to portray it (Marcin calls it 'cowboy mentality').
Again, I see Dan saying: "If you can trust me, Microsoft, Cisco, ISC and CERT for 30 days, I recommend the suggested response. If you don't, do what you must, but don't say I didn't warn you."
With that said, can you elaborate on what is wrong with this request? What could have been done if Dan had not made this request?
I appreciate your situation, but that argument is flawed. You are putting Dan at responsibility for circumstances that are out of his hands. Dan Kaminsky can not both disclose and not-disclose. The complaint for that situation you must lodge with God or Aristotle or general relativity or whatever. Meanwhile, Dan Kaminsky only has two options, and he chose the one that will leave the most critical system unsusceptible to attack.
Once more, enough with the finger pointing, please. Offer specific alternatives - because then it will be clear that in practice, there are none, and Dan has handled things quite well.
Isn't that a false choice? This isn't about trusting Dan, Microsoft, Cisco, ISC and CERT as to whether or not this is a serious vulnerability or not. I think the decision is "trust me that you would agree with me that the risk is worth the costs of patching immediately... and take my word for it, there are no other possible mitigations that your organization would find acceptable".
Also, the idea that no one else could have found this bug is naive in 2008. Even if they didn't already have it, once people know something exists, they will hunt it down. We have seen this before with WabiSabiLabi.
The knowledge then becomes asymmetric. Attackers are more adept, more motivated and have more time to figure it out than an F500 security admin. There is a really good argument that you can level the playing field by giving people more information and let people make better decisions and come up with potentially better mitigation strategies.
For example, maybe detecting that it is happening is sufficient for some organizations. Maybe they aren't in a position to patch right now.
I believe that once the seriousness of this issue comes to light, it will be clear why Dan and others chose this path. The effort being extended here to protect the "worldwide community" from the brunt of this issue by getting these patches out in a coordinated fashion should be commended.
Yes the patches may change some of the assumptions some have made in their environments when using naive stateless firewalls or ACLs, but that is a small price to pay for the valiant attempt at keeping all of us connected to our email, entertainment, security news and blogs we all crave. Not to mention trying to keep the b' herders at bay.
Yes, I believe this is that fundamental of an issue, and the response has been an appropriate step.
Believe me, I know.
I am not attributing responsibility or pointing fingers at Dan or the other 15 people I am just saying that they've given me just two options and scarcely more information than what I already had to let me find a solution of my own or to assess the cost/benefit ratio of adopting of each of the options I've been given.
I can't offer you specific alternatives because I don't know what really is the problem.
CAN'T YOU UNDERSTAND THAT?
Btw, I mentioned XQID and so far I have not been told that it wouldn't work (nor that it would).
I am sure there are more than 16 DNS security experts in the world (I am not claiming to be one of them) and if more of them analyzed the problem maybe more alternatives could be found.
Besides, I don't like to be told what is best for me I like to come to my own conclusions and I feel that I've been temporarily denied of that possibility in this matter... but that's just me, you are free to do whatever you want with your DNS servers right? I may decide not to patch mine and simply hope that nobody will poison my server's cache to make your domain point to some kiddie porn site, let's just hope that you're not servicing critical security updates, financial or otherwise privacy sensitive services on your domain.
Btw, I am sure that P2P software developers and botnet operators will love your patched BIND.
Yes, "maybe" it is. "Maybe" it isn't. Again, recommend a specific alternative to Dan's approach. Then, and only then, can we compare pros and cons. Anything else is idle speculation.
Your tone doesn't make you look very rationally or pragmatically (or any other arbitrary euphemism for that matter) inclined either.
The point that you have made so far is that you aren't happy about the situation and you have no idea what to do about it. Of what relevance is that to anyone here?
I think what folks here are suggesting is that after the patches came out, release more information about what the issue is. Enough for people to make an educated decision about how to fix it that makes sense for their environment.
Brian:
The co-ordination isn't really what some people are objecting to. It is post co-ordination. The fact is, right now, numerous people know about the details about this. Many don't. People that do can now do whatever it is against the people who don't. Their only option is to patch. Patching may be expensive. There may be another solution that works better for organizations.
I think we should poll people who actually are responsible for the security of DNS servers and see what they would prefer (disclosure that would allow them to pursue other options even if the entire Internet would know more about the issues, or have to decide on patching or not patching based on the information currently available.
30, 60, 90 days... doesn't matter, it will never be enough time to secure 'the most critical systems' if those system administrators don't know anything about the vulnerability they are patching. Proper disclosure has taught us a valuable lesson (at least I thought it did), that more information is better. The bad guys are going to obtain it anyway, mine as well let the good guys prepare.
more details are needed; our telecons with our customers makes us sound incapable of our jobs - we cannot be firm and offer advice that we have confidence on about a resolution to the problem because we dont completly understand the problem; likewise, we cannot have faith in the vendors that say they're not vulnerable because we fear they dont understand the problem either.
releasing information in this manner is harmful to information security from the business aspect, harmful to the PR image of security researchers, and detrimental to overall security as a whole due to the spreading of misinformation and speculation, imo.
The fact is that the issue that was recently patched has been making the rounds for nearly 18 years! It got some good attention with CA-1997-22, and then more later with CA-2001-09 and various adaptations of that, yet it was still never properly fixed. How much time do we need to know of (at least one) key problem and talk about solutions? It's about time that this workaround has been implemented, because frankly, the industry just hasn't come to a consensus fast enough left to it's own devices. (The recent collective effort aside.)
The s in DNS has never stood for Security, but the proliferation of the web has made it a defacto part of the security infrastructure. We all trust in the address bar of our browsers, the security level we apply to certain domains with Zones and tools like NoScript, the vain attempts to graft in spam protection and the sources we download updated bits of code from, etc.
Like it or not, the DNS is part of our security infrastructure. It is about time that we as an industry step up the efforts, skip the workarounds and go about solving the problem at the heart of it all.
This debate I don't see ending soon is great, but is it accomplishing much? Agreed. Some people know the details, some people don't (isn't that part of everyday life?). The hope is that the fewer who know will remain that way until high volume recursing servers are all patched. Those who keep asking for full disclosure will look back on this conversation when the truth comes out and say, "Oh, Ah, yeah, in retrospect, this was one issue where full disclosure was not appropriate."
I hope that most will take these patches as a gift you would be foolish to refuse! Should you choose to delay, just be prepared for the consequences and resign yourself to franticly deploying them around 3am some morning in August.
I guess I'm just wondering how everyone here and elsewhere calling media exploitation can be so certain that immediate public disclosure would be to their benefit. But I guess we'll know by the 6th.
Ivan:
Who cares if you are right or wrong? The question was how to improve the disclosure, not what the best fix for the vulnerability is.
Surely *internal* unicast reverse path forwarding, internally blackholed non-organisational subnets, no internal default route and some *internal* dns features on *internal* FWs may help to mitigate. Even RFC 2827 "Network Ingress Filtering" applied closer to your internal COREs, or some form of host/network IPS?
The point is, in my reading, the attack must exploit a trusted internal host.. almost trivially, think browser, to generate valid traffic requests and capitalise somewhat on subsequent transitive trust.
Sometimes a network level solution can temporarily mitigate and create time/support to address a more costly patching excercise, or even allow for surveillance and incident response to occur at least...
I am interested in figuring out what is the most effective and less expensive solution to this problem for vulnerable organizations so I can provide guidance to them when asked. I am not interested in using Dan Kaminsky's finding and the process he and the other 15 people to disclose it as a proxy for a debate on how to improve the vulnerability disclosure process.
If you have a suggestion that will make the Internet a more secure place before the 6th, please, put it forward. If not - please quit addressing me.
Findings-v2:
0fca0b42a9795b35f12c1d7fdb7abcd6a85e1f7810ede46fb59b8d318487da36
(SHA256 - Just for posterity ?)
Still reading RFCs to see if I can guess who your boss is. In the meantime, why not having him call up Vix for the 411 on the DL?
Lots and lots of patchii come out with vague info. This one has more press and affects something critical, but do you really delay installing MS or Cisco patches unless you have working exploits, srsly?
I realize that these vulnerabilities are not direct code execution issues, but they chain so darn well.
With the understanding that full details will be released in little more than 3 weeks, one would think more emphasis on the criticality, the better.
This vulnerability is still less important than the RSA signature forgery issue, which wasn't critical either.
So if they say ICMP, I say pure UDP and fragmentation.
Dan rightly indicates that it takes a lot of meetings, time, and money to get these things done in an enterprise space (I work in a shop with 90,000+ endpoints). Again, lots of stuff may break.
The advantage of enterprise, though? I have a huge test lab.
What I'm doing? Patching EVERYTHING in the lab now, so if stuff is going to break it'll break now. It also gives me some burn-in/testing time of my own. As soon as the BH presentation hits the fan (so to speak) and the press starts to turn into pressure to act, I can pull the trigger WITH full information.
ISC says that the "beta releases include optimized code that will reduce the impact in performace".
What I would like to know is when the when the beta code is expected to become GA? Anyone clear on ISC's release practices?
Dan / Paul Vixie- any insight here?
We're actively trying to get you what you need. Seriously though, you really want to pull the trigger before my talk. Really you want to pull it as soon as possible. Yes, I've bought you some testing time...but don't treat it like a snooze button :)
How's the perf impact hitting you right now? Most name servers are not run to the edge of capability. I'll look into the GA date.
I assure you, I'm not treating the time like a snooze button. It's really hard to justify some of the risk ($$ loss if something breaks in production) based on the facts in evidence today. I can try to get everything patched, but it's ultimately the business who makes the decision based on the risk assessment I can show them.
Right now, I can show them vendors giving an Important (not critical) risk rating, and no details (much conjecture) on how we MIGHT be vulnerable and what our ASSUMPTIONS are regarding the risk to our business.
Like it or not, from a non-technical business standpoint, that's hardly compelling.
I WANT to patch before your presentation, but I NEED to be ready for when (not necessarily in order of likelihood):
a) the business gets scares by all the press
or b) I can show the security risks are high enough to assume some business risk
This is helped by having everything up and running in the lab as long as possible before either of these things occur (due diligence). It's also helped along by making sure the various security teams (CSIRT, VulnMgmt team, etc.) are on side, and everything is in place so major apps can ensure zero impact.
I get a "POOR" security if I do the porttest:
dig porttest.dns-oarc.net in txt
localhost:~ Michael$ dig porttest.dns-oarc.net
; <> DiG 9.4.1-P1 <> porttest.dns-oarc.net
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30368
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;porttest.dns-oarc.net. IN A
;; Query time: 2323 msec
;; SERVER: 68.105.28.12#53(68.105.28.12)
;; WHEN: Wed Jul 16 13:10:51 2008
;; MSG SIZE rcvd: 39
localhost:~ Michael$ nslookup porttest.dns-oarc.net.
;; Got SERVFAIL reply from 127.0.0.1, trying next server
;; Got SERVFAIL reply from 4.2.2.2, trying next server
;; Got SERVFAIL reply from 127.0.0.1, trying next server
;; Got SERVFAIL reply from 68.105.28.12, trying next server
Server: 4.2.2.2
Address: 4.2.2.2#53
** server can't find porttest.dns-oarc.net: SERVFAIL
try that?
It's a shame DJB's (admittedly abrasive) demeanor has so often distracted so many from the quality of his work. I've deployed djbdns exclusively since forever, and I have never been disappointed with its performance and simplicity, particularly over conflated bloated pigware like BIND (imho). The simplicity of the zone file format alone is delightful. But I digress.
Imagine my delight to learn of this massive urgency and then conclude that I need to do NOTHING about it for the security of my networks (all systems here are showing GOOD on the TXT test).
To discover that DJB was right on this issue doesn't surprise me in the least. At the same time, to discover some personal distaste for him remaining at large doesn't surprise me much either.
Good on him, I say. And shame on so many others for not listening years ago.
the link that i got in my reader is/was: http://www.matasano.com/log/1103/reliable-dns-f...
does it means that what was written in the post is *not* the actual "Real Thing" and was "recalled" by the author? or does it means that it *is* the "Real Thing" and was recalled by the author whatever reason it may be?
just curious...
No one's confirming, but it sure does smell like the real deal.
Why don't you just wait for Dan's talk instead of blowing your trumpets and claiming his thunder. Dan deserves this more than anyone I know given his research on DNS over the years.
Anyway, that's my take on it. Put it to rest and wait for Dan's talk.