-
Website
http://www.matasano.com/log -
Original page
http://www.matasano.com/log/1105/regarding-the-post-on-chargen-earlier-today/ -
Subscribe
All Comments -
Community
-
-
Top Commenters
-
Popular Threads
All Comments
-Nate
The underlying premise of this entire article is "we keep it closed, we avoid security attacks". Which is flawed on two counts: the bad guys very likely have been using this (especially after the contentless advisory issued a few days ago) and it creates (confirmed) moral hazards that harm us all.
We deserve the right to make informed decisions. The entire process behind this "responsible" disclosure of the vulnerability (and most vulns today) actually impinges on that goal.
Look, we're not asking for PoCs, alright? But, at the very least, can we stop pretending like you (not you Thomas, but the security research community) are not actually *conspiring to bury* stuff when writing contentless advisories?
As you would say, The cat is out of the bag.
The bottom line in this case is pretty simple - If person Dan tells Thomas some information only because Thomas agrees to unconditional secrecy, then it's really not up to Thomas. He's got to keep his word, and that's pretty much it.
If Thomas had found it out on his own after hearing that there is some unknown issue, he's free to do as he pleases. But, when you get information with a condition attached that you agree to, you pretty much have to honor that.
Not knocking Thomas here, it was obviously a regrettable mistake. Just saying, please no more rants about the (non) disclosure process, as it isn't relevant in this case. :)
OTOH, did anyone _seriously_ expect that details wouldn't be discovered or leaked before Black Hat? :D
Halvar, you rock. Matasano, thanks for delivering.
The question now is, will Matasano keep the full description off of this blog until someone posts working code?
Every day after CoordinatedPatchDay was a gift. Yeah, you might have messed up, but if Dan didn't realize he was already on borrowed time when the patches arrived, he only has himself to blame for his disappointment.
We're in a bit of a tight spot here. We can't moderate comments piecemeal, so anyone who's posted something here before can post now. We don't want to turn off comments on this post, because it's really helping us to hear how people are handling this.
But we don't want to make things worse than how they already are. There are other blogs talking about technical details now. Can I ask a favor of all of you not to make our comment threads, which are almost always better than our posts, the epicenter for distributing DNS info today?
Seriously? A favor, to not post technical information, on this public forum?
Ahem. To what purpose? I picked up the story from other sites that mirrored it. And looks like it is, with all the details, a rehash of what Kaminsky *was* to present at BH Vegas 2008.
Was it an exercise on "hey, let's explain what the problem is in terms that Joe Sixpack can understand, in case Dan made it more difficult than needs to be" ?
"...or for other confirmed disclosure" - in case Dan was piss drunk and hence unable to post the details to his own site?
Or am I missing the point? 'cause it certainly looks like you folks were about to ride on top of his research . . .
I bet Kaminsky is pissed today - don't hold your breath waiting for him to share any more details about anything any time soon . . .
So, what was the point of having it "ready to go" ? Has the NY Times outsourced the "news from the Internet" section to matasano ?
At the end of the day, DNS is a gaping hole in the security architecture of the internet. Even with all the patches applied the basic design decisions and necessities require a ridiculous amount of trust in completely untrusted entities.
By posting the vulnerability and making the details available, you have enabled me to look for the problem. Sure, not everyone will look, but not everyone is running djbdns!
Thanks for the post. I sincerely hope you consider releasing it soon.
Thomas and the other Matasano folks have a history of writing educational posts about impressive hacks. The post under discussion was very much in the mold of previous Chargen posts.
so is halvar..
but hes obviously a prick as well..
what ever happened to respect?
cool, you speculate, you published details, you want a cookie?
unreal
As someone who knows Halvar personally, if not extremely well, he is most definitely not a prick. You don't have to agree with how he handled this, but he's no prick. He's doing what someone in the security research community is supposed to do, figure out bugs. I heard Dan's call for hush hush, as did others, but not everyone believes it should've been handled the way that Dan chose. Can't fault them for that.
-Nate
Dan called for a lid on speculation, but simultaneously issued a challenge, offering to bring anyone who figured out the vulnerability on stage at Blackhat. That's the thing I don't get in all this.. I can understand both the desire to keep speculation under wraps, and give folks time to patch, as well as the desire to figure out the mystery - that's the nature of people in this field after all. But to simultaneously present both sides seems... weird. You can't have your cake and eat it too after all.
Actually, the idea there was to see if I could get some attention on some new blood. I don't know if you've noticed (FX did a while back, to fairly deafening silence) but we're not exactly good at bringing in new kids and giving them a stage to be heard. Mudge really inspired me early on, with nothing but a positive reaction to a clarification I made from the audience. I thought it'd be interesting to open things up, maybe have a couple of college kids or random sysadmins explore something new and get rewarded for it -- while, you know, keeping the infrastructure safe.
And, you know, for the half dozen or so people who figured this out before Halvar, that would have been really nice.
Nate, I do believe our job is to help get bugs fixed. There's a subtle but important difference.
Anyway if what I've read is actually Dan's attack I'm disappointed (eh right, nobody cares anyway). I don't see why it justifies randomizing source ports any more than the previously known problems. Amit Klein talked about CNAME and NS referral chaining already and forcing a series of random queries to bring back to life a birthday attack. I suspect there is more than just that in Dan's finding. I also suspect that Vixie's pointer to a a DNS server that answers a query with a chained CNAME/NS response may also be telling but then again this is all pure speculation and I'm told I should not be doing that because it may break the interneks
"Appeared"?? "Posted in error"? What the hell!?
Can somebody translate that to me?
so the security theater and media circus ochestrated by Dan Kaminsky and friends is finally over.
It was so absurd that people actually played the Kaminsky game such a long time for no obvious reason. Keeping information like a DNS spoofing vulnerability closed from the general public and security researchers while many (bad) guys already know the details is NOT PROTECTING the internet. It is actually the opposite.
Do you really believe the Kaminsky circle of 18 (or was it 17) were the only ones knowing the details. How many members of that circle did really keep their mouth shut? How many of them are backdoored/owned (whatever)?
I congratulate Halvar for stepping up and beeing a real researcher and not a Kaminsky puppet.
Stefan Esser
whether you think dan was right or wrong matasano fucked up by posting what should've been private. anyone else is allowed to post it; but after matasano was told they should've definitely kept it private out of obligation to their agreement with dan. bullshit that it was broken here. fine if it happened elsewhere; but it didn't.
You are very professional and responsible. But I think this apology is overdoing it. A Linux advisory disclosed the cause on its description 11 days ago. Halvar Flake only put it more evidently.
Take it easy.
Alecco
The details were not Matasano's to publish, even if someone else decided to take a stab at it. Shouldn't Dan be the one to confirm (or not) the research? After publicly lashing Dan for playing the media (initially), it appears that Matasano was eager to do the exact same thing by having this post ready.
Cat. Kettle. Oops.
"Do you really believe the Kaminsky circle of 18 (or was it 17) were the only ones knowing the details. How many members of that circle did really keep their mouth shut? How many of them are backdoored/owned (whatever)?"
at least half of them are owned.
Cat was out of the bag even earlier.
07/08/2008 02:46:15 PM VU#800113 Multiple DNS implementations vulnerable to cache poisoning
http://www.kb.cert.org/vuls/id/800113
Credit
Thanks to Dan Kaminsky of IOActive for identifying the effectiveness and practicality of DNS cache poisoning, and to Paul Vixie of Internet Systems Consortium (ISC) for raising the urgency of these issues. Daniel J. Bernstein is credited with the original idea and implementation of randomized source ports in the DNS resolver.
This document was written by Chad R Dougherty.
:)
(Ok, it's not terribly funny *now*...)
Smooth move! You've doomed us all! =(
So, tell me Tommy, if you can't keep info private as a favor to a friend of yours that you respect (and who was nice enough to give you vuln info after you made a condescending post to a public blog about him and the vuln he found), how can we expect you to honor NDAs you may have with clients? Why should you expect your co-workers and interns to find 0day for you and then "never talk about them" (your exact words)? Seems like a bit of a double standard to me Tommy.
All whitehats are the same no matter how well known they are. They're all a bunch of attention seeking media whores. At least Dan was nice enough to work with vendors to get the vuln patched. I have a feeling that the Matasano crew would have put a PoC on milw0rm and a self glorifying blog post on chargen.
Way to go Tommy. Way to go.
I will kindly disagree today and at Black Hat.
Dan did not ONCE ask permission to use every one's DNS as his personal testing grounds and has merely spotlighted an older well documented problem.
Thank fully he did and I am appreciative of most of his efforts yet this is one large item most everyone has decidedly ignored.
My two cents, FWIW
Now that you've explained the reasoning, it makes a bit more sense. Still, I'm sure you can understand how those two points seemed a bit at odds.
As one of the 'new kids' albeit an early-30s new kid with a bunch of operational experience just recently moving into a security role, I definitely appreciate any efforts to help bring new blood into the fold. Personally, I've found many of the more experienced people in the community to be welcoming. I'm sure you don't remember, but I ran into you at the Defcon bar last year and we (and some other people) stayed up drinking until late in the night, while you re-played your Blackhat presentation in more detail, and answered many folks questions. That sort of approachability (and that which Thom shows at every Chi-Sec, even though I've only rarely attended) really help to expand the community and welcome new ideas. Ultimately that's a good thing for those we aim to protect. I'm way off topic here, but there you have it.
Anyhow, good work on this vulnerability. I know you've been taking a lot of flak for the disclosure process you've chosen, but it really seems the industry as a whole doesn't have any way to please everyone. Regardless of what anyone thinks of the actual process, I'm sure most will agree that your intention in this was (and is) noble.
Isn't ecopeland your wife's name?
(1) We staged the post on the blog; when we proofread it, we were playing russian roulette with the Wordpress UI to keep it "Unpublished".
(2) I decided that once the information was "in play" (confirmed by Kaminsky), it was open season. We have a huge audience, and we should have let it hit Kaminsky before we chimed in.
Either way, as soon as the story that Kaminsky was working on something big started making the rounds... then combined with an embargo on publication... really, what did folks think would happen?
This has been a fantastic story so far, and it's also been fascinating to see who's taken what kind of a stance.
Interesting times!
Please don't attack Jeremy's judgement. I never briefed Jeremy.
It feels similar to the stuff with Johanna a bit back.
What would I have done differently here? Almost everything.
Okay, fair enough. As an outsider who didn't pay a lot of attention, it seemed like an attack to garner press, realizing that I'm an outsider looking in on the situation, and didn't really pay much attention to it, I can see how I may be mistaken in my impressions. It's at least worth giving you et al the benefit of the doubt
You don't have to take our word for it or give us the benefit of the doubt, though: the slides from our talk are online, and they're pretty detailed:
http://www.matasano.com/log/925/slides-from-vt-...
Way to go Matasano.
>I would appreciate some info for the stoopid users.
Very simply: DNS works by asking the next guy the number associated with a name, following a chain of links until someone know the answer, part of the answer or knows that the question is wrong. (E.g. a typo.)
So a typical setup may be:
Your PC/Mac asks your firewall/router, the firewall will pass questions to the ISP, and the ISP will ask a root server.
Scenario #1
An attacker will typically try to give wrong answers to the ISP, as that will get the users. (E.g. they want credit card numbers and want to steer people to a fake auction or web shop site.) If the ISP is not vulnerable then a determined attacker may still select individual ISP customers to try her attack.
For protection you need your ISP to update their system, and you should probably check for firmware updates from your firewall/router vendor.
Scenario #2
The attacker has access to your local network. (Maybe you are on a college network or have WiFi running.) You are the target in this case. (You may also be similarly vulnerable to your ISP's other customers, depending on their network configuration.)
The Windows/Linux update protects your PC from this scenario.
To be reasonably safe you need to do your local updates and your ISP also needs to do their bit.
Business computer systems may be setup a little differently, and their principle concerns may not be credit card theives but interception of communications with trusted partners.
ttfn
I tested sending additional RRs and glue to queries back in 2000, and was able to see that many caching resolvers would cache the additional data. It occurred to me then that spoofing replies with the requested data as well as unsolicited data could be a problem. But as a net admin, and not a DNS dev, I noted it and moved on.
This is pretty much exactly what Halvar is describing, is it not?
If an extra 16 bits of entropy is going to fix this, great, but I doubt it is.
http://www.matasano.com/log/53/thanks-mjr/
I think at the end of the day, Matasano was doing what Matasano does best: provoking the "community" into actually thinking about problems out load and openly.
Personally I think it's a great thing & always have.
Maybe you jumped the gun, maybe you didn't.
I can't really see the fault with yesterday's blogpost.
If Dan didn't want anyone to know the details of the exploit prior to Blackhat, why would Dan say anything to Thomas?
This wreaks of smokescreen to me. Kaminsky + Ptacek = incahoots?
If there is indeed no hypebuilding conspiracy, then the least ya could do, is give Dan some love by sticking a link to his site under "People We Read".
but what do I know. I'm still hacking on my Adam.
"well congrats, you guys managed to squeeze your name in on dan’s find and garner some of the press for yourself. Is this a matasano marketing technique? Attack whoever is in the news now as a means of getting in the news also?"
this "accident" is worse than a marketing scheme gone bad, it's the type of thing that i refer to when a highly public data loss occurs as an OEE (pronounced "Oy"- organizatinal ending event). whether matasano can endure the data loss is anyones guess, but i find it beyond inexcusable for an infosec research company to faciliate the very breaches that Dan worked very hard at attempting to protect..
I think Dan has a right to feel like he took flak from me even after telling me what the vuln was. At this point, I've fumbled any moral authority I have to persist in those arguments. But I didn't make them to hurt Dan's feelings. He retains what is likely to be the best talk at Black Hat, though it's his business to tell you why.
The goal, once I saw Halvar's post, was to wait for Dan's imminent confirmation (we expected a blog post from him) and post then.
I was surprised that Dan continued to keep it quiet after Halvar posted, and even more surprised to see our draft had been published. It was a worst-case scenario for us.
There are a lot of things I could have done differently to keep us out of this story, almost all of which I wish I did.
ah! no! it's just that "a post appeared on our blog"
good work... (sarcasm)
Hmmm... I WONDER why that is... maybe it's because the "new blood" is busy keeping their discoveries under wraps like someone else I am just becoming familiar with?
Besides, it's not like this info isn't public knowledge now, so calling for "no technical info on this post" is preposterous, when Slashdot (100x the readership of this blog) already has the scoop.
2. I knew it was WordPress. Matter of fact, I dunno if Error 99 triggered the early release of the post, but WP has had, in the past, information disclosure vulns.
3. Jesse: "The bottom line in this case is pretty simple - If person Dan tells Thomas some information only because Thomas agrees to unconditional secrecy, then it’s really not up to Thomas.".
Please reread the first paragraph of my post.
4. Finally: I'm willing to bet a few thousands of us had the text of the post saved, but I only see a few posts around the net with it. So I guess this campaign for obscurity (let's call it for what it is) has simultaneously succeeded and failed.
If anything, Thomas shouldn't have promised secrecy to Dan in order to get the goods. It was probably a matter of reading the commits in BIND to figure it out. That way, Thomas wouldn't have been obligated to secrecy in a matter that was bound, sooner or later, to hit the public.
In all scientific fields (including IT), it is the first to publish who gets the credit. There have been many cases where others have made discoveries first but missed out on the credit due to their greed (RSA algorithm for example). We need to send a clear message to those who do not believe knowledge is for all.
You all have too big ego's and are just upset that someone else found a serious vulnerability in the Internet's infrastructure, and mostly because that person was not filling you in on the details for a mere 30 days.
The argument that "we don't have the exact details of the bug so I can't assess whether I REALLY need to apply this patch or not" is a complete bullshit excuse and everyone knows it. That's just the best lie that people can come up with to try to pressure Dan in to giving the details out earlier so that they don't feel dumb or excluded.
If every major IT vendor in the world, a creditable security researcher, and some of the most experienced and knowledgeable people about DNS are saying that this is a major issue than that should be enough reason to patch regardless of having exploit code handed to you and the rest of the world.
Sure, there is a good chance that the infamous "bad guys" were able to figure it out before the 30 days, but if it takes all of the world's best security researchers working together for 13 days to come up with an almost-right-answer, than it probably would take "the bad guys" some time as well. Security through obscurity is not good but it is certainly a layer of defense. I don't know how some people here(Esser?) can claim that giving out point-and-click exploit code provides a better defense..
and since when did Halvar rediscover this? Was his guess 100% accurate...???
@Tom
Either the people at matasano are completely ignorant with computers or you posted the details on purpose... If you were so careful about not hitting the "russian roulette" you would have noticed that you posted it the second after it happened and removed it within 30 seconds. 30 minutes of "not noticing" is ridiculous... just enough time to have everyone's RSS readers download it and enough time to act like it was an accident.
@Dan
I think the way you handled everything was great and I am sure there are many many sane people out there that are very grateful for your efforts. (Although you managed to piss off all of the security kiddies in the world.)
The only thing you fucked up was that you gave into peer pressure and let out the details. If only you could have hold your ground, you could have had more than 13 days...
Why didn't these monkeys get nominated for pwnie awards???
Even more, I don't recall seeing an advisory from Dan, just a bunch of advisories from the vendors crediting him with a bug find. He never went ape-shit posting places, so I don't see how everyones (over)reaction can be put on him. Sure, I think its silly (and futile) to ask people to not investigate it themselves, but its not like anyone asking for such things has ever gained any traction in the past (and typically inspires the polar opposite), so why all the fuss?
As for this (the leak) being an organization ending event, I think thats pretty much not going to happen, and its pretty absurd to consider that anything anyone could say would have that big of an impact.
First of all, once the information (and by extension, class attack) is out, it is out. You may contain it for a while but in the end, it will spread to everyone. I, for one, got the full Monty from a link contained in a comment in this blog ...
What makes you think that only Halvar was able to deduce this information? Is this not a sign of arrogance? Do you really guys think it is still 1995 (as conveniently written in the now retired blog post?) and that the world of network security research is still a closed self-centered social club? Let me let you in into some news for you:
MOST OF YOU (us, if you prefer) do not have a clue what is the current level of security research in countries like China, India, Russia or in non-state actors, yet you create a storm (someone is calling this a OEE, for crying out loud) in a teacup for what? That "evil" Matasano spoiled (maybe by a bona fide mistake, maybe by something more nefarious, I cannot speculate) the exclusivity for the conference brigade?
Give everyone a break please and let's get the patches out there :)
I don't think that's the same cat.
@scott morrison
Try to find a copy of the original Matasano post. Digging through the comments in the slashdot thread might get you a working link, or at least a re-phrasing by someone else. Havlar says on his own blog that "[he] was close... but no cigar."
Suppose a fellow is telling the world "please, update your critical infrastructure, fast." Then I'm there talking to the media saying "no, no, don't bother, it's probably nothing major." So that fellow confidentially tells me the details, because I'm seriously undermining his attempts to protect the public. And so I get more media attention when I say "oh, he means it folks."
And then when I accidentally spill the beans I get yet more publicity.
Seriously, what lesson does that teach? "Being an ass is rewarded" sums it up nicely for me. I could've just kept my big fat mouth shut at the start, but then I wouldn't have gotten media attention and lots of blog comments.
I'm not sure what I want. Unlike some other commenters above, I really don't think this is an OEE, and I'm not sure it should be.
Thomas will give many more mea culpas, but I see absolutely no reason for someone else not to follow this exact same path. Sure, he *says* he's sorry, and I believe he is, but he and M'tso will only benefit from all this. In a few months no customers are going to remember the details, just that they heard about the company during that DNS kerfuffle a while back.
Again, I'm not sure what I want to see happen. Maybe it's just the way our industry is, which is a pretty sad commentary. It gets harder and harder to keep one's moral compass.
Duly noted. I read "Mallory has combined attack #1 with attack #2, defeating fix #1 and fix #2," from said post and this, "[r]ecent additional research into these issues and methods of combining them to conduct improved cache poisoning attacks have yielded extremely effective exploitation techniques" and just drew my thoughts back to combining attacks and had a brain bubble.
Obviously this was an epic fuck-up, but I don't think anyone really believes that you guys would suddenly trade away a combined 4 or 5 man-decades of credibility for a 5-minute press bump.
Well, Dan probably believes it, but he'll eventually get over it. Besides, he's probably got a few more design bugs in 30-year old protocols left in his career. :>
I've been a reader of your blog for quite some time; I understand the DNS bug post was probably a bit embarrassing, but it seems like it time to get back on the horse. The community values your insight into security issues -- not to mention that the lack of updates has left a hole in my morning infosec blog reading. We all make mistakes, but that’s no reason to stop publishing all together.
Good luck.
Thanks,
Eric
stop sittting there feeling embarassed, we have all done this from time to time, the rest of us are just waiting for updates from you kids. Lets get chargen moving and dynamic again, one of the better security blogs as far as I am concerned.
gwen hastings
I came across your blog through some other blogs I was reading while I was doing some research for one my clients, "Solera Networks" in the network security industry. I got distracted and was intrigued by your blog post about your concern in network security. Hopefully you are familiar with the new Tivo. If you are I felt inclined to talk about you because I think Solera Networks has some products out there that are somewhat unique and new to the industry (data capture appliance devices) and would give you some great information to write about on your blog – I know how hard it is to find topics to write about sometimes. If you were interested I could even have them send you a demo version of the software if you want to check it out in more detail. Or if it would make it easy to write I could setup a time for you to ask questions from an engineer at Solera networks and transcribe the interview for you so you can post it on your blog. At the bottom of this email I will copy paste a general overview of what their products do.
If you are interested don’t hesitate to contact me, and keep up the great blog; yours was for sure one of the top in the industry that I came across.
-Thank you
Joshua Lewis
jlewis@twelvehorses.com
Solera Networks DS Appliances provide protection against the unknowns. They give your organization Total Network Recall—enabling IT and security professionals to get to the root cause of a network security or performance problem, minimize the effects on your business, and ensure quality of service. By recording all data that passes over the network, Solera DS Appliances give your network a memory so you can see everything on the network and can replay any traffic when needed.
•Capture speeds up to 10 Gbps (Miercom Performance Verified™ report – March 2008 – www.soleranetworks.com/miercom/)
•Storage scalability to expand window for longer recall time
•Up to 8 gigabit ports (10/100/1000)
•Two 10Gb fiber capture ports
•Appliance platform with certified hardware configuration
•Full traffic regeneration capabilities and PCAP creation
•Open API’s for integration with third-party tools and automation of data collection
All interesting traffic can be replayed exactly as it was captured, creating a controlled environment to investigate new unknown threats. Combined with Solera DeepSee™, organizations can search through the captured data to create a real world context around a threat by rendering “artifacts.
Please don't stop blogging just because you made a mistake. We all do that. It's how you conduct yourself following your mistake, and how you learn from it that matters. I think you folks conducted yourself with class, and made the best of a bad situation.
If you're not posting for completely different reasons, don't have the time, on fire, etc. please disregard.
Kindly,
--Mike