DISQUS

jskennedy · 3 years ago

Interesting, informative and amusing! I just became aware of you guys this week and am very impressed and excited about the work you guys are doing.

One note.. one of the worst enterprise remote command programs I have seen is the Compaq Insight Manager/ILO. If you have not looked at it yet, when you do you will have a lot of fun. What is sad is that it is rampant and basically exists on any substantially sized network that has HP or Compaq servers. The best mitigation currently since there is relatively little vendor support on these types of apps is to isolate them into a management zone where the only traffic that exists is the remote control of the servers. I guess that is one nice thing about Compaq Insight Manager/ILO since it is basically another computer on a board it can be isolated versus an agent on the box. Another interesting thing would be to see if it is possible to get to the CIM/ILO from within the OS that is on the box in a similar fashion as VMWare. Compliance of password management on these things are also a nightmare as most environments I have seen have a universal password that all admins use that has never changed! Scary when you see someone have remote bios level access to a system with a password that is "secret". Especially when the box is housing highly sensitive data.

Enough rambling.. again I appreciate the work you guys are doing and look forward to reading more!

wrc · 3 years ago

It was worth every penny.

wpn · 3 years ago

The Aristocrats joke had me rolling on the floor!!!

Ryan Russell · 3 years ago

I appreciate your making the clarification for me. After being listed as a player in the area (which we like), we would not have been in good company had we been one of the ones you looked at. So thank you for letting the audience know that we weren't one of the ones tested. Better to be left in the unknown category, in this case. :)

I will likely be publishing a paper about our architecture and design, per your suggestion.

Kal · 3 years ago

Great read and highly insightful.. I think the major issue here is getting around the mindset of those holding the purse strings. How often have you heard "but we spent all that money on firewalls, we're safe aren't we?". Considering the TCO reductions that these pieces of software bring I don't think that even as security professionals with this sort of excellent research, convincing the management will be any easier.

Thomas Ptacek · 3 years ago

Thanks, Kal. I just want the vendors to turn TLS with client certificates on by default. Is that too much to ask? =)

Ryan Russell · 3 years ago

Tom, if you just use TLS, then that still leaves the problem of compromising the central server gives you control of the agents. I assume you want the client-side certs to mitigate bad data being sent upstream? You still have to do adequate filtering, with so many clients to compromise. If you get one, then you can submit bogus data.

So, to give you a preview of our design, we don't do encryption down to the agents (the info isn't secret), but instead we do signing. The agents use a cert chain to verify that the command came from an authorized admin. All the security is in the private keys, and each admin has their own. I think that is solving the same problem you see, in a slightly different way.

Thomas Ptacek · 3 years ago

Ryan, what you're saying makes sense. What I want to clarify though, and I'm referring to vendors other than BigFix, is that TLS is a total stopgap solution.

The problems we found were such that an attacker than simply knew how to generate the right sequence of bytes could take over every agent on a network without owning either an agent or the server. They were effectively "blind"; you could write a scanner to track them down.

What TLS does in these circumstances is get you past the blind attacker; you at least need some legitimate cause to talk to the agents or the servers in the first place. Even that basic level of access control is a problem for some of these systems.

The problem you confront after you enable TLS is that the agents aren't going to have 10,000 different secret keys, and there's no way the management server is going to propagate certified authorizations everywhere, and even if it did, control of the agent still gives you a great deal of influence over the server and the various UI's that talk to the server.

One of my clients keeps yelling this at me. "Stop saying TLS, Tom! That doesn't solve the problem!" He's right. It does mitigate the exploit that worries me the most, though.

Ryan Russell · 3 years ago

OK, so you mean "If you do nothing else, use TLS." Fair enough. Our agents do pull via HTTP, rather than our server doing push. Though, there is a notification channel downstream via the world's simplest UDP protocol, which can be disabled if you like. But I'll try to refrain from going on further about how awesome our design is. I'll let you guys know when I've got a whitepaper together.

Douglas W. Stevenson · 3 years ago

Holy Sheep Dip, Batman! Looks like a wee bit of DAG!!!!

I see a whole new industry... Internal Enterprise Security!

On the management protocols side.... Theres NO reason anyone should not be running SNMP V3 with Auth + Encryption. no excuse. But what the hell do you do with SOA based crap? Or WMI? What about WBEM? Some of this stuff is soo wide open it almost seems unsecureable.

And running RPC ... Dangerous is what comes to mind.

The one thing I think about now is placing my thought process in the terms of a Worm or malicous attacker. OMG - More bugs than Joe's Apartment! Alot of these agents are pretty "tender". I can envision more buffer overflows that a handful of Alka-Seltzer tablets fed to a Seagull!!!

Hal@OTB(++) · 2 years ago

None of the 'Vendors' have even thought of this as a 'problem' and it's a bit difficult to imagine that they even think it's worth working on until it becomes financially important. One thing to think about from an adversary's perspective is that is it is extremely easy to 'hide' on a target network if you simulate the target's managment routine/use the same tools/look like the Admins..... so using 'backeddifice8' or something easily tagged as 'wrong' is not good, but a rogue copy of 'GOopenview-able' or some such make it real hard for IDS teams to spot... thus fixing this is really, really, .... well... just a plain ol' good idea... agree/disagree????

HAL · 2 years ago

One more thought for consideration. The updates to some agents are actually better, however, the bigs remain just as broken. WMI and the SOA comment from Douglas are on the mark.

When taking over a network, and if you don't have access at the lower levels (via hardware or embedded systems - you guys/gals know who you are, still a big fan, still have the death star badge), imitating management traffic of the target still remains the top method for remaining undetected. IP address provisioning is still spotty in most shops I've seen, and thus they are still not connecting the dots in the NOCs.

Thus, if you can't fix these tools, and must use them, the solution is to run them smartly. There is still a large gap in practice, leaving the intruder a very wide margin. Seen it, done it. Guess this whole subject is losing interest for folks, as VMs are the interest of the day. My money is on covert hostile VMs being deployed through this 'margin'. By the open, average, everyday types soon enough.

Best, H