DISQUS

Matasano Chargen: What I’ve Been Doing On My Summer Vacation or, “It has to work; Otherwise gdb wouldn’t”

  • Loic · 1 year ago
    Regarding your remark about iTunes in point 8, iTunes does not seem to interfere with DTrace (and ptrace(), truss, ...), it actually interferes.

    See for instance and its followup .

    In short, some process are (made) "untraceable" in Mac OS X.
  • Joe Ranieri · 1 year ago
    The basic problem is that OS X isn't UNIX - it's Mach. Therefore, you have to do all the same things you would do when writing a Mach debugger - task_suspend, task_threads, thread_get_state, vm_read, wm_write, etc. Once you ignore ptrace, everything's straight-forward.
  • Nate McFeters · 1 year ago
    Well intern, well played, looks like you are fitting in well over there then. Although, you probably could've asked David Weston and Tiller Beauchamp about this, since I think they went through the same stuff with RE:Trace and Redbg. That wouldn't have made for the learning experience though.


    Loic said, "Regarding your remark about iTunes in point 8, iTunes does not seem to interfere with DTrace (and ptrace(), truss, …), it actually interferes.

    See for instance and its followup .

    In short, some process are (made) “untraceable” in Mac OS X."

    I think this may have to do with the PT_DENY_ATTACH problem, which I have run into numerous times now. See here: http://landonf.bikemonkey.org/code/macosx/ptrac...
  • Nate McFeters · 1 year ago
    Also, I've got project number two for you... get your debugger to work with a PowerPC binary emulated through Rosetta on the Intel architecture. This is where I need Weston and Beauchamp to get RE:Trace/DTrace/Redbg to. You wouldn't believe the headaches I had trying to flesh out a heap overflow in Adobe Illustrator CS 2 because of this.

    See here: http://developer.apple.com/documentation/MacOSX...

    Nate
  • Thomas Ptacek · 1 year ago
    It seems to me that the basic problem here is that OS X is neither Unix nor Mach, but a strange hybrid of both. Apple's gdb, for instance, does not ignore ptrace.

    The Mach "debugger interface" is much nicer than ptrace, though.
  • Phill · 1 year ago
    That is awesome.
  • Wesley McGrew · 1 year ago
    Reading this really made my morning.

    Outstanding work, and very good writing and commentary on it. Makes me want to poke around in OS X myself now.
  • Tiller Beauchamp · 1 year ago
    System programming on OS X is like crawling through a dark cave. I highly recommend the OS X systems internals book, although its a bit dated. Also, rather than grepping through /usr/include, try this site for searching / browsing the XNU source: http://fxr.watson.org/fxr/ident?v=xnu-1228
  • Matt · 1 year ago
    Ahh, three cheers for throwing interns into the deep end. My first internship in college:

    Boss: "We have this spit-and-baling-wire router OS that needs some work."
    Me: "Um... I just now learned what a socket is."
    Boss: "Here's Ethereal. Go look at what such-and-such a thread is actually putting on the wire."
    Me: "... Thread?"

    Those were the days. Congrats on some worthy hacking, Timur!
  • Timur · 1 year ago
    @Joe Ranieri
    Thomas is right on this one. OS X is the epitome of the phrase "Unix like OS." It has parts from Unix (mostly BSD), Mach, and some things pulled from Linux.

    Secret Intern X had originally suggested a second "stealth" debugger for OS X written entirely in Mach system calls. In researching and implementing DebuggerX, I was unable to find a Mach replacement for PT_STEP. Also, as far as I could find there is no way to prevent the traced process from performing its default signal handling without attaching with ptrace. For the debugger to work the parent process has to handle the trap raised when the process hits the breakpoint.
  • Thomas Ptacek · 1 year ago
    You can stealth-debug single step using Conover and Rafal's user-mode single-stepping trick:

    http://www.matasano.com/log/628/rafal-wojtczuks...

    (Of course, you have the conditional logic problem, and if you're stealth debugging you can't use ptrace to resolve it).
  • Timur · 1 year ago
    @Nate McFeters

    If I knew how that gdb/rosetta setup worked I'd be all over it.

    I think the code I wrote would work to debug the process. However, you'd really be debugging Rosetta while it runs something else. Which would only work well if Rosetta doesn't call DENY_ATTACH.

    My guess is that setup uses the Mac message ports to allow gdb to hook into Rosetta.

    Definitely something I will look into more as I have time.
  • Nate · 1 year ago
    grep -ri /usr/include ptrace
  • Jesse · 1 year ago
    Your next task should be adding immunity to processes that have called ptrace() with PT_DENY_ATTACH to prevent themselves from being debugged. There are a number of ways to do this, but it's a bit of a hassle to deal with every time :)
  • Ryan Russell · 1 year ago
    Hmm... you did that in two weeks as a n00b coder? You're going to be a pretty good coder.

    Also, kernel panic from userland, you say?

    You're using just procmod privs?
  • Thomas Ptacek · 1 year ago
    I don't think he is; there was some, um, controversy in the office about whether he could get it working with "just" procmod, and he had wound up in a sudo irb to test.
  • Dave G. · 1 year ago
    Nate:

    grep -ri ptrace /usr/include :)
  • Snagg · 1 year ago
    I might have misinterpreted the problem Timur But you might be able to use thread_set_exception_ports() for your purposes and register a port (a port in the sense of Mach port) in the parent process. That way you should be able to do single step without using ptrace. uhm Anyhow I'm not that sure EXC_MASK_BREAKPOINT is supported under osx. As regards PT_DENY_ATTACH you should be able to overcome the problem in at least two ways: 1) prevent ptrace from being called (tons of people talked about that 2) use the trick with thread_set_exception_ports() I mentioned before, remember that Mach is under ptrace() so he will by-pass any kind of restrictions.
  • Snagg · 1 year ago
    Forgot to post this link: http://www.autistici.org/snagg/cve-2007-3749.c .
    This was a poc I wrote a while ago, you might find it useful; it deals with thread_set_exception_ports().
  • Timur · 1 year ago
    @Ryan Russel

    I am running this sudo'd and as procmod. Leopard requires code to be signed or run as root to use task_for_pid().

    Yes, kernel panic. I have a little more testing to narrow it down fully, though.
  • Timur · 1 year ago
    @Snagg

    While I did see those exception port functions, I didn't wan to spend the time just yet to get them working through Ruby/DL. I seem to remember them requiring callback functions, which I'm still not sure how to setup from Ruby/DL.

    As far as I was able to tell, the ptrace call you're thinking of that takes four arguments is a higher level wrapper for a Mach version of the function. Nate McFeters linked to an analysis of that function. The source code for it is located:

    http://fxr.watson.org/fxr/source/bsd/kern/mach_...

    Unfortunately, I was unable to find source code for the four argument version to verify this.

    About the only way I can think to get around the PT_DENY_ATTACH problem from Ruby would be to overwrite the function in memory and fork a child to debug.

    Or, installing the kernel extension linked above.
  • Nate McFeters · 1 year ago
    @Timur

    If you are going to Black Hat this year, try to find me or Tiller and David. I think that getting it all working with Rosetta is a bit more complicated then it seems, since they've struggled with it. Of course, who knows with Mac, it's like a black hole.

    I agree with @Jesse, it would be desirable to have PT_DENY_ATTACH just handled by the debugger. There is kernel patches (I believe this is what I used) and of course there's the methods that @Snagg mentioned, but it would just be so nice to not have to deal with that at all. At a minimum, maybe mention of it in any release notes you put out once this thing is done. It's pretty simple to fix once you know it's the problem, but I banged my head against the wall for a bit.

    If you guys are doing much Mac vuln research, you should really look into RE:Trace by David Weston and Tiller Beauchamp (see http://poppopret.org/). I've found it to be the most useful tool you can get.

    -Nate
  • Nate McFeters · 1 year ago
    @Timur, that's exactly what you'd have to do. I don't know that there would be another way. To be honest, just kernel patching is fine, it's straightforward and simple... I would just suggest that if you release this as a tool for mass consumption, put in a link to a place (or maybe even put in the readme) that this should be done.

    Funny thing is, I had heard of it long before, maybe even in an uninformed paper, but when I got to doing some research, totally forgot it, bashed my head against the wall for hours till I remembered.

    -Nate
  • freno · 1 year ago
    "A 'simple' porting task handed to an intern to better his understanding of debuggers and Ruby."

    For Ruby you might also want to read "Inspecting a live Ruby process"!

    http://weblog.jamisbuck.org/2006/9/22/inspectin...
  • Snagg · 1 year ago
    @Timur
    I don't quite get what you mean by "Unfortunately, I was unable to find source code for the four argument version to verify this.". Anyhow yes, it is something like a "wrapper" but still not entirely done with Mach, for example single stepping is done by setting the trap flag in EFLAGS.
    To be honest I'm not that much a ruby guru, but I guess you can set up callback function somehow.

    As you may notice reading the code, PTRACE_DENY_ATTACH basically drop exceptions ports from the process. But you should be able to attach a new one, providing you're root. So I guess using exceptions ports is the only "clean" method to avoid DENY_ATTACH at runtime.
  • Thomas Ptacek · 1 year ago
    freno --- that is indeed a good blog post, but Jamis is using GDB to inspect processes. Timur's job was to write GDB.
  • corio · 1 year ago
    You bloitin rock!

    About died when I saw the use of such fowl language in a technical post... but after reading the post, I understand!

    You were given a blit of a task and you didn't bloit it off!
  • Timur · 1 year ago
    @Nate McFeters

    I would love to work on that with you at Black Hat. Alas, I cannot attend this year.

    I am, however, working toward a version that forks, calls TRACE_ME, then breaks on ptrace to jump past it. I'll let you know how it goes.
  • Nate McFeters · 1 year ago
    Do that, that would be cool to see. Congrats on the position at Matasano, it's def. a big deal. If you are in Chicago, we should meet up for lunch sometime. Just saw Tom, Eric, and Dave G. last week and it was good to catch up... would be better to do beers though.

    -Nate
  • Rudd-O · 1 year ago
    Guys,

    Tell ecopeland that the next time he wants to publish something for a later date, he should not hit Publish before setting the future date (I'm sure it was an honest mistake). Stuff that goes through RSS stays cached on users' disks.

    Therefore, I (along with probably thousands of other people) have the DNS vuln article. Very cool work. Point 5. really surprised me; I never knew DNS code for BIND and co. was so braindead as to not perform such a simple and urgently-needed sanity check. Sometimes it's easy to skimp on data structures and algorithms if you're going for the quick fix... sometimes one pays dearly because one didn't look beyond 1995.
  • Jesse · 1 year ago
    As far as PTRACE_DENY_ATTACH goes, you can easily swat it down by injecting your own library to hijack ptrace() calls via DYLD_INSERT_LIBRARIES. But, that will only work for loading new processes, not attaching to those that have called it already. Be sure to set DYLD_FORCE_FLAT_NAMESPACE as well if necessary. Check the dyld man page for more info.
  • Nate McFeters · 1 year ago
    @Jesse, yeah agreed... probably not the most effective way to do it. Honestly, I'm thinking just documenting it somewhere with the tool would be more than enough... I mean, probably a lot of people would've searched the internet before banging their head off the wall for as long as I did... guess I'm just not hooked up that way.

    -Nate
  • Timur · 1 year ago
    @Nate McFeters and Jesse

    I managed to get the fork version to work and set a breakpoint in at ptrace before I running exec().

    Didn't work, at least not after Kernel#exec was called. I had worried exec would completely create a new environment for the new process. I was right. I'm going to look into exaclty what gdb is doing in that case. It seems like it can't really be using exactly the fork/exec model.

    Perhaps it's manipulating its own child process instead of using exec.

    I'll be at Chisec wednesday if you'd like to discuss this in person.
  • haypo · 1 year ago
    I wrote a Python binding for ptrace:
    http://fusil.hachoir.org/trac/wiki/Ptrace

    It's not only a ptrace binding: it also contains a basic debugger similar to gdb but writing only in Python, and strace.py: system call tracer with argument parser/formatter.

    The problem with a debugger is that you have to change constants for each OS/CPU. So I use different methods to get registers, read process memory, etc.

    ptrace/binding/func.py and ptrace/debugger/process.py are the most interesting files.
  • anonymous · 1 year ago
    C debugger not written in C?

    Sometimes I just don't get you [Matasano in general] and your [Matasano's] apparent hatred for this language.
  • Thomas Ptacek · 1 year ago
    The irony is, the rap on me is that I write too much in C!

    There's a zillion great reasons to have a debugger in a high level language, starting with "you can write a special-purpose debugger for a new project in 5-10 minutes".
  • Timur · 1 year ago
    Where I doing this in C, there wouldn't have been cause to blog about it.

    This debugger also will debug any process, not just those written in C.

    The C functions were wrapped because they are what the OS uses and were the most accessible. Many of the functions in this ruby object were tested in a small C program as well since I was having problems getting the Ruby to work and there is little written about some of the calls.
  • Timur · 1 year ago
    @haypo

    I don't know that much Python, so I could be wrong here.

    One thing you could do to get around having a specialized object per platform would be to build a core object and add to is by selectively loading files based on architecture and OS.

    That way on the user's end all they have to do is say they want a debugger object and the code does the heavy lifting to determine what that object needs to look like.
  • Timur · 1 year ago
    @Nate McFeters
    @Jesse

    Yesterday, I was able to figure out my fork and exec approach did, in fact, work. I managed to successfully break on ptrace and return immediately from it. The current method skips all ptrace calls by the child so it still needs some work.
  • mish · 1 year ago
    @Timur

    This was a fascinating post. I hope you'll have time to document your next interesting project that you learn a bunch on, that's always the most fun stuff to read.

    @Thomas

    I'm a total n00b when it comes to some of this stuff, static analysis, etc. Can you expand more on why you would want a purpose-written debugger for a binary? I was under the impression that debuggers kind of all did the same thing...I'm guessing that's not the case. Could you explain further what you mean and why it's advantageous to do in Ruby (really, what kind of things can you do now that it is templated out in a high-level easy to write language)?
  • Jesse · 1 year ago
    @Timur

    Cool, good work. It's probably rare that a typical application will call ptrace() for anything other than PTRACE_DENY_ATTACH, so blindly returning instantly will probably be sufficient...but for completeness, it would be good to verify.

    You should take a look at nemo's paper from uninformed if you haven't already, he's got a lot of info you might find interesting.

    Does anyone know if a process can undo a PT_DENY_ATTACH request that it has made previously?
  • Jesse · 1 year ago
    Oops, your original post already mentioned you went through the uninformed article, disregard ;)