Matasano Chargen - Latest Comments in Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Fri, 02 Mar 2007 14:16:27 -0000

I can't read this; there's zero chance I'm going to pay $175 to read "Virus Bulletin". Can you summarize?

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

newsham — Fri, 02 Mar 2007 13:37:16 -0000

Dr. O'Donnell jumps into the fray: http://www.virusbtn.com/virusbulletin/archive/2...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Mon, 28 Aug 2006 17:50:37 -0000

@thomas
if you're looking to evaluate the existence of the anti-virus research community, give it up already... it's real even if you don't believe in it...

if you're looking for actual research i've already provided a page with links to literally thousands of papers, most of which concern viruses... if you need more, google scholar will save you from having to search through multiple sources...

conferences such as vb and eicar tend to be more prestigious than various publications (as people such as yourself tend to disrespect the publications)... there is the journal of computer virology, however...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Mon, 28 Aug 2006 15:26:34 -0000

So list the three "most prestigious" venues for research purely on "self-replicating code", and I'll do the homework for you.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Mon, 28 Aug 2006 15:15:16 -0000

@thomas
"Give me a break. This isn’t a page of virus research; it’s a page of all security research, in networking, systems, algorithms, and architecture that apply in any direct way on malware in general."

the main focus is still self-replicating code...

"Vern Paxson is not a virus researcher, and neither is Nick Weaver."

if they research viruses then they are virus researchers - at least part of the time...

"How about this: make a list of people who have published more than once in an exclusively virus-oriented “journal”, like the proceeding of Virus Bulletin conferences. Then cross reference how many times those people publish in ACM journals, or Usenix."

how about this... see up there where i said i wasn't going to do your homework for you anymore - try reading that a few more times until it sinks in...

debating the very existence of an anti-virus research community is insipid... despite the current employment status of it's members the computer anti-virus research organization is not an industry organization, and the european institute for computer anti-virus research is not run by the vendors... the association of antivirus asian researchers is also independant from the industry, plus there are a variety of universities that are active in virus research such as (off the top of my head) the university of hamburg and the university of tampere...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Mon, 28 Aug 2006 13:29:01 -0000

Give me a break. This isn't a page of virus research; it's a page of all security research, in networking, systems, algorithms, and architecture that apply in any direct way on malware in general.

Vern Paxson is not a virus researcher, and neither is Nick Weaver.

How about this: make a list of people who have published more than once in an exclusively virus-oriented "journal", like the proceeding of Virus Bulletin conferences. Then cross reference how many times those people publish in ACM journals, or Usenix.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Mon, 28 Aug 2006 12:46:17 -0000

@thomas
"One of the aspects of this argument that may be frustrating you"

frustrating to me? i'm not the one leveling accusations of talking in code and dishonesty...

"is that I give absolutely no credence to the argument that “self-replicating code” is so dangerous that it can’t be created for the purposes of research or testing."

not "can't"... *shouldn't*... the risks are real whether you choose to believe in them or not...

"And no, I don’t believe “viruses”, as we are both likely to define them, “find new victims autonomously”. I think that in the manner that viruses use computers, their actions are deterministic and bounded."

autonomous and deterministic are not mutually exclusive... even a computer science undergrad would have heard of deterministic finite state automata...

if you truely doubt the autonomous nature of viruses then we are once again left to conclude that you do not understand what it is you're talking about... although certainly deterministic given a set of known inputs (drawn from the data, resources, and environment), the inputs are not knowable in the wild... you cannot accurately predict what kind of environment a virus will encounter, what data it will find, what kind of network resources it will have at it's disposal, or who the infected files will subsequently be shared with...

"Give me a scenario in which _I_ am likely to spread a virus (say, an OSX 10.4 kernel/dylib infector) that I write myself to test “AV” techniques on my Mac."

since i did not say that anyone who creates a virus will innevitably release it the above challenge is irrelevant... you may well not spread it (though i certainly don't trust you enough to share viral materials with you)... the fact of the matter is that the most likely avenue for a virus to escape through an accident and accidents happen - are you perfect? have you never had an accident?

"So, the best example of credible virus research you can cite from the literature:"

i'm going to nip this in the bud right now - it is not the best i could find, it's simply one from the first page of results from google scholar... you wanted something recent that was published outside of normal places one would find anti-virus research published so i found one from 2005 that was published by the ieee...

"Again I believe there’s credible virus research out there of a quality comparable to the best work done, even in malware, by, say, Stefan Savage, Vern Paxson, or Nick Weaver. You haven’t found it yet, though."

http://members.tripod.com/~k_wismer/papers.htm

you still think i haven't found it yet? all 3 of the people you mention are listed there... you wanted something recent and since i don't keep track of the publication dates i had to go to google, otherwise i would have just pointed you here to start with...

of course, my collection is not comprehensive, nor strictly constrained to just anti-virus research, however it should be adequate to demonstrate that there is indeed an anti-virus research community...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Mon, 28 Aug 2006 10:53:54 -0000

So, the best example of credible virus research you can cite from the literature:

1. Is from a security generalist

2. Is a guided tutorial of how to use IDA Pro

3. Has 1 journal cite, as a place to learn more about the Bagle worm

Again I believe there's credible virus research out there of a quality comparable to the best work done, even in malware, by, say, Stefan Savage, Vern Paxson, or Nick Weaver. You haven't found it yet, though.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Mon, 28 Aug 2006 10:46:17 -0000

One of the aspects of this argument that may be frustrating you is that I give absolutely no credence to the argument that "self-replicating code" is so dangerous that it can't be created for the purposes of research or testing.

And no, I don't believe "viruses", as we are both likely to define them, "find new victims autonomously". I think that in the manner that viruses use computers, their actions are deterministic and bounded.

Give me a scenario in which _I_ am likely to spread a virus (say, an OSX 10.4 kernel/dylib infector) that I write myself to test "AV" techniques on my Mac.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Mon, 28 Aug 2006 01:51:22 -0000

@thomas
"“They simply have to do it right” is code for “they need to do it the way we tell them to”. You don’t decide what “right” is."

i grow tired of having my words twisted to mean something entirely different than what i said... "doing it right" is simply a restatement of "not doing it wrong"... the CR/ISE test did things wrong that have already been explored in depth in the past and have been pointed out here and elsewhere...

"Virus Bulletin and the “field’s own periodicals” don’t count. You’re a smart guy. Go find me real research. Journal articles. ACM Transactions. Things where the majority of the “peer community” don’t work at AV companies. Virus Bulletin is a magazine. I might as well cite my Dark Reading press hits, or this blog."

fine - http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?...
but that's the last time i do your homework for you...

excluding the av field's own periodicals is pointless and arbitrary as it is where the lion's share of the relevant hits will be... your concerns about who the peers will be is meaningless as they will be drawn from the same places regardless of what publication a research paper is published in... peers in the av field don't magically change depending on where you publish...

"By the way, I’m sure quality research on viruses exists. Just not in anything resembling the numbers that quality systems security research does."

oh great, so now it's a "mine's bigger" argument is it? well duh - of course a highly specialized field is going to have less published research than a more general one...

"Best of luck tracking it down, though!"

oh yeah, so hard - google scholar is your friend...

"The fact that you can’t see why a countermeasure vendor would want to replicate an outbreak to test a product tells me you don’t spend a lot of time developing security product."

and the fact that you think it's necessary to replicate an actual outbreak to test anti-viral countermeasures tells me once again that you do not understand anti-virus technology, practice, or issues...

"A hint: what people first think will solve security problems usually doesn’t."

a hint: trying to apply principles from your field of expertise to a different field is the cornerstone of false authority syndrome...

"A “new virus”, exhibiting no novel capabilities other than the fact that antivirus companies haven’t heard of it and thus can’t detect it, developed by a researcher with her own private resources, presents almost no meaningful risk."

perhaps you meant to include more conditions there that might actually support your conclusion of no meaningful risk because so far you've failed... a virus developed by a researcher with his/her own private resources is able to spread just as well as one written by your average virus writer - in fact virus writers generally develop their viruses with their own private resources... some even do research...

"The notion that this activity is even comparable to the development of a new zero-day Win32 remote isn’t"

i assume you meant "remote exploit" and no, i don't think they're comparable - development of the virus is worse by far in the long run and since viruses in general don't rely on software/hardware flaws as exploits do there is no patch or other correction that can be made to mitigate the threat it poses...

"just ignorant. It’s dishonest."

great, so now instead of just speaking in a deceptive code i'm actually lying... tell me oh guru what could possibly motivate me to be dishonest about the threat posed by viruses when i am neither part of the anti-virus industry nor do i have any published anti-virus research to defend...

perhaps if i am being dishonest that dishonesty can be revealed by say identifying a statement of fact which is false... are viruses not able to find new victims autonomously? did stoned.empire.monkey not survive in the wild for over a decade? perhaps viruses really are based on software flaws that can be patched? or maybe it's something that hasn't yet been explicitly said, like maybe the fact that the set of people who qualify as the "wrong hands" for viruses to fall into is orders of magnitude larger (making the chances of encountering one much much higher) than the set of people who qualify as the "wrong hands" for exploits - perhaps that's false somehow...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Chris_B — Sun, 27 Aug 2006 21:29:44 -0000

"suprising problems" No. Not surprising at all. AV Engines from any company tend to be pretty much as good as the latest DAT file. AV companies are

Oh and yes, test case viruses do get out of the lab. Happned more than once while I was at NAI Japan.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Sun, 27 Aug 2006 14:00:14 -0000

1.

"They simply have to do it right" is code for "they need to do it the way we tell them to". You don't decide what "right" is.

2.

Virus Bulletin and the "field's own periodicals" don't count. You're a smart guy. Go find me real research. Journal articles. ACM Transactions. Things where the majority of the "peer community" don't work at AV companies. Virus Bulletin is a magazine. I might as well cite my Dark Reading press hits, or this blog.

By the way, I'm sure quality research on viruses exists. Just not in anything resembling the numbers that quality systems security research does. Best of luck tracking it down, though!

3.

The fact that you can't see why a countermeasure vendor would want to replicate an outbreak to test a product tells me you don't spend a lot of time developing security product. A hint: what people first think will solve security problems usually doesn't.

4.

A "new virus", exhibiting no novel capabilities other than the fact that antivirus companies haven't heard of it and thus can't detect it, developed by a researcher with her own private resources, presents almost no meaningful risk. The notion that this activity is even comparable to the development of a new zero-day Win32 remote isn't just ignorant. It's dishonest.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Sun, 27 Aug 2006 11:33:44 -0000

sorry for the tardy response - b-day celebrations take precedence over these sort of things

@thomas
"We’re well aware that the AV industry doesn’t care about the ISE test. ISE didn’t play by your rules or ask permission or “earn” their right to test your stuff by putting years into the trenches and making up Sara Gordon jokes."

no one needs to ask permission or earn any right - they simply have to do it right...

"Is there really even a “virus research community” to speak of? What’s a good example of a journal-quality article done recently in AV?"

you mean like the stuff that's published each month in virus bulletin? or do the av fields own periodicals not count?

"By the way, the “you can’t create new viruses!” argument? That’s code for “you don’t work for an antivirus company! Go away!” "

no, it's code for "don't be part of the problem"...

"If real security companies aren’t actually simulating worm outbreaks (you know, real malware?), I’m disappointed."

while i can think of a paper or two on computer virus epidemiology that most likely involved controlled outbreaks in a closed test network, i'm not sure what else you think can actually be learned from outbreaks that would have people orchestrating them as part of a commercial product or service...

"The “hazmat” a typical security researcher works with is 1000x more “dangerous” than some random virus that nobody is ever going to get infected with."

that remains to be seen... while i'm sure there are things that create a bigger 'bang' than a virus or worm, nothing is as hard to control as self-replicating code... unlike other types of malware or exploits that stop being an immediate threat when the people using them stop doing so, viruses and worms keep going and going without intentional assistance... stoned.empire.monkey was in the wild and on the wildlist for over a decade in spite of the fact that anti-virus software could detect and remove the virus for almost that entire time... few other threats are as completely autonomous as self-replicating code...

"Somehow we manage to talk about it, study it, and generate it without “ethics” hysterics."

concern over the needless manufacture of new hazards is not 'hysterics'...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

SpannerITWks — Sat, 26 Aug 2006 08:32:49 -0000

Hi,

There's thread going on in the Security forum over here - Our unique antivirus testing: How we did it - http://www.dslreports.com/forum/remark,16725030... - that several figures in the " Biz " have contributed to, in various ways. I and others have also included our thoughts and ideas etc in there.

I recently referenced this thread, and some of your views about Variants + Retrospective testing in there, which has been responded to, so you may like to add your comments etc.

Spanner

SpannerITWks

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

SpannerITWks — Sat, 26 Aug 2006 08:30:18 -0000

Hi,

There's thread going on over here - http://www.dslreports.com/forum/remark,16725030... - that many figures in the " Biz " have contributed to, in various ways. I and others have also included our thoughts and ideas etc in there.

I recently referenced this thread, and some of your views about Variants + Retrospective testing in there, which has been responded to, so you may like to add your comments etc.

Spanner

SpannerITWks

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Sat, 26 Aug 2006 01:42:44 -0000

Except for Peter Lindstrom, who wants to put us in jail.

Good luck with the IBM lawyers now, though, Peter. :P

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Sat, 26 Aug 2006 01:41:51 -0000

We're well aware that the AV industry doesn't care about the ISE test. ISE didn't play by your rules or ask permission or "earn" their right to test your stuff by putting years into the trenches and making up Sara Gordon jokes.

Is there really even a "virus research community" to speak of? What's a good example of a journal-quality article done recently in AV?

By the way, the "you can't create new viruses!" argument? That's code for "you don't work for an antivirus company! Go away!" If real security companies aren't actually simulating worm outbreaks (you know, real malware?), I'm disappointed. The "hazmat" a typical security researcher works with is 1000x more "dangerous" than some random virus that nobody is ever going to get infected with. Somehow we manage to talk about it, study it, and generate it without "ethics" hysterics.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Fri, 25 Aug 2006 22:45:28 -0000

@newsham
"The fact that ISE’s tests point this out is a thorn in their side."

existing retrospective testing shows the same things that ISE/CR's test showed, and retrospective testing does so *without* making new viruses...

the av industry and research community don't care about the results of ISE/CR's test because the results aren't anything that can't be found elsewhere, what the av industry and research community care about is the manner in which the test was performed...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

newsham — Fri, 25 Aug 2006 20:06:02 -0000

I think the underlying point of contention here is that Tom (and others like him, perhaps myself included) disdain anti-virus since it is nothing more than pattern matching against known threats and weak hueristics to catch a few unknown threats. Its not a sound analysis to detect all threats, and it can never be. The virus community understands that, and hence views the ISE tests as silly and without merit. But, the general population of people using virus scanners do not know this. The fact that ISE's tests point this out is a thorn in their side. The fact that Consumer Reports, a reputable name in consumer advocacy and education, is pronouncing this loudly and clearly is a major fiasco. This warms the hearts of people like Tom.
That's the downside of building a product around undecidable problems. As a consolation prize, the AV community gets job security in perpetuity (so long as they don't mind working in redmond).

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

andi — Fri, 25 Aug 2006 16:51:59 -0000

http://www.av-comparatives.org/weblog/?cat=7
I totally agree in nearly all points with Kurt Wismer :)

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Fri, 25 Aug 2006 16:13:41 -0000

@thomas
"No it wouldn’t. Look at what happened to TBAV."

yes, let's look at what happened to tbav, shall we?

tbav's technology was bought by norman data defence and what does norman have now? sandbox technology that goes above and beyond conventional heuristics...

"The AV industry does not run on innovation; it runs on and is forever locked into a stifling symbolic relationship with a bunch of teenagers and low-rent criminals who’s best avenue to create havoc, in 2006 mind you (!), is to write malicious Office macros."

a) i believe you meant 'symbiotic'...
b) macro viruses are considered old school now - that fad wore itself out some time ago...
c) i believe you are referring to the continued reliance on signature scanning - clearly you don't see that signature scanning remains the most effective technique in so far as it can deal with more of the set of known malware than behaviour based systems can and because unlike behaviour based systems it can operate on the malware *BEFORE* the malware gets cpu resources and is able to defend itself...

"There are smart people who work in AV, and they have my sympathy. With respect to the core CS problems AV grapples with, their research community has been eclipsed by OS and application security. In my field, people write instruction set emulators as plugin scripts, and release disassemblers and tracers as open-source freeware."

emulation can be detected, disassemblers and tracers require skilled operators...

reverse engineering an arbitrary suspect sample is still an intractible problem and neither your field nor the av field, nor any other field is ever going to be able to change that fact... as such, what can be done in a virus lab and what can be done on a user's desktop are 2 entirely different things...

"By the way, with regard to the “trust relationship” CARO is built on: trust is nice, but smart people shouldn’t have to ask permission to improve the state of the art."

building trust relationships is not the same thing as asking for permission... it's not even comparable...

smart people shouldn't have to ask permission to improve the state of the art with respect to biological viruses either, but you don't see the cdc handing out free samples at the local grocery store...

smart people can do whatever it is they want to do, but being smart does not automatically entitle one to hazardous materials, nor would there be any good way to make that distinction if it did entitle one...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Fri, 25 Aug 2006 13:04:21 -0000

By the way, with regard to the "trust relationship" CARO is built on: trust is nice, but smart people shouldn't have to ask permission to improve the state of the art.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Fri, 25 Aug 2006 12:54:42 -0000

No it wouldn't. Look at what happened to TBAV. The AV industry does not run on innovation; it runs on and is forever locked into a stifling symbolic relationship with a bunch of teenagers and low-rent criminals who's best avenue to create havoc, in 2006 mind you (!), is to write malicious Office macros.

There are smart people who work in AV, and they have my sympathy. With respect to the core CS problems AV grapples with, their research community has been eclipsed by OS and application security. In my field, people write instruction set emulators as plugin scripts, and release disassemblers and tracers as open-source freeware.

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

kurt wismer — Fri, 25 Aug 2006 12:49:59 -0000

@thomas
"Martin, if none of the techniques required to make malware “unknown” to an antivirus engine are clever, what does that say about the engines themselves?"

that they aren't written with the goal of detecting every conceivable transformation of an existing virus because it's just as intractable as detecting all possible viruses...

"Kurt, apart from knowing the secret password to VCL and not knowing what the “op” in “opcode” is short for,"

oooOOoo, i mistakenly typed opt-code instead of op-code... you should see what my brain used to have me type instead of himem.sys back in the days of dos...

"what qualifies you to determine who is and is not part of the “antivirus industry”?"

i guess after 17 years following the field i just have a knack for determining whose opinions come from knowledge of the field (and therefore are likely a part of it) and those whose opinions do not...

"I’m sorry to be such a jerk. I just think the “AV research” community is a big silly clique."

while i'll agree that CARO is a clique, i don't really see why it should be called 'silly'... it's a clique formed out of trust relationships rather than the more conventional cliques you see in highschool...

"“Real AV researchers have ways of getting access to what the bad guys write WITHOUT writing new viruses” is just code for “you can’t play in our sandbox unless you have a job with one of 10 approved companies”."

complete misinterpretation... real av researchers only share samples with those they know they can trust (to not do something bad, stupid, or irresponsible with the sample)... with enough of those kinds of connections you form a web of trust, which is the network by which many researchers are able to get samples...

those are precautions that developed out of necessity, by the way, as there have been cases 'tagged' viruses escaping due to being shared with the wrong people...

"What Halvar Flake does to diff vulnerabilities out of patches seems 10x more sophisticated than the ability to flag a decryptor in an instruction stream based on the specific combination of instructions one guy in Bulgaria decided to use when he wrote the thing."

and if we could package him up with a product we'd all be much better off, but unfortunately av products are software, not wetware (yet)... see, it's not what halvar's tools do on their own unattended, it's what *HE* does with them... experts can do a lot more than finite state automata right now and there's very little we can do to change that in the short term...

"Some undergraduate in a dorm room at umich is going to totally revolutionize OS security someday, or figure out a way to reliably detect shellcode in network packets without buffering or reassembling. Nothing like that is ever going to happen in antivirus, and I think part of the reason is that AV people don’t want it to."

and this, once again, is an expression of mistrust that comes from not knowing enough... revolutionizing virus detection would give whichever company controlled the new technology a huge competitive advantage and after all is said and done money is what motivates the industry - if they can develop it or buy out whoever else might have done so they will... the industry wants that revolution to happen because it means lots of money for them...

Re: Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek — Fri, 25 Aug 2006 00:30:38 -0000

Kurt, apart from knowing the secret password to VCL and not knowing what the "op" in "opcode" is short for, what qualifies you to determine who is and is not part of the "antivirus industry"?

I'm sorry to be such a jerk. I just think the "AV research" community is a big silly clique. "Real AV researchers have ways of getting access to what the bad guys write WITHOUT writing new viruses" is just code for "you can't play in our sandbox unless you have a job with one of 10 approved companies". What Halvar Flake does to diff vulnerabilities out of patches seems 10x more sophisticated than the ability to flag a decryptor in an instruction stream based on the specific combination of instructions one guy in Bulgaria decided to use when he wrote the thing.

Some undergraduate in a dorm room at umich is going to totally revolutionize OS security someday, or figure out a way to reliably detect shellcode in network packets without buffering or reassembling. Nothing like that is ever going to happen in antivirus, and I think part of the reason is that AV people don't want it to.

Warmest regards.