I used to use the Matousec website to see which security products (which had behavioural monitoring) where good at detecting 'malware activities' through use of HIPS (Program/Behaviour Monitoring) by testing using viral techniques rather than signature recognition, there has been much debate as to why products like Norton, G-DATA, Avira, McAfee and BitDefender score really poor with 1-9% scores etc... so I decided to google it and see if I could find out more...
I come across
this page which explains it in full..
but, in a nutshell its flawed because each of the 10 (now 11) test 'sets' has 10 sub-tests, the product has to pass half or more of these 'sub-tests' to advance onto the next test set, if the product does not score 50% or more then it does not get tested in the next set of tests and is halted...but is scored against those who DID advance and score... anyone with any sense can see the flaw here!
for eg. product A could pass all tests and score 100%, but product B could pass just the 1st test-set, fail the 2nd and be scored 1-9% , but will still be charted against those who finished more/all tests...this is unfair as program B might have passed every single other test and been scored 90%+ ... but we never know because it was never tested...
you can only trust the scores on that website if the level of testing is 11 (meaning the product finished all tests) - so those at the top of the charts with level 11 testing can be trusted with those scores, but all below can not! you could say that each level is worth 9% max (100/11), so a product that reaches level 4 could score a max of 36% (4x9=36), if its score was 18% then you'd know it was 'averaging' 50% (50% of 36 = 18)...but this just gives you an 'average' score from tests its completed, products could perform extremely well or much better in remaining tests which would give it 'much more' than an 'average' score.
According to the BitDefender test report it passed 6/10 in first test and 4/10 in second (hence it failing to move further) - even with the average thats 50% but yet it scores just 9%..!!
Matousec does offer the test-kit for download so one could test their own product.
Its a shame really because I still strongly believe that the test suite itself is one that should be counted (when scored correctly), this is because other testing websites use signature based mostly and for the 'unknown' viruses they grab a few hundred or few thousand of malware that the product has not been updated to detect, problem is majority of these infections are likely to be a few 'groups' of the same 'type' of malware and therefore its really like detecting just a handful of malware/attack types, Matousec takes known attack techniques and puts it to the product to detect them, this type of testing (done correctly by putting the product through ALL tests regardless of scoring) would be an excellent way of determining which products can actually perform best
(alongside the results of other signature/performance/wild100 test results from the likes of AV-Comparatives, AV-Test and VB100).
Here is the Matousec scoring list as it stands:
