So, a progress update of things that have happened since my last update:

* Spent a week or two working on a bug the fuzzer picked out of an application, only to find that I'd screwed up the application install and got an old version of a dll, and that bug is actually patched. Drat. Still, at least it hones my reversing skills, and thus it's not wasted time... right?

* I'm running the AV'ing fuzz cases through the properly-installed and up-to-date version of the software now. A lot of issues are still there - hopefully I can find something cool.

* b0rk seems to be in good shape (here's hoping it doesn't fall down immediately after posting this). Had a mild panic yesterday when a 3rd party lib I use started AVing all over the place, but that seems to have hopefully gone away after an upgrade of it to the current version. The VMWare code is probably the worst part of b0rk at the moment, between my code and the open-source lib (not written by VMWare, I hasten to add). The fact that the stuff I'm doing is pretty non-standard and crazy doesn't help, either. I get the idea that if I had a massive amount of cash, VMWare could make it a lot easier.

* Me and Kat rewrote a load of the VM interface code I wrote last year, simplified it, and fixed a fair few bugs. This is possibly related to the above.

* The only major feature left to add now is parallelism. At the moment, only one VM server runs, but I want multiple VM servers with dynamically shifting loads, etc (again, if I had money, this would be a lot easier). I don't think this will be _too_ hard to implement, since b0rk has been designed from the ground up to support this. I'm thinking that'll take me 3 to 4 weeks.

* After that, it's just bugfixes and incremental updates, hopefully.

I haven't been posting daily 'status updates' like I was going to - I decided I didn't want that much 'noise' on this blog. Instead, I've decided to post once I finish a discrete piece of work on b0rk.

So. I've finished the last thing I was working on - writing more test cases for the x86 instruction handlers, and improving the existing ones. I also fixed the other tests within in a day or two, which I had timetabled to take until the end of this week - so things were running about two weeks ahead of schedule at that point. I then confirmed that I'd inadvertently fixed a bug (I love it when that happens) which I have scheduled for fixing sometime mid-next-week. Awesome.

So the end result is that I'm around a week ahead of schedule. Yay!

I spent some time experimenting with a faster debugging feature - injecting a thread in to the debuggee to process first-chance exceptions and decide if it's worth notifying the main b0rk process - but it turned out to be nontrivial so I shelved that for a while.

I just downloaded the trial of VS2010 and, geez, I'm impressed. intelliTrace is a wonderful, wonderful thing and let me squish two long-standing [1] bugs within an hour or two of using it. Loving it! I do doubt that I'll be able to afford to buy an upgrade from VS2008, though, at least for now. Money is expensive.

An currently poking some strange failures in one of my unit tests (which I thought I'd solved earlier) and running some very in-depth tests on the buildserver.

[1] Few weeks old, at the "bugged-but-fixing-it-looks-hard-so-I'll-make-time-for-it-later" stage.

Official word from MS - they're not going to put out a security patch for that local DoS I found, since it doesn't affect 2008r2 or win7. A quick summary of what's affected:

Confirmed vulnerable: Vista (Ultimate), 2008 r1
Confirmed safe: XP (pro), win7, 2008 r2, 2003.

Their justification is that, since it's DoS only and it doesn't affect win7 or 2008r2, it's unimportant. Apparently a patch "would be a candidate for inclusion in a future service pack update should one be released on the affected platforms".

Pretty annoying (and not _just_ because I wanted my name on an advisory ;)). I'm concerned that so many machines are potentially vulnerable, but MS aren't, and there's not an awful lot I could do about it without increasing the risk to paying MS customers.

I could post it to full-disclosure or such, but then we'd just have a load of script kiddies blowing up Vista boxes. On the other hand, it might somehow coerce MS in to patching it (and if I could express how simple a hole it is, you'd probably understand my concern that many people must've found it already). Ack. The thing is, I can't even give any workarounds for it - the best I can do is "do not run any executables which use a console window", which isn't even a 100% workaround, since the app could probably just spawn another command window.

I guess I understand MS's position too, here - I guess they are thinking "if we change that and push out a bugfix we need to do XXXXX hours of testing and we want to minimize the amount of changes we do in case we introduce bugs".

I'll probably just keep it secret, I guess.

I haven't updated since my post on wednesday about how I'd timetabled my time, and was going to do daily progress updates! Ooops! I've been crazy busy making up for some lost time.

On Thursday I got in to Thursday's task - adding support for all the one-byte conditional jumps. I already had support for the simple ones that branch on a single flag, like JZ, but I wanted to add support for 'complex' conditionals, unify them all, and put some rigourous testing in. I had intended to do this in one day, but about halfway through the day it became obvious that this task would require around twice that. In addition, near the end of the day, I discovered a lump of funcionality was missing, causing erraneous branching in the solver.

So I rejigged my schedule - I hadn't planned to work the weekend, but figured that I could use the weekend to 'catch up' - deciding that I would spend Friday working on this missing solver functionality, saturday finishing off the conditionals, and sunday doing the work I was going to do on friday.

This went more to plan. While adding funcionality to the solver classes, I somehow made the constraint solver which I use - CVC3 - crap out with an error, and ended up upgrading it from the prehistoric version I was using to current. Got that done in one day.

On the friday, I got the multi-flag stuff working, after adding another funciton to the solver, and I've just about finished Friday's task - which was to extend my tests for each instruction to exercise different forms of the modr/m (eg, reg+disp8/32) instead of the simple register forms. Next, I've got two weeks alotted to write more instruction handler tests.

I really suck at estimating how long it'll take me to write code! IDK how to improve on this skill, though. I guess it's just experience.

So, progress on b0rk:

* Fixed the dog-slow MSVC builds! Turns out I had disabled precompiled headers, which was a bit silly in a project the size of b0rk. Enabling them again made my build times reasonable again - down from around six minutes (!) to around a minute-and-a-half.

* In the process of fixing that, I turned on build time statistics on the CI server, and split the build process in to multiple projects. My build takes about 90 seconds and the tests which run on every checkin take about three and a half mintues. I'd like it to be faster, but I don't think that's likely without some work to the VM server to speed it up, hardware wise, which I can't afford at the moment. I'm happy with a five minute checkin-to-tested cycle.

* Splitting up the build process seems to have been a good idea. The project undergoes the following cycle now, with my unit tests being split in to 'fast' and 'deep' categoties:

Code checkin to SVN -> MSVC build -> MSVC 'fast' tests run -> GCC build -> GCC 'fast' tests run -> coverage data is collected

Midnight every day -> MSVC 'deep' tests run -> GCC 'deep' tests run -> coverage data is collected

These are chained such that a failing step will halt that cycle - so if MSVC can't build, it won't try to run the tests, and if the tests fail, GCC won't try to build. The code coverage stuff is done via GCOV.
I'm liking the mix of fast feedback and deeper assurance.

* Had a look at sourceMonitor for getting stats on my C++ project. It looks quite handy, giving me stats on cyclomatic compexity and other things. I think it might come in useful when I unleash my girlfriend on the code, since she wants to help out and clean it up a bit for me, and I think a nice summary of the top 'worst offenders' functions would be quite useful. I was tempted to integrate it with CruiseControl.net, but decided I don't really have the time to do so.

* Fixed a bug in the GCC build of b0rk. Oddly, vsnprintf() seems not to return the amount of characters written, but _vsnprintf does - strange! I'm guessing that one is the MSVC version of the function and one is the cygwin version. Anyway, that's fixed now.

* Wrote some more instruction tests and found another bug or two. Observed a bug which I'm fairly certain is down to another instruction handler (more tests needed there!)

* Since the project is in a much more predictable state than it was a month or two ago - ie, I'm finding less 'hidden surprises' in there - I made a list of things I need in the codebase before it is functional, and made time estimates for each. Some would say I should have done this two months ago, when I came back to the project, but I think my estimates are more accurate now that I've cleaned up the code a bit.
I've come up with an estimate of 30 days work to get it running, and an additional 11 days to get it working over the network (most of that is already done). This will take me to the end of december, ish, which is conveniently also the time that my money will run out. I've got a plan of what to do with each day I'm going to be working on it, so hopefully I can stick to that and identify any slippage early on, if it happens.
If we get to the end of December, and everything has slipped and the project is still in the state it is in now, then it'll be time to re-assess how practical the project is, bearing in mind how long it has been in development for, and evaluate what the next step will be.

* Tomorrow's task is work on fix conditional jumps, including 'multi-operand' conditional jumps like the JNA - Jump if Not Above - which is taken if either the Carry flag or the Zero flag is set. There's some minor rework to do in the Jcc logic, and some tests to write for it.

So, for the last couple of months, I've been doing daily 'progress updates' over on my 'personal' blog, mainly as a motivational tool. I think I should probably move them to this blog, and keep all the technical stuff in one place.

So, yesterday was spent on b0rk. I've been continually adding support for more x86 instructions as I've been working on b0rk, and a couple days ago I added the ADC [1] instruction, which turned out to be more work than I'd anticipated, as it needed me to rework the flag logic slightly, to support being set by three operands (src, dst, and C) as opposed to just two, as other instructions used. That ended up taking most of the day - including a frustrating moment where I realised that I'd hugely overcomplicated things and there was a much simpler way. Oh well.

I've got ADC going, now, though. I think I might get the SBB [2] instruction going today, since it won't be long until I need it. I'm tempted to spend some time working on the instruction handlers, since some of the older ones don't even have unit tests (!), and because I did a lot of work on it a few weeks ago and made half of the handlers a lot cleaner, but didn't get round to the other half.

I'm a tad concerned that my unit tests are a tad _too_ comprehensive - the ADC test takes 4 minutes to run, once all the solver runs have been cached (and over 5 hours for the first run, which populates the cache DB), for example. My plan is to use a 'comprehensive' test like that for each instruction type - add, sub, xor, etc - which could mean my test suite takes a really long time to run.


[1] ADd with Carry: dst = dst + src + Carry flag.
[2] SuBtract with Borrow: similarly, dst = dst - src - Carry flag.

OK, so over the last few days I've been running some distributed fuzzing. I scribbled up a fuzzer using WPF and .net, which was fairly straightforward, and have been running on the blades. I thought I'd post some numbers and results up here, just for the sake of interest - I assume anyone is interested?

Anyway. First the hardware specs, which are pretty low mainly because I'm a poor poor ex-student: the database and centralised server part of the fuzzer run on my desktop, an aging dual-core E2180, propped up by 8G of RAM and an 80GB Intel g2 SSD. This sent requests to one of 16 blade systems - P4 3.4G with HT and a gig or so of ram. All interconnects are gig ethernet.

It'd be nice to have more blades, but as I say, I'm poor :)

Software wise, the server uses MS SQL as a DB backend, and the blade servers run VMWare Server 2.0 [1] with a single instance of Windows XP (mainly for manageability reasons).

The first run has just finished, which involves taking a media file template of 28875 bytes and replacing each byte, in turn, with one of 16 values, resulting in 457694 rows [2]. The entire run took 2012155 seconds of CPU time, spread out among 16 nodes (so a good estimate for 'wall time' might be 2012155/16). 2012155 seconds comes to about 559 hours of time.

If these 457694 rows:

•  The average run time was around four seconds.
•  72,039 - about 15% - logged at least one exception, which is more than I was expecting.
•  17,957 rows logged a timeout of some form, accounting for 71560 seconds (approx. 20 hours) of CPU time.
•  385,495 - about 84% (!!) - hit one of my 'user interaction' breakpoints (eg, threw a messageBox, or a dialog box). Under these circumstances we end that execution immediately to save timing out. On average, these 385,495 runs had an average runtime of 5 seconds. Note that the timeout I used was around 15 seconds, so this equates to a massive time saving.

Interesting stats, I think.

On to the interesting stuff - the exception detail!

Of the 72,039 exceptions:

• 5 exceptions were code 0xC00000FD, stack overflow
• 760 exceptions are access violations of some form
• 514 AVs appear to be straight NULL references (all reads). These were all done from 4 locations.
• 246 AVs appear to be non-NULL references (244 reads, 2 writes). Reads were done from 2 distinct EIP locations, writes from 16.
• 71,274 exceptions were MS-style "exception code e06d7363"-style exceptions, which need more investigation.

That's kinda cool. I think the next step is for me to write code to see what's up with those e06d7363 exceptions - apparently they are used for c++ throw()'s, so there could well be something interesting in there. I do hope so. That shouldn't be hard to write (one of the things the scanner does is log stimuli as XML, so I can easily generate the files that caused problems.

I'm quite pleased with an average run time of 4 seconds, considering I'm doing proper end-to-end fuzzing, and not just fuzzing a small piece of code.

[1] Don't ask. :D
[2] 457694 / 16 = 28605, less than 28875, but the DB doesn't log entries which are skipped because they would not modify the file - for example, if the template file contains only bytes set to 0x00, then runs which would set bytes to 0x00 would be superfluous.

Welcome to my blog!

I post here about my technical ramblings, usually on the topic of computer security and other fun stuff. Comments are enabled, and trolling will result in a swift deletion, but other than that, I'm more than happy to answer any questions that people have on any of the entries.

An RSS feed is available here for your bookmarking pleasure.


This journal is exclusively for 'technical' posts. Anyone who comments in it discussing my personal life will be speedily deleted. Comments on my personal life should go in my personal journal.

Enjoy!

-Alan/Aliz/Randomdude/Gamesfairy (goddamnit where do I get so many names)

My good friend Rick - "tsky" - sent me another "geek supplies" package last week. As he's clearing out his place, he's wanting to find a home for a lot of his stuff, and being the local hoarder of broken kit, I usually get asked.

Anyway, there were some really cool things in this haul - in amongst the ISA SCSI cards and 3c509's, there was an openMoko phone (!!), a book on starting my own business, and a copy of 2600. w00t!

Also included were two large, stage-style RGB lights, powered by LEDs. Fantastic! That's just the push I need to get back in to lavalamp. Last time I played with lavalamp, I hooked up some lighting to iD's DooM, so my computer was illuminated accordingly to my in-game environment. Very atmospheric! With RGB lights like this, it'll be fantastic.

On further inspection, the lights have a DMX port, and inside, a suspiciously PIC-looking chip (with the marking scratched off). The way I see it, I've got three options now:

1) Remove the PIC, reverse the board (shouldn't be hard) and make a PIC to control the LEDs and talk the lavalamp protocol to a nearby PC;

2) Make a DMX controller for the PC from plans online (Microchip even have an app note about it!) and link DooM/etc to DMX instead of lavalamp;

3) Make a DMX controller and give it a lavalamp interface, thus bridging any DMX device to lavalamp.

I guess option 3 looks most fun and most useful, but I'm pretty time limited so IDK if I'll go with it. I'd like to bring DMX devices to lavalamp - effectively bolting on nice configurable comms layer (optional crypto, wireless to come..) and the lavalamp UI.

In other news, I've been attacking the Eternity II puzzle. Damn good fun.

I've also quit my part-time job as a .net developer for Concorde Informatics, mainly because it was getting in the way of what I really want to do with my career - infosec and magic. I'd forgotten how much webdev sucks (and for other reasons I don't want to go in to).

Hopefully now I'll have more time to update this blog, too.

So, I've decided to post 'progress reports' to manage b0rk a bit more effectively, since things are going to slowly now that I work elsewhere three days a week.

The last couple of weeks have mainly been filled with me banging my head against a brick wall regarding virtualisation systems being integrated with b0rk. It's still in progress, so hopefully I'll actually make some progress before I go to work next week.

In other news, I've bought a new motherboard for use in my VM server. I can't really afford a new machine at the moment, much less the server-grade hardware I'd need to run VMWare ESXi, my virtaualisation product of choice for general use. I checked the unofficial VMWare ESX whitebox HCL, and bought an "Asus P5Q PRO", since its ICH10 chipset should be well-supported by VMWare. Indeed, it is - I've had to drop a different NIC in the machine, but other than that, it works like a dream.

I've got an Intel G2 SSD in there, which all the nine VMs have a swap partition on. I'm not totally sure if this contention is going to cause severe preformance degredation, but we'll see. I've then got a 500G HD for the virtual machines, and a 1TB disk for the 'archive' of data I keep around.

I've got 8GB of DDR2 in the machine (since DDR2-800 is dirt cheap). I don't like the lack of ECC, but I can't afford a similarly-spec'ced server-grade board. It's got a dual core CPU in it - with a clear upgrade path to a quad core chip if/when I need (and can afford) it - and it takes 4GB DIMMS, so if I need to stuff more than 8G RAM in it, I can, too.

I need to set up backups of it, now, and some other network services I want to get running.

Busy life, eh?