gamesfairy

Tue, 27 Oct 2009 10:29:13 GMT

So. Another post about the security scanner.

I've spent (most of) the last two days playing, in some way or another, with memory breakpoints.

The scanner requires that it be able to set breakpoints on memory accesses. It needs to be able to say "Tell me when memory at XXXX is accessed" and run the target code otherwise unsupervised. This is a common feature in debuggers - so common that Intel added the feature in hardware.

So, the CPU itself is capable of performing memory breakpointing. However, it has a couple of downsides - firstly, we're limited to four memory breakpoints, and secondly, the CPU only tells us _after_ memory has been accessed, not before. So, for example, if we set a memory breakpoint at some location and the program writes to it, there's no way of getting the data back.

OK. So we can't use hardware breakpoints. But that's no real biggie - we can just use the software breakpoints that the debugger provides (the security scanner operates in parallel with a well-known debugger package). So I went about doing this, and after a lot of headscratching, realised that - bizarrely - the debugger package I use notifies me _after_ memory accesses too. Odd.
I figure, OK. Fair enough. I'll just write my own memory-breakpointing system. All I have to do is mark pages as guard pages, and catch the exception that gets thrown when that page is accessed. An exception will be thrown for every hit on the _page_, so I'll filter them by address. It'll be slow, but it's cool. No problem. So I implement that.

And it works. Oddly, though, the exception info I get passed by the debugger looks like random crap from the heap. No amount of playing about will get a sensible result. Unfortunately, I kinda need that information - one thing it contains is the memory address which was hit. I play with it for a bit more (the debugger I'm building around is a bit finnicky) and still can't coax anything out.

Eventually I find a line in a changelog of v5.2 of the software. "BUGFIX: SDK: dbg_exception event parameters were wrong". I'm using 4.8. The reason I'm using 4.8 is because there's a not-for-profit version of 4.8 available free. 5.2 costs around 300 GBP.

I eventually cave. I'll pirate a version of 5.2. So I do. And it doesn't co-operate with my plugin at all. Things just don't work. I realise that there's going to be a lot of time spent effectively porting it to the newer version of the debugger, before I even get to see if my problem is fixed.

I spend a lot of time trying to coax the debugger in to doing what I want to do. Perhaps it's time to ditch it for another?

It'd be an amount of work to adapt the scanner to use GDB instead, but GDB has a couple advantages:

* Free-as-in-money to use
* Dead easy to replace with scripts, in order to test the scanner's response to various debugger behaviours (a BIG THING)

I'm loathed to sink more dev time in to doing _more_ stuff, but I think this is the best way forward. Better tests would make me happy.

STP caching

Sun, 25 Oct 2009 00:27:16 GMT

Got the caching layer for the security scanner written last night.

As the scanner runs, it produces ASCII files which are sent to a field solver (CVC3 in this case). Obviously, there's a bit of a performance hit in producing these ASCII files, as opposed to just passing a data structure, but it makes caching and load balancing just so much easier that I decided to go with the ASCII approach.

Some of the queries are identical, especially when repeatedly running in 'verification mode'. Since I want to set up a continuous build system, and since it'd benefit performance anyway, it makes sense to cache the ASCII solver input and the ASCII solver output and respond to repeated queries.

This was fairly straightforward to implement. Since there's already a 'co-ordinator' which runs centrally, and which talks to all the instances of the actual debugger, it makes sense to throw the caching code in there. I decided to use a full-blown database to store my data, since I think that's faster (in terms of dev time) than re-inventing the binary search, and about the same speed (run time) in actually running. If anyone can prove me wrong, go ahead :D

Anyway. After a few runs of the same code - pretty much ideal conditions for the cache - I've ended up with ~4000 rows in the DB, and 2112 hits to 856 misses (71~ hits). Each hit takes, on average, 87ms, and each miss 226ms. I've made a net time saving of 137ms per hit, or 293569 - almost four minutes - over the whole run. Yay!

Security scanner progress

Fri, 23 Oct 2009 01:12:07 GMT

So, work churns on in the topic of the scanner.

I've just been doing a 'verification run', in which the program is run and the scanner observes each instruction, simulating it in terms of dataflow, and then compares results.

It's falling over at the moment, due to an inability to handle a particular instruction in 16bit mode (I'll fix that in a mo). But before it does so, it simulates 250 (non-unique) instructions mainly in the C runtime library, using a variety of addressing forms, and testing my modr/m and SIB decoding, and instruction coverage. (in addition to that 250, there are 35 occurances of around 6 unsupported-at-the-moment opcodes, mainly because certain CPU flags aren't implemented yet (ie, it's not worth writing the 'ADd with Carry' opcode until the carry bit works) or because they reference memory by segment (which isn't supported just yet).

I have, of the 256-ish[1] single-byte opcodes, 67 (26%) implemented.

The 'API call mocking' system is running. When an app calls, for example, Recv(), the call isn't executed, but trapped and symbolic data returned. Other calls which are difficult to verify are similarly modified (such as GetTickCount, which returns different data on each call).

Next big functionality:

* More instructions, until I can run all of my VS-compiled C++ test app (including most of the C runtime library)

* More API call mocks

* Database-driven caching for solver files (yay)

* Minor modifications to allow parallelisation (the app is architected to parallelise really well, but this was dropped a couple months ago to speed up dev)

* Add logic to actually find holes ("... if EIP can be equal to 0x41414141..")

* Extend variable-length buffer support with some ideas I had in the shower yesterday

* (maybe) speed up constraint solving with some other ideas I've been mulling over

* Fix memory leaks.

Still hoping to have the app finding holes (preferably generating exploits) in my example 'vuln' simple stack-based buffer overflow code by the new year. This'll depend, though, on how much time gets dedicated to the scanner over other things I have to juggle.

[1] there are 256 opcodes - 0x00 through 0xff - but not all are used, some won't be supported (MMX or SSE or whatever) and some opcodes are escapes to much more functionality.

Status of Lavalamp

Thu, 01 Oct 2009 04:22:04 GMT

So! You could be forgiven for thinking that the Lavalamp project is dead!

The website (along with two others I own) was unfortunately lost during the great VAServ hack, and up until recently, I have been simply too busy with University to redesign it. However, the time has come that I have some free time - so hey presto! A new site. As the more astute may be able to tell, web design is really Not My Thing, so please do feel free to point out any glaring errors or horrible face-burning design issues I've stumbled upon!

The project itself is coming on well. As outlined in my previous postings, there's a lot I want to do with Lavalamp, but precious few moments in which to do it. I am, however, planning (at the moment) to have some working code released within a couple of months (of October). However, since I've been promising a release since, ooh, about two years ago, you probably don't want to hold your breath!

I've stuck some screenshots in the 'user documentation' section that should show how things are running. The 'rule' mechanism is shaping up fairly well, and looks like it'll be quite powerful if I ever manage to finish it. Lets hope, eh!

Security scanner progress

Sat, 29 Aug 2009 08:23:29 GMT

Man. Coding this security scanner - in C++ - is really reminding me how much more enjoyable things are, the 'closer to the metal'. It's also, thankfully, giving me a great insight in to how the 'upper layers' work - mainly C#-type stuff. My C++ has always been terrible, although my C was usually fairly strong, so I had a lot of trouble previously with working out how c# is implemented by the compiler - but now, knowledge gained in C++ is helping to fill the gap.

Anyway. Progress on the security scanner.

I've finished implementing the 8bit memory layer, and have adapted a couple of the x86 instruction handlers to use it. I've got decent tests for the ones I've adapted - three instructions in total - which test thigns like the values received and how dependencies are mapped between the variables. Unfortunately, each is quite substantial, in terms of code, so it looks like it'll be prohibitive to write tests for every single instruction (especially since there's a gazillion of them in x86).

There's another 'verification mode', in which the scanner traces instructions with IDA and verifies that it gets the same answers, but that's pretty useless for dependency tracking. Perhaps I can use this mode, and investigate bugs that it flags using per-instruction tests.

The 8bit memory layer took longer to write than I'd hoped, coming in at arrounnnddd 4 or 5 days. While doing that, though, I've cleaned up a load of old code, and re-organised to make things more OO. I was also called to rewrite much of a solver function, so while I was at it, I implemented functionality that was on my waiting list, estimated to take a day or two, so that can't be bad.

The functionality I talk about is variable re-labelling - when I write a file to send to the solver, I label each variable in order of appearance in this file, instead of with its per-run 'name'. This'll mean that I can cache things a lot more easily and effectively (and yes, caching is still on my to-do list).

The next steps are to implement emulation of lots more instructions, make a few tweaks here and there, finish a bit of functionality, and it should get to a usable state. That shouldn't, hopefully, take more than a month (barring huge gaps in functionality which I've forgotten about).

Here's hoping, eh? :D

Gutting the 32bit security scanner memory model and replacing with 8bit

Thu, 27 Aug 2009 11:37:44 GMT

I've mainly been working on the security scanner over the past couple of weeks, when I've been able to find time. I've been very busy moving house, and doing other IRL stuff.

Anyway.

I think I mentioned earlier, that the security scanner models memory as a set of discrete 32-bit locations, and how much headache it's caused me. I've mostly been 'converting' to an entirely 8-bit memory model, and using 32-bit operations purely as constructs to model the interaction of various memory locations. For example:

4 bytes of memory at 0x00 thorough 0x04 are added to memory at 0x10-0x14, using a 32bit operation.

The scanner would create 8 entries in the 'memory to variable' map:

memory at 0x00 is represented by variable A
memory at 0x01 is represented by variable B
memory at 0x02 is represented by variable C
memory at 0x03 is represented by variable D
memory at 0x10 is represented by variable E
memory at 0x11 is represented by variable F
memory at 0x12 is represented by variable G
memory at 0x13 is represented by variable H

and would then create a 32-bit 'helper' value X. Next, the scanner marks that A through D are dependent on X, so you have:

AB CD
| | | |
VV_VV___
| X (32bit )|

sorry about the ASCII art!

Similar is done for a new var, Y, and E through H:

E F G H
| | | |
V_V_V_V__
| Y (32bit )|

The scanner then defines that Z is equal to the result of a 32bit add, X+Y.

Z = X + Y

To store computer data in memory, we can't simply set EFGH - what if something else depends on them (ie, before this add, what if a different operation used EFGH)? We make new 8bit variables, IJKL, and assign them to Z. We then tell the system that memory at 0x10 through 0x14 correlates to IJKL.

The final outcome of the system:

AB CD
| | | |
VV_VV___
| X (32bit )|

EF GH
| | | |
VV_VV___
| Y (32bit )|

Z = X + Y

I J K L
| | | |
VV_VV___
| Z (32bit )|

memory at 0x00 is represented by variable A
memory at 0x01 is represented by variable B
memory at 0x02 is represented by variable C
memory at 0x03 is represented by variable D
memory at 0x10 is represented by variable I
memory at 0x11 is represented by variable J
memory at 0x12 is represented by variable K
memory at 0x13 is represented by variable L

This is then serialised in to an input file for the STP solver I use - cvc3 - which comes out like this (as an example, we say that ABCD is hardcoded to 11223344 and CDEF is hardcoded to 55667788):

(snip uninteresting bits)
ASSERT( X = A:B:C:D );
ASSERT( Y = E:F:G:H );
ASSERT( Z = X + Y );
ASSERT( A = 0x11 );
ASSERT( B = 0x22 );
ASSERT( C = 0x33 );
ASSERT( D = 0x44 );
ASSERT( E = 0x55 );
ASSERT( F = 0x66 );
ASSERT( G = 0x77 );
ASSERT( H = 0x88 );
TRANSFORM(Z) ;

Obviously, I've edited that code to make it clearer. It denotes a series of 'simultaneous' assertions - the solver will find an answer which satisfies all the assertations. The final line - TRANSFORM- asks the solver to tell the user a value of Z which does this. The only value it can come out with, for this particular STP file, is 0x6688AACC - which is 0x11223344 + 0x55667788.

Notice that 'memory' has been totally abstracted away, here, in favor of 'variables'. Modelling memory is bad, mmmkay. We're running at so low a level - the machine code level - that heap corruption will be modeled just as if it were a legal situation.

This example shows how easy it is to then add other nnon-32-bit constraints. You could then add

ASSERT( F = N OR P );
ASSERT( N = 0x60 );
ASSERT( P = 0x06 );

and get the same result.

The current codebase differentiates between a register - reg_32_t - and four bytes of memory. This is bad, because we can't treat the two identically. Especially in x86.
x86 has a concept called the 'modr/m byte', which is a byte tacked on to most instructions specifying what to operate on. It can specify a register (well, strictly speaking, two, actually, target and source) or a memory location (or a deref'fed pointer, but that's almost the same thing, as far as we're concerned). Having a function to decode this byte was dead easy when memory was modelled as a series of registers - it just returned a 'register' stru- er, object - and the code did with it whatever it liked.

It looks like, to return to this nice neat state of affairs, I'm going to model registers as groups of 8bit discrete registers. I'm a bit worried this is going to be dog-slow, but it's the neatest (only?) way I can think of doing things.

I really miss c#.. this project is reminding me how inexperienced I am at C++, and OO in general. If I was to restart this project from scratch, one thing I'd do would be to OO-erise it a lot more.

Apologies for the hard-to-read nature of this post. It was a tossup between ascii art and just scanning my 'working out' notepad.. And its midday and I'm tired, having been coding all night. Comment, or ask me via IM, if there's anything I should say more about.

Wiring SMART drive health status to SNMPD

Wed, 26 Aug 2009 06:45:22 GMT

* Been playing some more with Opsview. First step was getting its 'agent' program running on Jaunty. I did a few things to get it going:

1) I just downloaded the latest version (https://downloads.opsera.com/opsview/yum/3.1/centos/5/i386/opsview-agent-3.1.0.2553-1.ct5.i386.rpm)

2) Turned it in to a dpkg it with alien, and installed it
alien -d opsview-agent-3.1.0.2553-1.ct5.i386.rpm
dpkg -i opsview-agent-3.1.0.2553-1.ct5.i386.rpm

3) Added the 'nagios' user, chgrp'ed the nagios dirctory, and symlinked from the openssl libs to libssl.so.6, which is (another) dirty hack, but peh. :D
useradd nagios
chgrp -R nagios /usr/local/nagios/
ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.6
ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.6

Kapow! It worked (for me).

* Need to install NTP to sych times.

* After this I installed snmpd on the machines I'm monitoring, and scribbled up some simple scripts to monitor SMART information. First off, I installed snmpd and got that working, then installed smartmonctl, and checked that I could get data from all my drives. Success! I went ahead and wrote a dead simple one-line script to get drive info, and parse it (improvements are welcome, bash scripting is not my forté!):

root@anna:/home/smb-aliz# cat /usr/bin/smart-drivetemp
#!/bin/bash
exit `smartctl $1 -a | grep Temperature_Celsius | cut -b 85-90`
root@anna:/home/smb-aliz# smart-drivetemp /dev/sda
root@anna:/home/smb-aliz# echo $?
42

Ace. So now we've got a script that accepts a drive name, and returns the drives temperature. Now to wire it up to snmpd. The only problem I ran in to here is that the SMART tool needs to be run as root, for obvious reasons. snmpd runs as the snmp user, so I decided to make smartctl suid root (chmod u+s /usr/sbin/smartctl). I could've made the script suid, but I'm not quite that silly. Obviously, it's a bad idea to make anything suid root, and an even worse idea to make a tool which can take your drives offline suid root. Don't do this on important machines which evil hax0rs could get an account on, kids. If there's a better way than setting smartctl to be SUID root, please let me know..

So. After doing this, we tell snmpd to run it:

echo exec smart-sda "/bin/sh /usr/bin/smart-drivetemp /dev/sda" >> /etc/snmpd.conf

And we check it's working.

root@anna:/home/smb-aliz# snmpwalk -v 1 -c public localhost .1.3.6.1.4.1.2021.8
UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extNames.1 = STRING: smart-sda
UCD-SNMP-MIB::extCommand.1 = STRING: /bin/sh /usr/bin/smart-drivetemp /dev/sda
UCD-SNMP-MIB::extResult.1 = INTEGER: 42
UCD-SNMP-MIB::extOutput.1 = STRING:
UCD-SNMP-MIB::extErrFix.1 = INTEGER: noError(0)
UCD-SNMP-MIB::extErrFixCmd.1 = STRING:
root@anna:/home/smb-aliz#

Awesome. Now opsview can give me nice graphs of my drive temperatures! And can SMS me if they get too hot! Of course, the possibilites are endless. It's a good idea to also wire up alerts on other SMART properties - Seek_Error_Rate and Raw_Read_Error_Rate spring to mind - and it's an even better idea to wire an alert up to the general SMART health status (smartctl -H/dev/sda). Happy monitoring!

Opsview on ubuntu jaunty jackalope, and libltdl3

Sun, 23 Aug 2009 19:43:30 GMT

matthewrwright recently recommended Opsview to me as a network monitoring tool. It's built on Nagios but is apparently a lot easier to configure.

Unfortunately, though, packages are only provided for debian Etch, Lenny, and Hardy. this post seems to indicate that there are no plans to make a Ubuntu Jaunty package available (although the packages seem to work with previous versions of Ubuntu). Nevertheless, I've gotten opsview working on my Jaunty box, so here's the way I did it:

0.5) Understand that this is completely without warranty, and may break your apt configuration, your machine, your network, etc. I did this on a virgin jaunty box - I make no statements that it'll work for you. Don't forget to backup first!

1) Install the 'equivs' package.
apt-get install equivs

2) Create a dummy package, purporting to provide libltdl3.
equivs-control libltdl3
Edit the created libltdl3 file, changing 'package' and 'proves' lines to contain 'libltdl3', like this:
Package: libltdl3
Provides: libltdl3
Now, create the package:
equivs-build libltd3

3) Install the dummy package:
dpkg --install libltdl3_1.0_all.deb

4) This step is a bit of a hack, but works so far (again, don't blame me if it breaks stuff..)
ln -s /usr/lib/libltdl.so.7 /usr/lib/libltdl.so.3

4) Add opsview repo to sources.list, and install it.
apt-get install opsview

If you don't feel like doing steps 1 through 3, you can simply download the deb package I made and install it. You'll still need to symlink libltdl, though.

Telecom musings

Mon, 17 Aug 2009 23:56:49 GMT

Since last time, on channel gamesfairy:

* Talked to a couple of friends - one of whom works for BT - about telecoms. Really reminiscent about all the stufff I used to do with telecom (well sort of).

* Saw the slashdot article about the dude that ran his own telco carrier. Interested.

* Remembered the recent iphone stuff, apparently obtianed by fuzzing the iphone through some 'custom' stuff. IDK how he did it - I'd probably have emulated it, if it wasn't too much trouble? IDK.

* I'd really like to play with this. Run a femtocell, hooked up to one of my phones (and with too little power to get out of my house) and send the phone bizarre shit until it explodes. I should add that to the to-do list.

* I wonder how hackable the commercially available femtocells are. Obviously, the best idea would be to use something like the USRP2 and GNU radio (software radio that does 'DC to 5ghz'), which would all fit together nciely and be awesome, but the USRP2 is far too much for me ($1400US + transciever boards). I really don't want to start fuzzing the telco system live, in case I break it.. maybe I'll have An Idea about this stuff later on. Perhaps I could remove the phone's PHY and hook it to an FPGA and then to the PC.

* Might be easier to just emulate the phone - particularly with stuff like the iPhone where a lot of the hardware is well-observed. If I can emulate it (or probe it live, ICE style) I should be able to hook it to the l33t security scanner I'm (still) writing.

* Need to rewrite a chunk of the scanner. I (rather short-sightedly) modelled memory as a set of 32bit integers, under belief that I could just mask in 16- and 8-bit memory accesses. I kinda could, but I get in to big trouble when I look at 32bit operations which span two 32bit values in memory - (eg: I'll have two memory locations, 0x00-0x03 and 0x04-0x07, and the target will request 0x01-0x04). Going to rewrite the memory access layer as dealin gin 8bit values, and tweak it a bit so it becomes easier to write tests for the instructions, too.

* Really on a telecom bender at the mo. I bet there's cash in hax0ring telco kit, too.. IDK how I'd go about selling stuff I found (to the telco, obv, my hat isn't black enough to sell natino-cripping bugs to The Terrorists). I guess if nothing else it'd earn me skills and rep in the Industry, both of which I seem to be lacking.

* Still pondering post-grad career options, if my current job plan doesn't work out. I think postgrad education might be a good bet. I'd like to go 'freelance pentesting' but realistically, I've got no chance in the market (not enough people skills, tech skills, etc). Maybe in a few years.

* Been working on lavalamp. Looks like my plan to scale it back and get a functinal release out should go ahead.

Lavalamp source found! Woo!

Mon, 10 Aug 2009 08:39:16 GMT

So, I found a copy of the lavalamp code yesterday. Previously I thought I'd lost almost all of the code in the Great VAServ Meltdown, but evidently not. I've lost all the repo info, but I've got the most current working copy, so that's enough for me.

All I have to do now is find enough time to work on it! It's really competing for time with other projects - ie, the securityscanner - and various real-world stuff I've got on at the moment (ie, moving house). Having said that, I'd really like to get something running with it, even if I only make it work enough for my own use and rely on the open-source patching process.

The real bottleneck in the development, at last count, was the 'native' home-grown RF link. It soaks up a lot of dev time and frequently leaves me confused. Most of my problems could probably be circumvented if I were to switch to an RF chip that does much more work for me - but I'm really keen on the 'all parts available from (highstreet shop) Maplin' idea, and all they have available is the dumb RF modules.
One way forward is to move to Zigbee - since transcievers are now cheap enough - although using that requires a fairly beefy PIC chip. The dev time required to move to Zigbee is also not inconsiderable.

I think the best way forward is to do is the following (when I get time):

1) Implement modularised 'network' layers, allowing nodes to run over different, uh, network transports.

2) Implement a straightforward and dead-simple daisy-chained RS232-over-cat5 network, allowing nodes to be hooked up via cabling. This'll let me get "something working", and get some real-world use out of the thing. Being cabled, this network wouldn't even need any encryption.

3) Finish enough of the lavalamp PC-side software for it to be useful (ie, enough that it is solid, can be scripted, and can be used interactively in a GUI).

4) Do a release under an open-source license, inviting people to use the code, give feedback, and contribute.

5) Start work on all the 'extra' features:

* Zigbee interfacing, as a different network layer (allowing people to use Zigbee if they want the extra power requirements/space requirements/complexity)

* Linux support (probably via Mono). This would be really nice as I could run the server on a tiny ARM-based machine instead of a full PC,

* More exotic peripheral drivers (LCDs, VGA, GPS, you name it),

* More exotic transport layers (homebrew RF, perhaps, and maybe even X10)

* Buzzwordy ajaxy control app, WAP access, all that jazz)

I also need to rewrite the lavalamp.gamesfairy.co.uk website, too, at some point. If only I had more hours in the day!

Ack, HD failures!

Thu, 06 Aug 2009 23:22:13 GMT

Well, I guess this confirms another hardware failure!

root@anna:/mnt/space# dd if=/dev/urandom of=randomdata bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 661.065 s, 1.6 MB/s
root@anna:/mnt/space# md5sum randomdata
62a72439f880ba0a032086e1b12155e0 randomdata
root@anna:/mnt/space# cp randomdata morerandomdata
root@anna:/mnt/space# cp randomdata /mnt/backup/randomdata
root@anna:/mnt/space# cp randomdata /mnt/backup/morerandomdata
root@anna:/mnt/space# md5sum randomdata morerandomdata /mnt/backup/randomdata /mnt/backup/morerandomdata
0f611ad252be943e75e2fcd9e9483103 randomdata
87fc39b9e745f46d54b5a037088dce8e morerandomdata
ad572e39cb577c5d79116ed909f11f89 /mnt/backup/randomdata
ad5093cffcd43caf00ccb55be3e2dfe5 /mnt/backup/morerandomdata

Ouch! Bizarrely, nothing bad appears in dmesg, and if I repeat the above using /dev/zero, I get no errors. Since all this happened when I transplated the drives in to the Intergraph ZX10, which also bitched at me about PCI resource conflicts, I suspect some weird oddness between the PCI SATA controller and the CPU.

This machine has two volumes, /mnt/space and /mnt/backup, which are both on the PCI SATA card. Normally I back up from one to the other every night, which protects against those 'oh shit I just rm'ed the wrong disk' situations that RAID 1 wouldn't. Unfortunately, putting them in one machine was fairly short-sighted, as they have both been rendered useless. Fortunately, I have an additional off-site backup at my mum's place, which should be intact (if a week or two old).

The machine I keep the backup on at mum's house, however, also suffered at least one harddisk failure, a couple of days ago. I appear to have lucked out, I hope. The machine contains some five hard drives, four of which are in a linear RAID array (the idea going that availability isn't crucial on a backup machine) and one is the OS drive. None are redundant, and the OS drive has failed - so I'm really, really hoping that the remaining four drives are functional enough to have the data read from them.

Frustratingly, I was in the process of moving the two volumes at my place in to two separate machines, but decided to do the initial backing-up locally on the Intergraph ZX10, for speed. Damn.

Moral of the story:

* Backups should be on different machines. I already knew this to some extent, fearing that an Evil Haxx0r or a PSU blowout could take out two copies of the data, but neglected the possibility of a controller failure (or a controller driver failure, or whatever happened). I've always been reluctant to use hardware RAID in applications like this, for fear of a controller 'dying', and the RAID array it had being inaccessible without an identical controller.

* Some script to detect large amounts of file changes and alert me would be good. I only noticed the corruption after I attempted to play an mp3 and didn't get very far.

* Had this occurred in a server I wasn't in the middle of setting up, I would've been emailed the ext3 errors. Damn. As it was, the Intergraph didn't have logwatch set up correctly at this point.

EDIT : A reboot and a BIOS upgrade later and it's stable. I don't know how long for, though, so I'm going to stress-test it with some heavy parallel file IO and see what happens.

Crowd-sourced SAT solving

Thu, 30 Jul 2009 11:40:26 GMT

Yes, really, those crazy web2.0 kids are attempting to crowdsource SAT solving. Check it out. http://funsat.eecs.umich.edu/.

Proper post later.

Cruisecontrol.net and other networky stuff

Sun, 19 Jul 2009 17:41:03 GMT

Things done since last sleep:

* Set up Linux VM as a router, meaning I don't have to have the main windows box directly on the public internet (!!). Have set up DHCP3/BIND9/squid/ipchains/logwatch/arpwatch/fwlogwatch, so I get transparent web proxying (yay for saving bandwidth) in a nicely secure environment. Hopefully. Here's hoping no-one pwns it.

* Finally gotten CruiseControl.net, a CI system, running! Now, when I checkin some of the security scanner to subversion, it'll pull it back out, do a build and run my 'tests' for me, letting me know if I've inadvertently broken stuff. TBH the security scanner is pretty non-agile, so it's a bit kludgey in places.. "#define VERIFYMODE" style.

* Spec'ced a 'cheap VM server' using consumer-end components. E7600 CPU (with VT externsions), 8GB RAM (board takes up to 16), motherboard with enough PCI-E to take a nice raidcard if I afford one later, and enough SATA to get hefty bandwidth. Came to a shade over £300. Haven't quite convinced myself I can afford it yet, but aim to soon.

Copy-on-write in virtual machines

Wed, 15 Jul 2009 04:57:07 GMT

So. Todays post centers around the use of copy-on-write. If you don't know what CoW is, check out this gPXE page before proceeding; things will make more sense then.

^^ Copy-on-Write. CoW. Geddit?

So. Since my security scanner revolves around examining each code path in a given program individually, as I've discussed previously, I'm in need of some method of 'cloning' the process and everything it has, similar to the unix fork() command. I'm still not 100% sure on exactly what is cloned in a unix fork(), but I _am_ 100% sure that it won't be much use in a Windows-based environment, or when I'm cloning instances of an OS.

The Plan involved using VMWare to do this, taking snapshots at each code branch and duplicating them. This would create a CoW disk image for each branch snapshot - ie, the cost of splitting would be approximately nil for the actual disk image - and would dump current 'physical' RAM of the VM in a flat file. The obvious drawback is that it'd require a decent amount of disk traffic as memory is dumped/restored/etc.

Unfortunately, though, VMware doesn't support cloning snapshots of a 'live' machine. Bummer. GSX server will support something called 'templating', which should, but it isn't supported by the .net wrapper I'm using for VMWare comms, and it's unclear if templating will use CoW for the disk images (or indeed, for system RAM).

The obvious solution is to run VMs from a syste that supports a CoW disk filesystem - like the following:

Windows VM box -> SAMBA -> Linux file server -> ext3CoW

Then, I could suspend a VM, causing RAM to be dumped to disk, copy the files, causing the linux box to mark them as empty CoW 'shadows', and then tell the VM box to load the new files as a new VM. It'd be dead simple, but instinctively, I really don't like this - there's a million things to go wrong (what if VMWare doesn't flush stuff to the SAMBA share before I copy? What if SAMBA doesnt flush it to the FS?) and that's before you get to the possibility of requiring the VM-controlling app executing scripts on the fileserver to manipulate CoW.

Bit of a bummer, really.

It's a shame I'm not some mad l33t code ninja, otherwise I'd grab the source to QEmu, or the like, and hack some 'CoW memory' code in. That way, when I wanted my code flow to branch, I could just tell QEmu to freeze the current VM and make two CoW shadow copies of RAM and disk. runtime cost - almost nil.

constraint solver magic

Sun, 12 Jul 2009 20:57:35 GMT

Just reading a paper I stumbled across:

Abstract
The automatic patch-based exploit generation problem is: given a program P and a patched version of the program P', automatically generate an exploit for the potentially unknown vulnerability present in P but fixed in P'.

Interesting stuff - they're talking about using STP (or any other constraint solver) to generate 'exploits' given a patch for 0day (link). It's interesting stuff, and really close to the STP stuff I was postulating around about this time last year.

STP is used as the driving force behind my l33t security scanner, hopefully. Interestingly, the STP homepage notes that STP is used by 'various government agencies'.. Intreguing.

Definite food for thought.

Experiments with the IOS

Sun, 12 Jul 2009 20:55:06 GMT

Been reading a lot of stuff online about hax0ring. Came across (the) Silvio's wordpress blog, which I found a pretty good read - lots of low-level bugfindy stuff. I read an entry about searching for double-free bugs in the Cisco IOS via an emulator and it really got me thinking about stuff - since malloc() is on the CPU (as opposed to any custom hardware) it'd be pretty simple to plop an ICE - hardware debugger/emuator - on a real-life Cisco box and track calls to it (and free()) to look about for heap corruption. Silvio didn't have much success finding double-free's, and TBH exploiting something like souble-free on IOS is way above my skill level - but I'm reminded of an old vuln in which you could flood a box with CDP updates and it'd just consume all available RAM and die, and wonder if I should hook up some 'network-based fuzzing of an IRL router with an ICE hooked up to log malloc/free calls'.

* Had a really fucking awesome idea while I was in the shower last night, around the topic of the security scanner. It sounds batshit insane, so I'm not totally sure it'll work yet, and thus reluctant to post about it (and I don't want naughty hax0rs stealing my ideas).

* Work on the security scanner is fairly solid. A few people I could find online have implemented scanners around the same kind of idea, but they're mostly 'mad ninjas'. I'm hoping that, once I've got the scanner up, I can tweak and hax0r it to reveal classes of vuln that other people haven't (and use my batshit ninja idea from the above point)

Cisco box probing

Sun, 12 Jul 2009 20:54:10 GMT

Popped open a Cisco 2500-series router yesterday and attached loads of probes to the memory DIMM, in the hope that I could turn CPU caching off and use it to observe malloc() and free() calls natively. Ignoring the 'oh crap' moment where I miswired one of the GND pins to VCC (and melted one of my test clips), I suddenly realised that it's pretty much impossible to pick out genuine DRAM access from the DRAM refresh cycles (Which I'd somehow forgotten about). It'd be a better idea, I think, to probe directly at the host CPU, but it's a QFP (CQFP to be exact) and the pins are too closely spaced for my test probes. I don't have a test clip for QFP-132, and they are fairly expensive, so perhaps I should trace out the board it's on and attach probes to test points or even the DRAM controller, if it's not also QFP. Does anyone know much about the M68K (68030 to be exact?) AFAICT there's no hardware debug support (am I right?) unless you get an ICE, and msot people debug via a GDB target (trivia for today: the production IOS has a minimal GDB stub!)

Work on the security scanner is slow but steady. A lot of time is taken up by me implementing what basically amounts to an x86 emulator on x86. Still can't work out why my 'ninja idea' won't work. I might try it later, but it's the kind of thing that'll only really be relevant once the project is at a much later point than it is now.

Security scanner minddump

Sun, 12 Jul 2009 20:51:15 GMT

Am sleepy but incredibly good. This is quite possibly the best way to be.

So, I promised some coherent explanation of the key parts of the security scanner last night! I've got this, which I've just scribbled now. It's not too clear, so don't be afraid to comment asking me for vital details I've forgotten about.

The heart of the scanner lies in constraint solving. Essentially, you take a given set of instructions and track how they affect memory/registers. For example, check out the following sequence:

1 : CALL someApiFunctionThatGetsDataToEAX
2 : PUSH EAX
3 : RET

From this we infer the following:

1 : At this point, EAX is 'anything'.
2 : At this point, memory location pointed to by ESP is 'anything'.
3 : At this point, EIP is 'anything'.

From 10,000 feet, that's all there is to it. You track what's going on, and examine for certain desirable conditions (eg, EIP is user-controlled).

Of course, there's a lot more to it than this. You'll notice my caveat 'at this point' in the above sequence. I'm not going to dwell of the simple stuff like that; I'm just going to dive straight in to the non-obvious bits.

I use an external constraint solver, CVC3, to do the 'maths magic' for me. It can tell me if things are 'satisfiable', for example:

a = 13 ; b = 12 is satisfiable, but
a = 13 ; a = 12 is not, as 'a' cannot be both 12 and 13.

It can also simplify equations down - eg, "A XOR B XOR B XOR 10" becomes "A XOR 10".

Constraint solving is magic, and I borrow from work done at Standord by a load of dudes (Cristian Cadar , Vijay Ganesh , Peter M. Pawlowski , David L. Dill , Dawson R. Engler), published in the paper 'EXE: Automatically generating inputs of death' and a couple of other papers I've linked on this here blog. The EXE paper is recommended if you're interested in this stuff.

Assumptions

Since it is dog-slow to examine each instruction, we make some assumptions.

1) The program, OS, and everything, is totally deterministic (see later on about malloc).

2) We can disregard all instructions which are not affected by user input (for example, that obtained via Recv()). Because of this, we can simply breakpoint functions that pull in user input, and set our program executing. We don't care what it does, as long as it isn't in response to input from an untrusted user. When there is no untrusted data in memory or registers, then there is no point examining instructions - so we can just execute our process, and wait for it to call Recv(), or whatever. At that point we resume examination. This is sort-of-similar to EXE, I think? EXE approaches the problem differently, by instrumenting the target binary.

Conditional jumps

Conditional jumps which depend on user-input data cannot be resolved without applying artificial constraints to the user-supplied data. Consider:

1 : CALL someApiFunctionThatGetsDataToEAX
2 : JZ foo
3 : xor EAX, 0x20
4 : jmp end
foo:
5 : xor EAX, 0x10
end:

How do we tackle this? Is EAX equal to 'the input byte XOR 0x10' or 'the input byte XOR 0x20' (or something else)?
The original EXE paper runs under unix, and creates two copies of itself, and defines one copy to have Z=1 and one copy to have Z=0. Two paths are examined:

1 : CALL someApiFunctionThatGetsDataToEAX
2 : JZ foo (starts processes)

(process 1)
(implicit inference: EAX != 0)
3 : xor EAX, 0x20
4 : jmp end

(process 2)
(implicit inference: EAX == 0)
5 : xor EAX, 0x10
end:

EXE observes that EAX can either be 'unconstrainted XOR 0x20' or '0 xor 10'. Awesome.

This isn't really practical on a win32 system, though. You end up giving the target process room for change in between runs, breaking the determinsm. Obvious stuff like the PID changes, but more subtly all malloc()'ed address will be different and could even conceivably be in a totally different order - so the constraints we applied to constrained memory will point to completely different locations!) This is No Good.
We could do some Valgrind-esque tracking of memory blocks, but we end up slowing everything else down and generally making everything suck.

The way I'm going to attack this is to have a full VM for each run. Every time a conditional jump is reached, two VMs will be used to examine each trace. This will greatly help (although not totally solve) issues of determinism and repeatability. Obviously, it'll be slow to start up each VM shapshot (and we obviously want to snapshot, and not boot fresh every time), but it conveniently helps with another problem - scalability. I can simply dedicate a VMWare cloud to the thing, write some network-aware code, and it'll cluster really well. Adding and removing VMs could conceivably be done even mid-run, or via a WAN link, or even in rented CPU time on some supercomputer somewhere. VMWare, I think, solves enough issues that the startup performance hit is worth it.

Jump tables

Consider the following C:

switch(someApiFunctionThatReturnsData)
{
case 1:
..
case 2:
..
}

and the generated asm (not actual legal x86 asm, but you get the idea)

1 : CALL someApiFunctionThatGetsDataToEAX
2 : AND EAX, 0x000000FF
3 : MUL EAX, 0x10
4 : ADD EAX, EIP
5 : JMP EAX
...
16 : {case 1 asm}
...
27 : {case 2 asm}
...

(this asm is horribly contrived and buggy, but it gets my point across!)

Now. How on earth can we simulate this? The scanner simply knows:

* EIP = (((input data&0xff)*10)+previous_eip
* previous_eip = 5

How on earth do we find out where the code will branch to in a non-highlevel-dependent, arch-independent way?

.. I think I've solved this, but I'm not going to tell. I'm holding on to this one! If you want to work it out and comment, or ask me, I'll tell you if it's how I plan to do it!

Other stuff

I've got a whiteboard full of stuff to implement and problems to solve. I'm really looking forward to it.

Sun, 12 Jul 2009 20:50:05 GMT

So yeah. Am forging onward with the security scanner. I worked out a way of adding a 'self-test' mode to it the other day, which tests (to some extent) the simulation of each instruction. I got that working today, and found some really, really obvious bugs (ie "how on earth can PUSH work if you have no support for subtraction of pointers?!") which I was able to zone in on fairly quickly.

The only problem is that it's damn slow. I could optimise it by only verifying effects to a subset of registers (ie, I could chop off the FPU regs right away, since I don't support the FPU at all) but I still end up running the solver many times per instruction. At the moment I shell out for each solver run, incurring the huge createProcess delay, which was a design decision mainly so I can bolt on some load-balancing stuff really easily. It looks like it wasn't the best decision, though, and I might move to in-process solving instead. Not totally sure on this one, and I won't change anything yet - first, I want to have planned a fairly coherent caching/load balancing system that I can just slot it in to.

Speaking of which, yeah. Thinking of doing some mysql+php stuff to make a diddly little caching server.. not sure, though, since I absolutely suck at SQL and don't know much php beyond the phpinfo(). .Net and c# is my usual for webdevvy stuff, but I think that'd be pretty overkill for what I need it for.

Oh, I also realised that over the past few days, I've accidentally implimented 'lazy flags' (ie, you don't work out the flags, such as ZERO, until you find an instruction that needs them, like JNZ). This is nice. I like it when stuff just slots together like that.

Code is awesome.

CBC weaknesses

Sun, 12 Jul 2009 20:49:24 GMT

I can't remember if I posted about this before, but I found this article at matasano on crypto really interesting (if a bitch to read). It's all about carefully breaking CBC-mode crypto by fucking about with selectively corrupting blocks.

Best quote: Turns out, I can make the cookie say whatever I want. It’s a property of CBC.

It's a really good read, but if you're anything like me, you'll need to read it three times then have a good think about it before you get it. Crypto really fucks with my head.

C-style varargs - WTF LOL

Sun, 12 Jul 2009 20:43:09 GMT

So. C-style varargs.

All C coders know them. They're the magic that lets you "printf(" %s is cool", name);". Variable-length argument strings. Anyone know how they're implemented? I didn't until I read up on it. I guessed - and happened to be right - but the full horror of the details didn't hit me until later.

So. You have your function.

tellTheUser(char* fmt, ...) {}

You use some magic compiler macros to pull stuff off your vararg list, and you call it such.

tellTheUser("%d is my fave number", 3);

The compiler will set up a two-argument call. Awesome. But think about this - to do this, it needs to know, at compile-time, how many arguments are in the list. So you can't call your function with a variable number of arguments, strictly speaking - the number of arguments must be determined at compile-time. This doesn't sound like a major problem, right?

Wrong.

This evening, I decided to write a wrapper around a vararg function that IDA exports, 'err(..)'.
This function takes some printf-style args, displays a messagebox with them, then exits IDA. However, I want unattended testbenches to run - so I don't want the messagebox. "No problem," I thought, "I'll just wrap err(..)".

No. You can't do that (without inline asm/etc).

You really do have no way (in hell) of calling the vararg function with compile-time variable arguments. So you, quite simply, can't. Life's a bitch. The best I can come up with is to count the arguments, then switch() on the arg count, and execute a line based on that. Like this:

ulong main::throwError(char* fmt, ... )
{
ulong args[20];
ulong argCnt = 0;
ulong i;

va_list myVA;
va_start(myVA, fmt);
i = (ulong)fmt;
while( i != -1 && argCnt<20 )
{
i = va_arg( myVA, ulong);
args[argCnt] = (ulong)i;
argCnt++;
}
va_end(myVA);
argCnt-=2;

char error[1024];
switch(argCnt)
{
case(0):
qsnprintf(error, 1024, fmt);
case(1):
qsnprintf(error, 1024, fmt, args[0]);
case(2):
qsnprintf(error, 1024, fmt, args[0], args[1]);
case(3):
qsnprintf(error, 1024, fmt, args[0], args[1], args[2]);
case(4):
qsnprintf(error, 1024, fmt, args[0], args[1], args[2], args[3]);
default:
// handle error somehow
}
dosomeloggingstuff(error);
}

A zillion drawbacks, like the hardcoded limit on args, but this is a lot cleaner than my original implementation. I'm going to quote the following, leave the rest to your imagination and horror, and leave it at that.

static ulong main::throwError(char* fmt, ulong arg1=NULL, ulong arg2=NULL, ulong arg3=NULL, ulong arg4=NULL, ulong arg5=NULL, ulong arg6=NULL, ulong arg7=NULL, ulong arg8=NULL, ulong arg9=NULL, ulong arg10=NULL );
...
else if (arg7==NULL)
qsnprintf(error, 1024, fmt, arg1, arg2, arg3, arg4, arg5, arg6);
...
It was NOT PRETTY.

I didn't even dare _look_ at the forum entry I found while googling about P/Invoking a varargs-style function.

If you have a better solution, which doesn't involve inline asm (I'm really not up to making my code that platform specific (and unstable, knowing my asm)), I'd be simply delighted to know it..

(as a postscript - the 'while( i != -1)' makes me wonder wtf happens if I call a vararg function with a parameter of -1.. )

Thu, 28 May 2009 02:11:42 GMT

Right, University term has ended, and I want to get some serious progress with lavalamp. Here's hoping. A summary of the night's work:

* Am intended to add support for Zigbee, via the Microchip MRF24J40 2.4Ghz transceiver. This may well require that I rewrite a lot of PIC-side code, but I think it'll be worth it, since the wireless layer I wrote isn't particularly great, and using Zigbee opens the door for some cool interoperability. It does, however, require quite a beefy PIC chip to run the software stack.

* I'm using a PIC18F46K20, instead of an PIC18F4620, mainly because it's cheaper (and more powerful). Hopefully I can make it work. I had to make the following changes to the example code (the co-ordinator project) to change microcontroller:

- in zLink.lkr, the linker script, change

DATABANK NAME=gpr0 START=0x0060 END=0x00FF
to
DATABANK NAME=gpr0 START=0x0080 END=0x00FF
(since bank 1 GPRs start at 0x80 and not 0x60)

and

DATABANK NAME=gpr15 START=0x0F00 END=0x0F7F
ACCESSBANK NAME=accesssfr START=0x0F80 END=0x0FFF PROTECTED
to
DATABANK NAME=gpr15 START=0x0F00 END=0x0F5F
ACCESSBANK NAME=accesssfr START=0x0F60 END=0x0FFF PROTECTED
(since bank 15 GPRs start at 0x0F5F and not 0x0F7F, and SFRs start at 0x0F60.)

also change
FILES p18f4620.lib
to
FILES p18f46K20.lib

- In zNVM.c, change

#if defined(__18F4620) && !defined(USE_EXTERNAL_NVM)
to
#if (defined(__18F4620) || defined(__18F46K20)) && !defined(USE_EXTERNAL_NVM)

- and similarly, in zigbee.def

#if !defined(__18F4620)
to
#if (!defined(__18F4620) && !defined(__18F46K20))

I just know this info will be useful to someone!

* My ICD2 warned me that Vpp was going to go too high for the device. I put a ~8V zener across it, under recommendation from The Internet, and it seems to work.

* I realised that the PIC18F46K20 is a 3.3V device, sometime after applying 5V to it. Ooops.

* I realised that the three-pin 3V3 regulators I have aren't 7805-pinout'ed, after neglecting to read their part numbers and hooking them up as if they were. Burnt my finger.

* Realised that having a pluggable 'transport' layer in lavalamp would be a really damn good idea, as then I can make a 'simple cabled' transport (probs SPI/ttl-level rs232 or the like) for simple stuff, a 'lavalamp native' transport (the one I have been working on to run over any RF Tx/Rx, which currently needs a lot of work) and 'Zigbee' for running Zigbee via the Microchip transcievers. This'd be a lot better than my current design, which has little abstraction between the transport layer and those above.

Videos of recent Uni work

Thu, 15 Jan 2009 09:01:10 GMT

Since it's the end of Semester 1 at Uni, a couple of projects have been coming to a close. I thought I'd post some videos of them running.

First up is the above, an interface from the Universities' leaning environment - a load of HTML in an education-targetted CMS named BlackBoard - in to Second Life. Unfortunately, the version of BlackBoard used by the Uni doesn't make any nice data feeds available, so most of the hard work is 'screen scaping' the relevant data out of pages. Because most of the work went in to this, I didn't get to experiment much with new-paradigm UIs - the above is the result of 12 weeks of code (!!).
Each module the student is taking is represented by an orb, which (when clicked) tells the student any new announcements that have been posted to that module.

(click here to open the video in a new window
Above is the fun one - a demo of my crypto accelerator. It's a fairly undramatic demo, but it represents, again, around 12 weeks of work. The accelerator is configured to crypt using XTEA at 32 rounds, and fed with some test data which it encrypts and then decrypts. Pay no attention to the timestamps - this is a fairly early and unoptimised version of the design. Things should be faster later on.

Sun, 28 Dec 2008 04:49:40 GMT

I recently threw together a small compute cluster from some spare parts I had. It comprises five Pentium 3 chips, clocked at 1Ghz, and one mobile P4 clocked at 1.4Ghz. Each has a relatively small amount of RAM - around the 64MB mark - mainly because I don't have more lying around.

The cluster boots via a modified version of (the rather excellent) LTSP environment. Each node is diskless, and boots via PXE, downloads a kernel/initrd, and mounts root via NFS. Obviously, disk access is slow in this configuration (compounded by the fact that the 'server' machine runs IDE disks, at UDMA 66) but it kicks out relatively little heat and chomps relatively little power. Each node can be powered on via WoL - important since the cluster is stored away from my house, making physical maintainence hard. There's a MOSIX kernel compiled, but not set up properly, as I don't need MOSIX yet. It's a shame that openMosix died (although great that I can get a student license of MOSIX!)

Anyway, one thing that the cluster is configured for is to use distcc, a distributed C compiler. There is some discussion over the optimum number of C programs to compile in parallel (the '-j' flag as passed to 'make') and so I thought I'd run some investigation.
I took a vanilla 2.6.27-10 kernel, ran make defconfig, and then ran make CC=distcc HOSTCC=distcc -j $j, where $j ranged from 1 to 10. The results were interesting, shown below:

These results were taken when the cluster comprised of four PIII 1GHz chips, with around 64 or 128MB memory per node. distcc was configured not to use the server machine as a compilation host (any more than was neccesary).

We can clearly see that 6-8 parallel processes is optimum - taking 15 minutes to compile the kernel. The elevated time at 5 is possibly a freak value, and values above 8 are probably the result of network congestion.

So there you have it - for a five node NFS-root no-swap low-memory p3-1Ghz cluster, use -j 7 :D

Hope this post is useful for someone. Please leave a comment if it is, or comment just if you think I'm wrong!

Sat, 20 Dec 2008 16:13:39 GMT

Really silly problem I just spent a good three hours figuring out, just posted here in case anyone else has a similar problem:

Situation was that I was booting diskless clients via TFTP/NFS root (ie, using LTSP). They're debian workstations. In order to get WoL working, I built a new kernel, pruned it (over-enthusiastically) and broke something.

Symtoms were that:

* No TTY open on the machine console
* SSH reports "sshdPRNG is not seeded"
* X didn't start

I scratched my head for ages, verifying that /dev/random and /dev/urandom were present, twiddling .config options, verifying that random.c was being compiled, etc. All the TTY options were present/enabled. Further up the kernel logs was the warning that /dev/mem was not found, which was bizarre, since all that was enabled, too.

Now, I have no idea how - but I had disabled UNIX domain sockets. D'OH!