Rootkits and Blue Pill

Please read WFA pp. 308-328 if you haven't already.

What is a rootkit?

It's a modification, usually of the operating system itself or access to the operating system, in an attempt to hide the presence of malware.

What is Blue Pill?

It's a complete replacement of the user's operating system by a virtualized environment. While it is generally termed as a rootkit, the idea is that the only state that is the same is non-volatile or non-local state.

Rootkits in the real world

The first place to look was the venerable http://www.rootkit.com from HB Gary. Unfortunately, with the compromise of HB Gary, rootkit.com seems to be defunct. Some of the more interesting techniques mentioned were:

"Hide in plain sight": reversing the paradigm of trying to be as unobtrusive as possible, the rootkit simply exists in an ordinary file and use standard WIN32 API calls.
"Catch me if you can": use a technique where the rootkit is always moving itself around, and uses ostensible HTTP traffic to obfuscate command and control communications.
"Project B": use a Firewire port's vulnerable to memory manipulation to inject the initial code; the code then moves to the MBR to circumvent defenses.
"Air gap" attack: the scenario is that the defenders are using an air gap to protect their most important machines. The vulnerability that the attacker attempts to exploit is if the defender uses USB sticks to move data; try to infect those (like Stuxnet, for instance). Then (the wilder part) try to generate high frequencies (above normal hearing) and have these picked up by microphone on compromised machine.

Detection

One prevalent idea is the idea of "crossviews"; you use any differences in the view from the suspect system and a different one to detect rootkits. Another is direct analysis (to the extent possible, at least) of memory from within a suspect system and what standard tools are telling you is there.