Since Microsoft is now (May 2018) providing symbols for debugging Hyper-V, and encouraging security researchers to look at Hyper-V. I figured it would be a good timing to open-source a utility for Hyper-V I wrote back in 2010.
LiveCloudKd was the first utility to focus on Virtual Machine introspection for memory forensics purposes, it was released in 2010 after some initial research on Hyper-V v1.
This feature was later added into LiveKd 5.0 by Mark Russinovich and Ken Johnson. https://blogs.technet.microsoft.com/markrussinovich/2010/10/09/livekd-for-virtual-machine-debugging/
One of the initial attribute of LiveCloudKd is that the solution was completely user-mode (!!!) due to some design flaws inside vmwp.exe (Virtual Machine Worker's process).
- The process had no isolation so it was possible to read its memory address space from another process running with administrator privileges.
- Memory Block's handles were not indexes inside an object tables but kernel mode addresses pointing to those Memory Block kernel-objects. Yes, you read that correctly. vmwp.exe used to pass kernel-mode pointers to vid.sys
- No official documentation was available regarding vmwp.exe vid.dll and the drivers winhv.sys and vid.sys BUT an Microsoft open-source project (Singularity) that was focusing on experimenting .NET as a foundation for a new Operating System leaked full headers for Vid.dll (Virtualization Interface Driver).
Each Virtual Machine has one vmwp.exe, which has one Partition Handle (PT_HANDLE) - and multiple Memory Block handles (MB_HANDLE), this is particularly true if the target Virtual Machine has the Dynamic Memory (VidDm*) feature enabled.
First, we needed to retrive the original Partition Handle (PT_HANDLE) returned by VidCreatePartition() to the vmwp.exe. Unfortunately, no API was present to retrieve existing partition handles. But since the process was not isolated we could just look for handles within each vmwp.exe process with an object name starting with \\Device\\000000, and then we could validate each of the retrieved handles with a basic API call such as VidGetPartitionFriendlyName().
More details are available in [partition.c!IsPartitionHandle()](https://github.com/comaeio/LiveCloudKd/blob/master/hvdd/partition.c#L141):
Secondly, once we recovered the valid Partition Handles corresponding to each Hyper-V Virtual Machine, we need to retrieve its Memory Block's Handles. This is where I brute-forced the memory address space of each vmwp.exe to collect all the kernel pointers before verifying if they were valid Memory Block handles or not. More details can be found in memoryblock.c!GetMemoryBlocks().
Once we have a PT_HANDLE and a MB_HANDLE we can pass them as arguments to VidReadMemoryBlockPageRange(). Overall, LiveCloudKd was using only few Vid.dll functions:
- VidDmMemoryBlockQueryTopology()
- VidQueryMemoryBlockMbpCount()
- VidGetPartitionFriendlyName()
- VidTranslateGvaToGpa()
- VidReadMemoryBlockPageRange()
- VidGetVirtualProcessorState()
Last but not least, in order to create a livekd style image on the fly I would just hook the Export Address Table (EAT) of WinDbg.exe / kd.exe. More details can be found in DumpLiveVirtualMachine()
After discussing with @gerhart_x on why LiveCloudKd was not working anymore on the current version of Hyper-V, @gerhart_x offered to work on an expiremental feature to revive LiveCloudKd again. Unfortunately, this requires a kernel driver.
Thanks to @aionescu who pointed out to me the existence of the Windows Hypervisor Platform API (WHVP) where I noticed the presence of the ReadGuestPhysicalAddress() API which is now publicly available as of Windows 1803 (10.0.17134.48). This may be a good lead to create a current lean version of LiveCloudKd and re-enable on the fly memory forensics for Hyper-V Virtual Machines. Simpleator is a great example of an application leveraging those APIs if you want to learn more about it.
Add new methods for access guest Hyper-V VM Memory: ReadInterfaceWinHv (uses Hyper-V hypercall for reading. Slow, but robust method) and ReadInterfaceHvmmDrvInternal (read data directly in kernel memory. Much faster, then ReadInterfaceWinHv, but uses undocument structures). See description of -m option. Default reading memthod is ReadInterfaceHvmmDrvInternal. ReadInterfaceHvmmDrvInternal was tested in Windows Server 2019 (march 2019 updates), Windows 10 x64 1803, 1809, 18362 Tested on Full VM, Windows Defender Application Guard and Windows Sanbox from 18362
For launch:
- hvmm.sys driver is not signed, therefor start host OS with TESTSIGNING option (Bcdedit /set TESTSIGNING ON)
- Place LiveCloudKd.exe, LiveCloudKdSdk.dll, hvmm.sys to WinDBG x64 folder (tested on WinDBG from WDK 1809 and WDK 1903)
- Launch LiveCloudKd.exe with admin rights (It needs Visual Studio 2019 runtime libraries - https://aka.ms/vs/15/release/vc_redist.x64.exe)
- Choose virtual machine (it can be Full Vm or Hyper-V container) for attaching