2023.01

eShard acquired Tetrane in July 2022! You can read more about this on the blog: eShard takes a step into Reverse Engineering by acquiring Tetrane.

Highlights

Summary

esReven version 2023.01 is the first version released by eShard, and is a major step forward in the life of the product:

  • esReven is the new product name for Reven. The release versioning scheme will now be "Year.Month" like 2023.01, as opposed to 2.11.0 which was the last version of Reven by Tetrane.
  • The Reven engine is now integrated in eShard's platform to become esReven, bringing JupyterLab & other features.
  • Following eShard's strengths and practices, two knowledge-base modules now come bundled with esReven: "Getting Started" and "Vulnerability Analysis" - this is a first step in providing more knowledge & practical know-how to our users about the product, its usage and use cases.
  • To even better support the JupyterLab environment, and following the trend set by the previous 2.11.0 version, multiple API objects have been introduced: you can now easily parse the call tree, or display the framebuffer, among others.

In more details

  • Reven is now esReven: this is of course the most visible element. Reven has been integrated in the more generic eShard framework, which is now the top-most UI and includes the existing Project Manager. It also comes along with JupyterLab, which is the evolution of the regular Jupyter we know and love. Moreover, we will be able to better handle separate users from now on, and this is already working for the JupyterLab side of things. This is only the beginning, expect more work on this side in the future!
  • Knowledge-base modules: eShard has a history of providing both the platform for studying targets (hardware & software) and the knowledge & experience for doing so. The latter does not only come from selling services but more importantly from knowledge modules. Each module is a set of interactive JupyterLab notebooks produced by experts, that provides theoretical knowledge and practical know-how about a subject, such as a specific attack or vulnerability. This approach has been very successful, and we are working on bringing it to esReven as well. The result of this work so far is a set of two modules that we decided to include with this version of esReven for free: the "Getting started" module to on-board new (or existing but rusty) users, and a "Vulnerability Analysis" module, which takes the user on the ride of recording and analyzing a CVE in Google Chrome. We want to continue on this path: expect more work on this side as well in the future!
  • New APIs: Following the trend set by the previous 2.11.0 version, we kept on improving working from the Python API, making algorithms easier to build, and in general making the experience of exploring a trace from Python smoother. For this, we strive to make APIs that provide access to high level objects while keeping the complexity as low as possible.
    • We improved the Type API that enable parsing typed data in the target memory.
    • We added access to the "stack event" information, which among other allows parsing the tree of calls from a function, or in general focusing on the code of a function to answer questions such as "What are the memory access made by this function".
    • We gave a simple access to the framebuffer, to make illustrating notebooks more convenient.

A focus on the Python API

Type API

First of all, we have improved our Type API, which allows a user to parse high-level data in the target's memory such as typed structures, arrays, etc.:

  • It is now easier to browse the content of instances of typed object in memory:
    • using StructInstance.fields() we can now parse all members of a struct without having to fetch member's names first.
    • ArrayInstance now reliably contains PointerInstance objects, while they would sometimes contain addresses instead.
  • There is now a dedicated __format__ method for StructInstance objects. Supported by proper format methods in all objects that can be contained (FieldInstance, ArrayInstance, etc.), this allows for very straightforward formatting:
    • Here is an example of the default formatting: 
      >>> print(f"{instance}")
struct _RTL_AVL_TREE /* 0x8 */ {
    /* 0x0 */ Root : _RTL_BALANCED_NODE* = (...)* @ds:0xffffe00013174e80,
}
    • Alternatively, you can choose to display the inner structures, up to a certain depth (default is 1): 
      >>> print(f"{instance:max_depth=2}")
struct _RTL_AVL_TREE /* 0x8 */ {
    /* 0x0 */ Root : _RTL_BALANCED_NODE* = (struct _RTL_BALANCED_NODE /* 0x18 */ {
        /* 0x0 */ Children : [_RTL_BALANCED_NODE*; 2] = [...],
        /* 0x0 */ Left : _RTL_BALANCED_NODE* = (...)* @ds:0xffffe000138fa510,
        /* 0x8 */ Right : _RTL_BALANCED_NODE* = (...)* @ds:0xffffe00012818bc0,
        /* 0x10 */ Red : U8[0..1] = 0,
        /* 0x10 */ Balance : U8[0..2] = 0,
        /* 0x10 */ ParentValue : U64 = 0,
    })* @ds:0xffffe00013174e80,
}
  • If more than one type share an identical name, we can now get them all by using the Binary.types method. Notably, this allows fetching the proper _KPCR structure type on Windows 11.

Stack Events and Call Tree

We have introduced the "Stack Events" data into the Python API under the reven2.stack objects (accessible through ctx.ossi.stack). In this API, a Frame represents the time span one function call lives in, and what it does.

  • stack.frame gives access to the current StackFrame, while stack.frames() still gives all parent frames as was the case in previous version.
  • Each frame's exact time boundaries are explicit, and allow not only a display that is on-par with Axion, but precise control for your algorithms:
    • first_context and last_context give you the natural start and stop of the frame. However, there are plenty of edge-cases possible, so we also added:
    • first_executed_context gives the first time a frame appears if it has started before the beginning of the trace
    • function_start and function_location gives the resolved function start (in time) and location (in memory) (taking everything into account, such as trampolines or out-of-trace starts).
  • You can easily parse all calls made in a Frame with children, or access the Frame's parents with ancestors. Using this API recursively is possible, so you can imagine displaying a call tree as such: 
    >>> def print_children(ctx: reven2.trace.Context) -> None:
...     active_frame: reven2.stack.StackFrame = ctx.stack.frame()
...     print(active_frame)
...     for child in active_frame.children():
...         print(f"|- {child}")
...
>>> print_children(server.trace.first_context + 200_000)
#199862 - usbhub!UsbhDecHubBusy
|- #199886 - ntoskrnl!KeWaitForSingleObject
|- #200032 - ntoskrnl!ExFreePoolWithTag
|- #200239 - ntoskrnl!KiAcquireKobjectLockSafe
|- #200269 - ntoskrnl!KiExitDispatcher
  • Moreover, we need to handle the fact that there can be many different situations when analyzing a function: we might be interested in its own code only, skipping its children. Or we might want the children's code, up to certain binaries. Finally, we often want to skip external code such as kernel code, or of other processes that might get scheduled between first_context and last_context. In order to handle all these various situations easily, we introduced another call: descendant_events. It yields all the relevant events from a frame:
    • FrameStart: a child frame starts. From this point, I can check the child's symbol, binary, etc.
    • FrameEnd: a child frame end, allowing me to resume my analysis if I skipped over the child call.
    • StackLeave: I am entering a portion of the trace where I'm no longer executing from on this particular Stack, for instance I've entered kernel code, or another process. I should really ignore everything that is going on.
    • StackEnter: I am back into my stack, so into my frame or a child.
  • We also know that the complexity of this call tree can rapidly grow out of control, so for these situations we ensure the iterator from descendant_events could skip entire frames with skip_children(). For example, you can control the display depth as such: 
    >>> def print_descendants(ctx: reven2.trace.Context) -> None:
...     depth = 0
...     it = ctx.stack.frame().descendant_events()
...     for event in it:
...         if isinstance(event, reven2.stack.FrameStart):
...             # [...] Do something
...             depth += 1
...             if depth >= 3:
...                 # Skip any child this call may have and skip to its return
...                 it.skip_children()
...         elif isinstance(event, reven2.stack.FrameEnd):
...             depth -= 1

Framebuffer API

There is now a framebuffer entry point in the Context object! Use it to get a straight PIL.Image object representing the VM screen at this point in time. For example, fetch the screen at the start of the trace with: server.trace.first_context.framebuffer.image(). This is also very convenient to use in notebooks, as you can for instance control the resulting image's size with resize((width, height)).

OSSI API

You can now access information about the OS running in the trace from server.ossi.os(). For instance:

>>> print(server.ossi.os())
Windows x64 10.0.17763 (Windows 10)

Workflow API

Finally, there have been improvements on the side of managing scenarios and archives:

  • You can now open a scenario by uuid or by name, using ProjectManager.get_server_by and specifying either name or uuid argument.
  • You can now upload a scenario archive from the API with ProjectManager.upload_scenario

Changes & Improvements

Installation

  • esReven's package now uses Docker images & Docker Compose. Migrating any previous type of installation and its data is fully supported. See the note about upgrading for more details. The Docker approach unifies the installation procedure, allows to install esReven on various Linux distributions and versions, allows the use of standard tools to operate the server.

esReven web interface

  • The eShard's web framework is now used for the top-most user interface. It provides access to the components of esReven:
    • The Project Manager, which you may already know from Reven.
    • A Knowledge Base, served by a JupyterLab instance, that replaces Reven's Jupyter instance and contains several KB modules.
    • The offline documentation - it has now the same form as the online documentation and is fully searchable.
  • The Web Interface supports multiple distinct users.
    • JupyterLab spaces are individual to each user. There is also a shared space that users can use to exchange notebooks and data.
    • On the other hand, all users share a single Reven instance, hence also Reven's VMs, projects, etc.

esReven Project Manager

  • The Project Manager is now a part of the more general esReven Web Interface.
  • You can now upload a scenario archive directly from the "Scenario" tab.
  • The Reven Server ports when opening a scenario will now use a port within a configurable range, to make it easier to route them from the host to the correct docker instance. The default range is 14000 to 14099.
  • The same is true for debugger-assisted recording incoming ports (aka VMI). The default range is 14100 to 14199.
  • Creating VBox VMs and recording VBox scenarios is no longer possible: this capability's perimeter was very specific, and was seldom used - it was time for us to let it go.

Reven engine

  • The OSSI module can now detect the Windows 11 OS.
  • The Cap'n Proto library has been bumped to version 0.10.3.

Python API

  • Compatibility note: reven2.stack.StackFrame.first_context may return a different context in the case of a frame of type Unknown. Previously it would return the first context where the frame was directly executed (not one of its children call), now it returns the first context where this frame or one of its child frames is being executed. The old behavior is now provided by reven2.stack.StackFrame.first_executed_context.
  • Compatibility note: reven2.stack.StackFrame.type and reven2.stack.StackFrame.creation_transition are now deprecated. Use the reven2.stack.StackFrame.start_event to retrieve this information.
  • The reven2.stack module has been extended to support the stack event API:
    • A reven2.stack.StackFrame now gives access to its call subtree and super tree via reven2.stack.StackFrame.ancestors, reven2.stack.StackFrame.children, reven2.stack.StackFrame.reversed_children and reven2.stack.StackFrame.parent.
    • A reven2.stack.StackFrame now gives access to the location and transition of the function executed in that frame via reven2.stack.StackFrame.function_location and reven2.stack.StackFrame.function_start.
    • A reven2.stack.StackFrame now gives access to the relevant event in the stack events, such as its reven2.stack.StackFrame.start_event, reven2.stack.StackFrame.end_event and reven2.stack.StackFrame.descendant_events (that include stack change events). To support this, the reven2.stack.EventsIterator, reven2.stack.FrameStart, reven2.stack.FrameEnd, reven2.stack.FrameStartType, reven2.stack.FrameEndType, reven2.stack.StackEnter and reven2.stack.StackLeave, reven2.stack.StackSpace types have been added.
    • Added reven2.stack.StackFrame.last_context that provides the last context where a frame exists.
    • Added reven2.stack.Stack.frame as a shortcut for next(stack.frames()), giving more convenient access to the current frame.
    • Added __eq__ and __hash__ methods to reven2.stack.Stack and reven2.stack.StackFrame.
    • For more information, please refer to the updated documentation of the reven2.stack module.
  • Added reven2.ossi.Ossi.os to get the OS family, version, kernel version and architecture of the current scenario.
  • reven2.types.ArrayInstances whose element type is a pointer now always contains a list of reven2.types.PointerInstances. Previously, it would sometimes contain a list of addresses.
  • The instance types in reven2.types now have a string representation for printing.
  • Added reven2.types.StructInstance.format, reven2.types.PointerInstance.format and reven2.types.ArrayInstance.format.
  • Added reven2.types.StructInstance.__format__, reven2.types.PointerInstance.__format__ and reven2.types.ArrayInstance.__format__.
  • Added reven2.util.parse_colon_separated_key_values.
  • Added reven2.types.StructInstance.fields to iterate on the field and bitfield instances of a struct instance.
  • Iterating on fields with reven2.types.Struct.fields now always resolve the types of the field. Previously, the fields were resolved only when calling reven2.types.Struct.field on a specific field.
  • Fixed a RecursionError when attempting to resolving a type containing an array of pointers to that type.
  • Add reven2.ossi.Binary.types to get named types from a debug object for Windows scenarios in case there are multiple types with the same name.
  • Fix a bug in preview.windows when the kernel had twice the structure _KPCR in its PDB.
  • Add reven2.framebuffer.Framebuffer API to get the current framebuffer as an image.
  • Add reven2.trace.Context.framebuffer to get the reven2.framebuffer.Framebuffer at a context.
  • Add reven2.ossi.Location.__eq__, reven2.ossi.Location.__ne__ and reven2.ossi.Location.__hash__.
  • Add preview.project_manager.ProjectManager.upload_scenario to upload a scenario archive.
  • Add ProjectManager.get_server_by scenario name or uuid and ProjectManager.close_server in preview.project_manager.
  • reven2.trace.Transition and reven2.address are not displayed as a clickable link in Jupyter anymore.

Fixed issues

Reven engine

  • The ASM-stub auto record now properly resumes the VM when the recording stops. Moreover, errors encountered by the ASM-stub or the Binary auto-record that would stop the recording will now properly resume the VM as well.
  • Trying to generate the OSSI Range resource does not crash when OSSI are not available for the recording, which could be the case of exotic traces such as recording GRUB.
  • The Memory History replayer now will not crash when registering a 0-byte access, but instead properly report an error.
©2019-2023 eShard