Code segment encryption
Implementation of "Rootkit Arsenal" cryptor.

Partial encryption of a binary and self-decryption at runtime. This article is based on the cryptor described in Bill Blundens’ "The Rootkit ARSENAL" book(Ed1).

Article published on 2 March 2014
last modification on 29 May 2016

by Emeric Nasi

Note: This document requires base of C programming and Microsoft Windows C programming. Also this document is made by non-expert in the field of windows system programming so I hope experts reading this document will be indulgent, do not hesitate to tell me how to improve my code!
License : Copyright Emeric Nasi, some rights reserved
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Creative Commons License

I. Introduction.

In the Bill Blundens’ "The Rootkit ARSENAL" Edition 1 book, Mr Blunden describes how malware can evade detection and slow down reverse engineering by using code armoring. I implemented Bills’ cryptor and completed it to add patching of target program.
This article presents the modifications I made to the original code and the different tests I did on segment encryption.
It is not necessary to have the "Rootkit Arsenal" book to understand this article but the functions I did not modify are not in this article and if you need the full source code you can ask me by email or may also want to buy this excellent book:-).

II. Theory

II.1 Memory segmentation

An executable file, and its virtual memory image are made up of several sections. Some of this sections are represented by memory segments. Some segments are used to store data and others to store executable machine code. In this article we present how a program can encrypt one of the segment of another application and how this application can self decrypt its encrypted segment at runtime.

Note : Even if it is possible to merge the segments in a way that the entire application is encrypted (except decryption stub), this article does not describe how packer works. In this article we need access to the source core of target application. It is easier to code and also more felxible since it can be useful to encrypt a small part of an application. The next article about hiding meterpreter shellcode in an executable program and bypassing AntiVirus demonstrates this.

Here is the classical segmentation of a Windows portable executable binary file.

Other segment can appear in your files but these are the most important ones. PE headers is complex and I recommend you find some information about it. In this article we use the PE optional header which contains information about the application segments.

The .textbss segment is empty on the binary file. It is used to reserve room in virtual memory for non initialized global variables

The .text segment contains the executable code of the program

The .rdata segment contains read-only data. It is used to global constants (which includes strings). For example in printf("hello"); "hello" goes in .rdata.

The .data segment contains initialized global variables which are not constants. for example this global variable char var[] =  "var"; is not a constant string, it is an array and goes in .data.

The .rsrc segment is present if you include resources to your file (*.rc files). Resources file can be used to provide information such as company name and application version.

When the executable is loaded in memory, the stack and heap segment are initialized to store local variables and dynamic allocated value. A part from this the segment layout will not change that much as you can see

II.2 Mechanism

What we want to do is modify an application (the target) in such a way that another application (the cryptor) can encrypt one of its segment. We also want the target application to decrypt itself at runtime.
Following the "Rootkit arsenal" book, we are going to create new segments in the target.

  • a .code segment that will be encrypted.
  • a .stub segment that is used to decrypt .code segment

In his book Bill Blunden merged the .text and .data segment into .code before encrypting it. There are a lot of other ways to do it. You can merge everything into one segment. You may want to have .text and .code separated so you don’t encrypt all your executable code. you can also create your own data segment and merge it to .code so that you don’t have to encrypt all the application initialiazed global variables and just hide the data you want.

In this article we are going to create a .code segment that will contain the executable code we want to cipher. To make it simple we are not going to merge another segment to it.

Here is the segment layout we will have in this example

II.3 Tools

Here are the tools you need:

  • a C source code editor
  • a C compiler/linker
  • a debugger
  • dumpbin.exe tool
  • a hex editor

Personally I used Microsoft Visual Studio express 2010 which is free and provides the source code editor, the compiler, the linker, the debugger (with a very practical code machine view), and dumpbin.exe is provided by the Visual Studio shell.
I recommend next to Visual Studio you open the Visual studio CMD shell (can be found in startup/program menus). You can easily get PE file segments information and verify your values are ok using:
c:\test>dumpbin /HEADERS file.exe

Example of output:

For the hex editor I used the free edition of Cygnus Hex Editor.

III. Adapt target software

III.1 Create .code segment

It is very simple to create a new segment using #pragma section.
You can then specify in which segment your information goes. There are 4 types of information in segments: executable code (pushed with #pragma code_seg), initialized data (pushed with #pragma data_seg), uninitialized data (pushed with #pragma bss_seg) and constant variables (pushed with #pragma const_seg
Remember that a code segment will not store global variables or strings.

In out example, we want to put executable code of a C file in .code segment.

III.2 Create decryption stub

Why do we need a .stub segment since we do not encrypt all the code? Well we need the cryptor to be able to patch the target self-decryption routines and the code we want to patch will be easier to locate in the .stub section. Also variants of this article can be used to merge all segments into one (.code). So having .stub section is more generic.


III.4 Compilation and link options

Because the target will self-modify itself, we must avoid security cookies (used for stack verification). For that we need to remove security check. The next compilation options are thus mandatory:

/GS-  -> Disable functions stack verification relying on secure cookies 

Another option is almost mandatory, statically include the Runtime Library. If the Microsoft Runtime Library is not included in the target, this code will work but you will face portability issues.
Use one of the next two Runtime Library options.

/MTD -> for debug
/MT -> for release

To avoid complications we want to have fix address and remove data execution prevention. For that we have to use the next options for linker.


You may also be interested into removing debugging informations but it is not mandatory for this code to work (and very useful to debug when it doesn’t work!).

III.5 Other possibilities

Other segments layouts are possible, and will be described in future articles.
For example, you can encrypt data and constant in your code by creating a custom data segment.

Another possibility is to merge .text, .data, .rdata segments into .code which results in the next memory layout:

Note that in this case, the runtime library is encrypted. That means you need to declare a new entry point for the target, example:

There is however a drawback to this method. Because of the strange PE header and the encryption of most of the file, some AntiVirus may consider the result of this method to be a malware.

IV. Build Cryptor

IV.1 Retrieve wanted information

The role of the cryptor is to encrypt the target programs’ .code segment as well as patching the.stub segment so that the target is able to self-decrypt when launched.
For that, the cryptor will browse through the binary file generated in the previous section and will find the address and size of .code and .stub segment. We need both file offset (to modify the binary target) and virtual memory address (to indicate to target program where is the segment he should decrypt at run-time).

First you need to map the file in memory using CreateFileMapping and MapViewOfFile. I did not do any functional modifications on this code. It can be found in the book or on the Internet.
Once this is done we parse the mapped file to fetch section headers informations.

The next function is used get the next information for any section:

  • The segment offset in raw file
  • The file segment size
  • The virtual memory offset of the segment

IV.2 Encrypt target segment

Now that we have the size and location of .code segment in binary file, we can open the file and encrypt the wanted bytes.
This code is almost the same as provided by Blunden except the adaptation for the multiple bytes XOR encryption (witch is not much!). The code is not really optimized but very practical for debugging purpose. In this function we:

  • Open the binary target file
  • Seek .code segment
  • Load .code segment in buffer
  • Encrypt buffer
  • Write encrypted buffer in place of clear-text .code segment
  • Close file and leave

IV.3 Patch target .stub section

After having encrypted the .code section, we need to patch the .stub section so that the target can decrypt itself. In this function we:

  • Open the binary target file
  • Seek .stub segment
  • Load .stub segment in buffer
  • Replace values by virtual memory offset and size of .code section
  • Write patched buffer in place of .stub segment
  • Close file and leave

IV.4 Main function