Part 1: Running the KMS inside the enclave
ℹ️ Info
This part can be done in simulation mode.
As promised, let's launch our very first enclave!
To do so, we'll need to set-up the compilation logic, code the HTTPS server on the enclave side and launch it with the host side.
Makefiles and compilation logic
The structure of the code will be separated in two.
- One part will be the code running in the host that will launch the enclave.
- One will be the code running inside the enclave.
✅ Ready-to-use Makefiles 😉
The purpose of this tutorial isn't to torture you into endlessly writing Makefiles, so we prepared a skeleton of what you'll need to build and run the code for the enclave!
Cloning the project
Let's begin by cloning the GitHub repository containing the skeleton of this mini-KMS project:
$ git clone https://github.com/mithril-security/Confidential_Computing_Explained.git
$ cd mini_kms
In the mini_kms directory, you'll find a skeleton directory, and other folders (part_1, part_2...) where each one correspond to a part of this Confidential Computing course.
You can also fork the repo into your projects, to work on your own files and maybe add all the features that you need.
🖌 You already know all this?
If you are already familiar with how to launch an enclave and want to use Open Enclave directly,
skip to the remote attestation in part 2, this is the moment! You can copy the folder onto another directory or work on it as is.
Exploring the skeleton folder
The tree for the skeleton compilation logic is the following:
skeleton/
├── Makefile // The Makefile that runs the Enclave & host Makefiles
├── README.md
├── config.mk // Used by the Makefile to detect and set the compiler installed on the machine
├── enclave
│ ├── Makefile // Enclave Makefile
│ ├── enclave.cpp // Where the enclave code will be written
│ ├── kms.conf // Sets up information related to the execution of the enclave
│ └── trace.h
├── host
│ ├── Makefile // Host Makefile
│ └── host.cpp // Where the host code will be written
└── kms.edl // Our EDL file
2 directories, 10 files
The enclave folder
The Makefile in the enclave folder specifies rules for building an Enclave program using the Open Enclave SDK (Software Development Kit), and signing the program using its oesign tool.
In the spirit of not wasting time on configuration not relevant to this tutorial (and avoiding torture), we will use a C embedded web server library called Mongoose to help us build the HTTPS server. We chose this library because it is easy to use, well-maintained and because it works well with Mbedtls, the crypto library we'll be using.
To install it, run the following commands from the root of the skeleton folder:
$ cd skeleton/ && git clone https://github.com/cesanta/mongoose.git
The Makefile defines several variables which are used in the build process:
- CRYPTO_LDFLAGS
- CFLAGS
- LDFLAGS
- INCDIR
- CC
- CXX...
The all target depends on three other targets:
- Build to compile the enclave program.
- keys to generate cryptographic keys.
- Sign to sign the program using the keys.
To add socket, network and syscall support to the enclave, we must link rewritten libraries for OpenEnclave at compilation time. They are necessary to run the web server and communicate through it.
- loelibcxx for C++ components
- loehostsock for socket support
- loehostresolver for DNS resolution services
- loehostepoll for epoll system call implementation
The clean target removes all generated files.
ℹ️ A build in 3 steps
The Makefile uses the oeedger8r tool to generate interface code from the kms.edl file (which defines the enclave interface).
Then it compiles the enclave and links it with the Open Enclave SDK libraries and the cryptographic libraries specified by the CRYPTO_LDFLAGS variable.
Finally, the program is signed using the private key generated by the keys target and the kms.conf configuration file.
include ../config.mk
CRYPTO_LDFLAGS := $(shell pkg-config oeenclave-$(COMPILER) --variable=${OE_CRYPTO_LIB}libs)
ifeq ($(LVI_MITIGATION), ControlFlow)
ifeq ($(LVI_MITIGATION_BINDIR),)
$(error LVI_MITIGATION_BINDIR is not set)
endif
# Only run once.
ifeq (,$(findstring $(LVI_MITIGATION_BINDIR),$(CC)))
CC := $(LVI_MITIGATION_BINDIR)/$(CC)
endif
COMPILER := $(COMPILER)-lvi-cfg
CRYPTO_LDFLAGS := $(shell pkg-config oeenclave-$(COMPILER) --variable=${OE_CRYPTO_LIB}libslvicfg)
endif
CFLAGS=$(shell pkg-config oeenclave-$(COMPILER) --cflags)
LDFLAGS=$(shell pkg-config oeenclave-$(COMPILER) --libs)
INCDIR=$(shell pkg-config oeenclave-$(COMPILER) --variable=includedir)
all:
$(MAKE) build
$(MAKE) keys
$(MAKE) sign
build:
@ echo "Compilers used: $(CC), $(CXX)"
oeedger8r ../kms.edl --trusted \
--search-path $(INCDIR) \
--search-path $(INCDIR)/openenclave/edl/sgx
$(CXX) -g -c $(CFLAGS) -DOE_API_VERSION=2 enclave.cpp -o enclave.o
$(CC) -g -c $(CFLAGS) -DOE_API_VERSION=2 kms_t.c -o kms_t.o
$(CC) -g -c $(CFLAGS) -DOE_API_VERSION=2 ../mongoose/mongoose.c -lmbedtls -lmbedcrypto -lmbedx509 -D MG_ENABLE_MBEDTLS=1 -o mongoose.o
$(CXX) -o enclave kms_t.o mongoose.o enclave.o -D MG_ENABLE_MBEDTLS=1 -loelibcxx -loehostsock -loehostresolver -loehostepoll $(LDFLAGS) $(CRYPTO_LDFLAGS)
sign:
oesign sign -e enclave -c kms.conf -k private.pem
clean:
rm -f mongoose.o enclave.o enclave enclave.signed private.pem public.pem kms_t.o kms_t.h kms_t.c kms_args.h
keys:
openssl genrsa -out private.pem -3 3072
openssl rsa -in private.pem -pubout -out public.pem
The host folder
The Makefile in the host folder builds the untrusted host program.
The build target uses oeedger8r to generate interface code from the kms.edl file, which defines the interface between the untrusted host program and the trusted Enclave program. It then compiles the host program and links it with the Open Enclave SDK libraries specified by the LDFLAGS variable.
The clean target removes all generated files.
The Makefile defines the CFLAGS, LDFLAGS, and INCDIR variables to obtain compiler and linker options from the Open Enclave SDK. It compiles host.cpp and kms_u.c into object files and links them into a single executable file named kms_host.
The build target does not build or sign the Enclave program.
The kms_host executable launches the Enclave and runs it.
# miniKMS
include ../config.mk
CFLAGS=$(shell pkg-config oehost-$(COMPILER) --cflags)
LDFLAGS=$(shell pkg-config oehost-$(COMPILER) --libs)
INCDIR=$(shell pkg-config oehost-$(COMPILER) --variable=includedir)
build:
@ echo "Compilers used: $(CC), $(CXX)"
oeedger8r ../kms.edl --untrusted \
--search-path $(INCDIR) \
--search-path $(INCDIR)/openenclave/edl/sgx
$(CXX) -g -c $(CFLAGS) host.cpp
$(CC) -g -c $(CFLAGS) kms_u.c
$(CXX) -o kms_host kms_u.o host.o $(LDFLAGS)
clean:
rm -f kms_host host.o kms_u.o kms_u.c kms_u.h kms_args.hNow that we know how our project will compile, let's start writing the Host code that will launch the Enclave and start the server via the Ecall.
Makefile commands
To run the enclave and the host, all you need to do is:
- Verify that all the libraries and name are correctly written on the Makefiles.
- Run make all to clean and build.
- Run make run to run the host binary that will launch the enclave.
- OR make run simulate for simulation mode.
Host code
We begin by adding the necessary header files for the host:
// host/host.cpp
#include <stdio> /* for standard I/O operations */
#include <openenclave/host.h> /* for creating enclaves */
#include <sys/stat.h> /* for working with file permission */
#include <sys/types.h> /* same */
#include <fstream> /* for working with file streams */
#include <iostream> /* same */
#include <string> /* for working with strings */
#include "kms_u.h" /* one of the oeedger8r generated files */
using namespace std;
Then we'll open the host.cpp file in the hostrepository and finally start coding our first functions!
We'll begin with check_simulate_opt, which will check if the --simulation option was passed as a command line argument. It returns a boolean value indicating whether or not to run the program in simulation mode:
// host/host.cpp
bool check_simulate_opt(int* argc, const char* argv[])
{
for (int i=0; i<*argc; i++)
{
if (strcmp(argv[i], "--simulation"))
{
cout << "Running on simulation mode" << endl;
memmove(&argv[i], &argv[i+1], (*argc - i)* sizeof(char *));
(*argc)--;
return true;
}
}
return false;
}
Then we'll write create_enclave, a function that will load the enclave binary image from the specified enclave_path and initialize the enclave with the specified flags.
// host/host.cpp
oe_enclave_t* create_enclave(const char* enclave_path, uint32_t flags)
{
oe_enclave_t* enclave = NULL;
printf("[Host]: Enclave path %s\n", enclave_path);
oe_result_t result = oe_create_kms_enclave(
enclave_path,
OE_ENCLAVE_TYPE_AUTO,
flags,
NULL,
0,
&enclave);
if (result != OE_OK)
{
printf(
"[Host]: Enclave creation failed at enclave init : %s\n", oe_result_str(result)
);
}
else {
printf(
"[Host]: Enclave created Successfully.\n"
);
}
return enclave;
}Then we'll write the main function that is executed when the program is run. Here's how the logic goes:
- First, it declares and initializes some variables, including a flag for debugging, a pointer to an enclave, and variables to specify the port and whether to keep the server up.
- It creates the enclave using the create_enclave function and the provided enclave path and flags.
- If the enclave creation fails, the program jumps to the exit label and terminates the program.
- Otherwise, it sets up the http server using the set_up_server function with the enclave, port, and server keep-up variables.
- If the server setup fails, it also jumps to the exit label and terminates the program.
- Finally, it terminates the enclave and returns a status code.
// host/host.cpp
int main(int argc, const char* argv[])
{
oe_result_t result;
int ret = 1;
uint32_t flags = OE_ENCLAVE_FLAG_DEBUG;
oe_enclave_t *enclave = NULL;
char* server_port_untrusted = "9001";
bool keep_server_up = false;
cout << "[Host]: entering main" << endl;
// if (check_simulate_opt(&argc, argv))
// {
// flags |= OE_ENCLAVE_FLAG_SIMULATE;
// }
// if (argc != 2)
// {
// cout << "Usage" << argv[0] << "enclave_image_path [ --simulation ]" << endl;
// goto exit;
// }
enclave = create_enclave(argv[1], flags); // Call to create_enclave
if (enclave == NULL)
{
goto exit;
}
printf("[Host]: Setting up the http server.\n");
ret = set_up_server(enclave, &ret, server_port_untrusted, keep_server_up); // call to the ecall set_up_server
if (ret!=0)
{
printf("[Host]: set_up_server failed.\n");
goto exit;
}
exit:
cout << "[Host]: terminate the enclave" << endl;
cout << "[Host]: running with exit successfully." << endl;
oe_terminate_enclave(enclave); // Ending the enclave and freeing memory
return ret;
}Enclave code
The enclave code will be the core of the KMS. To get it started, we'll first write the following C/C++ code snippet that includes several header files and defines two character pointers certificate and private_key.
Open the enclave.cpp file in the enclave repository and start by importing the necessary headers for the functions:
- Openenclave/enclave.h: contains Open Enclave SDK APIs for creating and managing Enclaves.
- stdlib.h, string.h, sys/socket.h, arpa/inet.h, errno.h, netinet/in.h, stdarg.h, stdbool.h, stdio.h, unistd.h, netdb.h, sys/types.h, fcntl.h, and sys/epoll.h: standard C/C++ libraries and headers for working with sockets, files, and I/O operations, as well as error handling and formatting.
We'll also need the C++ files aes_genkey.cpp and rsa_genkey.cpp which will hold our future KMS functions for key generation. Let's create them:
$ touch aes_genkey.cpp rsa_genkey.cpp
// enclave.cpp
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
#include <unistd.h>
#include <netdb.h>
#include <sys/types.h>
#include <sys/epoll.h>
#include <fcntl.h>
#include "../mongoose/mongoose.h"
#include "kms_t.h"
#include "trace.h"
#include "generation/aes_genkey.cpp"
#include "generation/rsa_genkey.cpp"
Loading the OpenEnclave modules
While it is necessary to link the network components libraries at compilation time, we still need to load them. The function load_oe_modules serves that purpose by calling the loading function for each feature:
// enclave.cpp
oe_result_t load_oe_modules()
{
oe_result_t result = OE_FAILURE;
// host resolver
if ((result = oe_load_module_host_resolver()) != OE_OK)
{
printf(
"oe_load_module_host_resolver failed with %s\n",
oe_result_str(result));
goto exit;
}
// sockets
if ((result = oe_load_module_host_socket_interface()) != OE_OK)
{
printf(
"oe_load_module_host_socket_interface failed with %s\n",
oe_result_str(result));
goto exit;
}
// epoll
if ((result = oe_load_module_host_epoll())!=OE_OK) {
printf(
"oe_load_module_host_epoll failed with %s\n",
oe_result_str(result));
goto exit;
}
exit:
return result;
}
HTTPs server
- To get started on the HTTPS server, we will need to generate a TLS certificate, handle incoming requests and configure the listening server.
- Generate a TLS certificate: To use HTTPS, we first need to generate a TLS (Transport Layer Security) certificate.
- To generate a self-signed certificate and the private key associated with it, we use the following command with OpenSSL:
$ openssl req -new -x509 -key private.key -out certificate.crt -days 3650
The certificate and private_key character pointers contain example values for an X.509 digital certificate and private key, respectively. These values are used as test data and should be replaced with appropriate values for a given use case.
This will generate a self-signed certificate that is valid for 10 years (3650 days). However, self-signed certificates are not trusted by default, so we will need to manually install the certificate on any client that needs to communicate with the server over HTTPS.
One way to do so is declare the certificate and private key directly as a constant variable.
⚠️ Warning
As it is an HTTPs server for testing purposes, the certificate and private key can be imported, or in our case, copied into variables. In a production environment, the certificate and private key must be protected and stored securely.
// enclave.cpp
const char* certificate = "-----BEGIN CERTIFICATE-----\n" \
"...\n" \
"-----END CERTIFICATE-----";
const char* private_key = "-----BEGIN PRIVATE KEY-----\n" \
"...\n" \
"-----END PRIVATE KEY-----";
- Handle incoming requests: That goes by implementing a request handler that will be requested when the server is up and listening.
Each request route (/generate-aes-key, /generate-rsa-key-pair...) performs a certain KMS operation:
// enclave.cpp
static void api(struct mg_connection *c, int ev, void *ev_data, void *fn_data)
{
if (ev == MG_EV_ACCEPT && fn_data != NULL)
{
struct mg_tls_opts opts = {
.cert = CERTIFICATE,
.certkey = PRIVATE_KEY,
};
mg_tls_init(c, &opts);
} else if (ev == MG_EV_HTTP_MSG) {
struct mg_http_message *hm = (struct mg_http_message *) ev_data;
if (mg_http_match_uri(hm, "/generate-aes-key")) {
unsigned char *key;
key = (unsigned char*)malloc( 32 * sizeof(unsigned char) );
generate_aes_key(key);
TRACE_ENCLAVE("key is equal to : {%s}", key);
mg_http_reply(c, 200, "Content-Type: application/json\r\n", "{%m: \"%s\", %m: \"%s\"}\r\n", mg_print_esc, 0, "aes_key",
key, mg_print_esc, 0, "encoding", "base64");
key = NULL;
free(key);
} else if (mg_http_match_uri(hm, "/generate-rsa-key-pair")) {
unsigned char *rsa_public_key = NULL;
rsa_public_key = (unsigned char*)malloc( 2048 * sizeof(unsigned char) );
unsigned char *rsa_private_key = NULL;
rsa_private_key = (unsigned char*)malloc( 2048 * sizeof(unsigned char) );
generate_rsa_keypair(rsa_public_key, rsa_private_key);
TRACE_ENCLAVE("key pointer is equal to : {%p}", &rsa_public_key);
TRACE_ENCLAVE("public key is equal to : {%s}", rsa_public_key);
// TRACE_ENCLAVE("private key is equal to : {%s}", rsa_private_key);
TRACE_ENCLAVE("private key is equal to : {%s}", rsa_private_key);
mg_http_reply(c, 200, "Content-Type: application/json\r\n", "{%m: \"%s\", %m: \"%s\", %m: \"%s\"}\r\n", mg_print_esc, 0, "public_key",
rsa_public_key, mg_print_esc, 0, "private_key",
rsa_private_key, mg_print_esc, 0, "encoding", "plaintext");
// freeing the variables
rsa_public_key = NULL;
free(rsa_public_key);
rsa_private_key = NULL;
free(rsa_public_key);
}
else {
TRACE_ENCLAVE("request");
mg_http_reply(c, 200, "", "{\"result\": \"%.*s\"}\n", (int) hm->uri.len,
hm->uri.ptr);
}
}
(void) fn_data;
}
- Configuring the listening server: We'll run the Mongoose web server with the certificate and the private key we defined earlier.
Our web server also defines the Ecall that we will be calling to:
// enclave.cpp
int set_up_server(const char* server_port_untrusted, bool keep_server_up )
{
TRACE_ENCLAVE("Entering enclave.\n");
TRACE_ENCLAVE("Modules loading...\n");
if (load_oe_modules() != OE_OK)
{
printf("loading required Open Enclave modules failed\n");
return -1;
}
TRACE_ENCLAVE("Modules loaded successfully.\n");
char listening_addr[21];
strncat(listening_addr,"https://0.0.0.0:");
strncat(listening_addr, server_port_untrusted);
struct mg_mgr mgr;
mg_mgr_init(&mgr); // Init manager
mg_http_listen(&mgr, listening_addr, api, &mgr);
// Setup listener
TRACE_ENCLAVE("Listening at %s.\n", listening_addr);
for (;;) mg_mgr_poll(&mgr, 1000); // Event loop
mg_mgr_free(&mgr);
return 1;
}Having all of this set up, we can go ahead and do some testing:
# to run the server and enclave
$ make all && make run
# to test respond with simple requests
$ curl -k https://127.0.0.1:9000/
You should get a status json response.
Functions implementation
In OpenEnclave, the oe_random() function is used to generate random numbers. This function uses the RDRAND instruction, if available, to generate entropy from the processor's hardware random number generator (RNG).
If the RDRAND instruction is not available, the function uses the operating system's random number generator.
The oe_random() function is used to generate keys, nonces, and other random data in OpenEnclave. Mbedtls should normally calls to RDRAND instruction when calling to a random number generator.
AES generation key
While generating an AES 256 bits key, a strong entropy source and seed the DRBG with sufficient entropy to ensure that the generated key must be used. This gives you the property of a cryptographically secure generated key.
// enclave/generation/aes_genkey.cpp
#include <stdio.h>
#include <string.h>
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/base64.h"
#include "../trace.h"
void generate_aes_key(unsigned char* key_base64)
{
mbedtls_ctr_drbg_context ctr_drbg;
mbedtls_entropy_context entropy;
char *personalized = "aes gen key miniKMS";
int ret;
unsigned char key[32]; // variable containing the AES Key
mbedtls_ctr_drbg_init( &ctr_drbg ); //Initializing DRBG and entropy
mbedtls_entropy_init( &entropy );
if ( ( ret = mbedtls_ctr_drbg_seed( &ctr_drbg, mbedtls_entropy_func, &entropy,
(unsigned char *)personalized, strlen(personalized) ) ) != 0 )
{
TRACE_ENCLAVE("Failed ! mbedtls_ctr_drbg_seed returned with -0x%04x", -ret);
}
if ( ( ret = mbedtls_ctr_drbg_random(&ctr_drbg, key, 32) ) != 0 )
{
TRACE_ENCLAVE("Failed ! mbedtls_ctr_drbg_random returned with -0x%04x", -ret);
}
size_t outlen;
if ( ( ret = mbedtls_base64_encode(key_base64, 256, &outlen, key, 32*sizeof(unsigned char))) != 0 )
{
TRACE_ENCLAVE("Failed ! mbedtls_base64_encode returned with -0x%04x", -ret);
}
}
// enclave/generation/rsa_genkey.cpp
#include "mbedtls/config.h"
#include "mbedtls/platform.h"
#include <stdio.h>
#include <stdlib.h>
#define mbedtls_exit exit
#define MBEDTLS_EXIT_SUCCESS EXIT_SUCCESS
#define MBEDTLS_EXIT_FAILURE EXIT_FAILURE
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/bignum.h"
#include "mbedtls/x509.h"
#include "mbedtls/rsa.h"
#include "../trace.h"
#include <stdio.h>
#include <string.h>
#define KEY_SIZE 2048
#define EXPONENT 65537
void generate_rsa_keypair(unsigned char* public_key, unsigned char* private_key)
{
mbedtls_pk_context key;
mbedtls_entropy_context entropy;
mbedtls_ctr_drbg_context ctr_drbg;
const char *pers = "rsa_keygen";
int ret;
// Initialize contexts
mbedtls_pk_init(&key);
mbedtls_entropy_init(&entropy);
mbedtls_ctr_drbg_init(&ctr_drbg);
// Seed the random number generator
if ((ret = mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy,
(const unsigned char *) pers, strlen(pers))) != 0) {
TRACE_ENCLAVE("Failed to seed the random number generator: %d\n", ret);
}
TRACE_ENCLAVE( "Setting up the context...\n" );
// Generate the RSA key pair
if ((ret = mbedtls_pk_setup(&key, mbedtls_pk_info_from_type(MBEDTLS_PK_RSA))) != 0) {
TRACE_ENCLAVE("Failed to set up the PK context: %d\n", ret);
}
TRACE_ENCLAVE( "Generating the RSA key [ %d-bit ]...\n", KEY_SIZE );
if ((ret = mbedtls_rsa_gen_key(mbedtls_pk_rsa(key), mbedtls_ctr_drbg_random, &ctr_drbg, KEY_SIZE, 65537)) != 0) {
TRACE_ENCLAVE("Failed to generate the RSA key pair: %d\n", ret);
}
// Print the public and private keys in PEM format
mbedtls_pk_write_pubkey_pem(&key, public_key, 2048 * sizeof(unsigned char));
mbedtls_pk_write_key_pem(&key, private_key, 2048 * sizeof(unsigned char));
mbedtls_pk_free(&key);
mbedtls_ctr_drbg_free(&ctr_drbg);
mbedtls_entropy_free(&entropy);
}Now that we've seen how to run an enclave and how to interact with it, let's see how to securely establish the connection and implement a remote proof that we are using the right application in the right environment!
Next
