Supporting CHERI capabilities in GCC and glibc

CHERI

Coplan started by describing CHERI as a research project that has been running for a decade or so, based at Cambridge University. It introduces the concept of "capabilities", a term which has a specific meaning in this context. A capability, Coplan said, is a token of authority to access a range of memory that cannot be forged. Capabilities can be passed around, and they can be derived from other capabilities, but any such derivation can only narrow the access permissions that capability provides; they can never be expanded.

Lots of details can be found in this overview of the CHERI architecture. In short, a capability can be thought of as a special type of pointer that occupies 129 bits. The bottom 64 bits are a conventional virtual address, while the upper 64 bits (often referred to as the "provenance") describe the associated access permissions. They include a bitmask of allowed operations (read, write, execute) and the range of memory to which the capability applies. Together, those 128 bits can be used like a pointer to perform the allowed accesses on the allowed range of memory.

The 129th bit is stored elsewhere and managed by the CPU; the capability is only valid if that bit is set. There are CPU instructions to derive a new capability from an old one that will keep the bit set; any direct changes to the capability, instead, will cause the validity bit to be cleared. Other disallowed operations, such as trying to use a capability to write outside of the allowed range of memory, will also invalidate the capability. If the hardware works as designed, a capability provides the right to access a range of memory in certain ways — and nothing more.

When this system boots, the firmware provides the kernel with a capability allowing full access to all of memory. Every other capability used during the life of the system will be ultimately derived from this "root capability". In a well designed system, capabilities may go through several levels of derivation until they refer to narrowly constrained regions of memory.

Morello GCC

CHERI is meant to be implemented as an add-on to an existing architecture; the Morello project is adding CHERI capabilities to the ARMv8-A architecture. Prototype boards implementing this combination exist now. A mature LLVM-based toolchain already exists, but ARM wanted a GCC option as well, so the Morello GCC project is working to provide that option; this project is also porting binutils, GDB, and glibc.

There are two capability models, called "pure-cap" and "hybrid", that have been implemented; the former puts capabilities on all pointers, while the latter only uses them in parts of the system. The hybrid model allows mixing capability-aware code with non-aware code; a kernel port to the Morello architecture has been done using the hybrid mode.

Challenges are not in short supply when attempting a port of this nature. One of the first steps was to remap the intptr_t type (describing an integer that can hold a pointer value). The problem being solved there is, of course, that a normal long cannot hold capabilities; any attempt to use a pointer taken from a long value will trap at run time. Code that uses such types in that way will, thus, break on a CHERI system. Pointer comparison is a bit tricky; two capabilities pointing to the same location may have different access rights and thus different bit patterns, but a comparison should call them equal.

Despite these traps, Coplan said, most code will just work when compiled in the pure-cap mode. Low-level software, instead, can require a lot of changes, and code that plays around with pointers can be problematic. There is a sorting function in GCC, for example, that plays games with pointers, leading to errors from the compiler.

Teaching GCC about capabilities in general is a big challenge, requiring many changes throughout the code. They break two fundamental assumptions made within GCC: that pointers and integers are interchangeable, and that addresses and offsets are essentially the same thing. The compiler must ensure that all pointers have the correct provenance, that pointer comparisons are done correctly, and so on.

Work on GCC started by adding a -mfake-capability option that causes the compiler to always use capabilities in its internal representations, but to generate standard code from the back end as before. This allows the separation of capability concepts from the specifics of the Morello architecture. The developers worked to make everything work with this flag first, and only then set to the task of generating code that would use capabilities on real hardware.

One remaining problem is that the address-range bounds in the capability provenance are stored using a compressed, floating-point representation; otherwise that information would not fit in the allotted space. But, as a result, not all combinations of address, base, and limit can be represented. This problem mostly affects memory allocators, which must be able to operate over large ranges of memory, Coplan said.

Porting glibc

Nagy then took over to talk about the work that has been done to port glibc to the Morello architecture. This work is currently being done in a separate branch, with no plans to submit it upstream until a more clean patch set can be created. A lot of code changes to the library have been required; CHERI C is a different language, he said. Any code that manipulates pointers may need changes, for example.

More subtly, special tricks have to be done with basic functions like memcpy(). Any capabilities stored in the memory to be copied will not be valid at the destination unless special care is taken. Similar issues exist with pointers stored in shared memory or sent over sockets. syscall() must return an intptr_t type — an ABI change — to be able to successfully return pointers; similar changes are needed for other system calls as well.

Then there is the issue of capability derivation. When the kernel launches a process, that process is provided with a capability for the entirety of user space. It is possible to make things work with this capability, of course, and that is the first step, but it does not really take advantage of the capability mechanism. So the next step is to start narrowing capabilities in specific situations where access to all of memory is not needed.

A trickier problem is the dynamic linker, which does a lot of manipulation of 64-bit addresses. It is working with ELF binaries, though, so most of its work takes the form of "base plus offset" calculations. If the base value is a properly formed capability, things will work correctly. As the initial problems are overcome, the next phase is to start separating capabilities that allow writing from those that allow execution; that will require a lot more care to be taken in pointer derivation, he said.

Then there is the question of malloc(). Ideally, this function would return a pointer that can access the allocated object (and nothing else). In the first phase, this sort of narrowing wasn't done; the focus was on getting something working. The second phase then narrows the bounds of the returned pointer to just the object in question. That creates a problem for free(), though, which must be able to access metadata stored outside of the object itself; this is handled by maintaining a special capability for use by free(). There are various other challenges to making all of this work; he would really design the malloc() interface differently for CHERI, he said.

Currently the test suites run, he said, but only if stack bounds are not used in GCC. There are a number of missing features, including profiling, support for an executable stack, support for LD_AUDIT, and support for pointers stored in shared memory or sent via a file descriptor. Those latter cases may never be able to work, he concluded.

[Thanks to LWN subscribers for supporting my travel to this event].