Extending the VeeR EL2 RISC-V CPU core with User mode for Caliptra 2.0

In recent years Antmicro has been working for customers within the CHIPS Alliance’s Caliptra Workgroup, led by AMD, Google, Microsoft and NVIDIA, to maintain and gradually enhance the RISC-V VeeR EL2 CPU core that is used in the open source Caliptra Root of Trust project. Some time ago we introduced Physical Memory Protection support, and our next goal was adding the RISC-V User privilege mode, on top of the Machine (M) mode VeeR EL2 supported originally.

In this article, we describe how Antmicro implemented U-mode support in the VeeR EL2 core, what this enables and the software and simulation infrastructure that was used to verify U-mode support including Verilator, RISCV-DV, Tock and Renode.

Implementing U-mode support in VeeR EL2

The RISC-V architecture provides three software privilege levels: User mode (U-mode), Supervisor mode (S-mode), and Machine mode (M-mode), only one of which can be active in the CPU at a given time.

RISC-V associates specific privilege levels with different sets of Control Status Registers (CSRs). This prevents system-critical registers from being tampered with by malicious code running in lower privilege levels. In U-mode, software cannot access certain hardware resources (e.g. read/write CPU’s Control and CSRs, handle exceptions) or modify PMP settings. With the addition of User mode, we included logic that enforces CSR access privileges to prevent such actions.

Adding U-mode support to the VeeR EL2 CPU core, in compliance with the RISC-V Privileged spec, required implementing changes in various areas.

First of all, we added the logic required to store the current operating mode and switch between modes. This includes correct handling of exceptions and continuing execution after returning from executed subroutines. Since some RISC-V instructions are privileged and can only be executed from Machine mode, we’ve implemented instruction privilege enforcement.

We also implemented extensions of the standard RISC-V CSRs already present in the EL2 core, including setting the U bit in the misa CSR which contains information about features supported by the core. This indicates the presence of User mode support for the running software. The mstatus CSR was extended with the MPP and MPEV fields that indicate the previous operating mode of the core (e.g. after entering an exception handler) and the effective privilege mode for load/store access checking respectively.

New CSRs have been added that are either required or applicable in User mode (see the CSR read multiplexer code), including the menvcfg/menvcfgh CSRs that contain bits that control the behavior of particular FENCE instruction flavors, along with bits relevant to Sstc, Zicboz and Zicbom RISC-V extensions. To allow accessing performance counters from User mode, shadow CSRs are now utilized along with the mcounteren CSR which is used to dynamically restrict performance counter access. Additionally, we’ve introduced Physical Memory Protection (PMP) related mseccfg/mseccfgh CSRs that control the extended PMP (Smepmp) behavior if it is enabled in VeeR’s configuration.

All the changes described above have been included in the main pull request. It also includes software tests that exercise U-mode features. For example, the following program verifies if we can switch between User and Machine mode:

// == Code section ==
.section .text.init

// Entry point
.align 4
.global _start
_start:
    la      sp, STACK           // Setup stack
    la      t0, _trap           // Set trap handler
    csrw    mtvec, t0

    ecall                       // Issue an ECALL from M-mode

    csrr    t0, mstatus         // Tell the core to switch to U-mode
    li      t1, 0xFFFDE7FF
    and     t0, t0, t1
    csrw    mstatus, t0

    la      t0, _user           // Jump to the U-mode entry point
    csrw    mepc, t0
    mret

// User mode entry point
.align 4
_user:
    ecall                       // Issue an ECALL from U mode
    j .                         // Loop indefinitely

// Trap handler
.align 4
_trap:

    csrr    t0, mcause          // Read exception cause

    li      t1, 0x8             // mcause == 8, we got called from M-mode
    bne     t0, t1, _S0
    la      a0, string_user
    jal     puts
_S0:

    li      t1, 0xB             // mcause == 11, we got called from U-mode
    bne     t0, t1, _S1
    la      a0, string_mach
    jal     puts
_S1:

    csrr    t0, mepc            // Return from the exception handler
    addi    t0, t0, 4
    csrw    mepc, t0
    mret

// puts() implementation
puts:
    li      t0, 0xD0580000
_L: lb      t1, 0(a0)
    sb      t1, 0(t0)
    addi    a0, a0, 1
    bnez    t1, _L
    ret

// == Data section ==
.section .data

string_mach:
.ascii "Hello from machine mode\n"
.byte 0

string_user:
.ascii "Hello from user mode\n"
.byte 0

By default, the VeeR EL2 Core is configured for Machine mode only, so User mode needs to be explicitly enabled in the CPU configuration. To provide greater flexibility, we separated the U-mode logic, so that no additional resources are used if the M-mode only variant is selected.

PMP and Extended PMP (Smemp) in VeeR EL2 U-mode

The RISC-V Privileged spec section 3.7 defines an optional Physical Memory Protection (PMP) unit, the support for which in the EL2 core has been described in a dedicated blog article. PMP is responsible for controlling core memory access for instructions fetching as well as data load and store.

The Machine mode is a highly-privileged mode for the operating system, with access to all registers and memory locations, unless explicitly blocked with an additional lock. In U-mode, however, a given memory region can only be modified if the access has been explicitly granted. For RISC-V platforms, the M-mode is the only mandatory privilege level.

In our EL2 implementation, when no memory region is configured for protection, PMP grants access to the entire memory from both Machine and User mode. When at least one region is configured, the access to the entire memory outside that region from User mode is denied by default. When a region is not locked, its permissions are enforced only for User mode, while Machine mode has full access. When a region is locked, its permissions are enforced regardless of the operating privileged mode. Locking a region freezes its configuration, and related CSR fields cannot be written anymore, even from Machine mode. The only method to unlock a region is to reset the core.

To provide greater flexibility in memory access control, Antmicro has now enhanced VeeR’s PMP by implementing the extended PMP according to the Smepmp specification, in addition to supporting User mode. The main goal of this enhancement is to provide a mechanism enabling a PMP rule to be enforced only in non-Machine modes and denied in Machine mode, to only allow access to a memory region by less-privileged modes. It is particularly useful in RISC-V architecture implementations where a hart has only Machine and User modes available, as is the case for VeeR, so the whole OS will run in Machine mode instead of the non-existent Supervisor mode.

In such implementations the attack surface is greatly increased, and the same kind of attacks performed on Supervisor mode and mitigated through SMAP (Supervisor Mode Access Prevention) / SMEP (Supervisor Mode Execution Prevention) can be performed in Machine mode without any available mitigations. Thanks to the Smepmp extension, it is possible to prevent attacks against the Firmware and/or the Secure Monitor running in Machine mode.

Support for this extension is optional and can be changed in the veer.config script. To enable User mode and Smepmp in the VeeR core, the script needs to be called with the following arguments:

veer.config -set=user_mode=1 -set=smepmp=1

Testing and verifying the implementation

Beside the CPU work itself, Antmicro also implemented a number of software tests for verifying U-mode’s functionality, as well as testing the separation between M and U-mode. The tests are implemented using mixed C and assembly code to provide fine-grained control over specific instructions being executed. They exercise features related to CSR access, exception handling and PMP operation. The tests are run in RTL-level simulation using Verilator. For tests that target external interrupt handling, side-channel communication between the test and the simulation environment is utilized to control interrupt signals of the core.

To further extend the verification suite of the core, we employed the RISCV-DV framework which is an open source SV/UVM based instruction generator for RISC-V processors. The framework is used to generate programs for the RISC-V architecture, which are constrained to use some selected features of the platform, such as PMP or U-mode. These programs are executed in RTL simulation of the VeeR EL2 core and several RISC-V ISS simulators (such as Spike or Renode) and upon completion, the core states of both are compared after each executed instruction. The test passes if all executions yield identical results. The described flow, including the generation of the programs, is part of the public GH Actions workflow for the Cores-VeeR-EL2 repository.

Additionally, we ported the Tock operating system to work on the VeeR EL2 core with PMP and User mode enabled. Tock makes it easy to run both C and Rust applications on VeeR EL2, which helps to perform more complex testing. The repository provides a reproducible scenario which lets anyone run custom applications on VeeR EL2 in simulation using Verilator and Renode, Antmicro’s open source instruction set simulator.

For more information about the EL2 core, its architecture and testing infrastructure, refer to the project’s newly updated documentation.

Advanced RISC-V-based designs with Antmicro

Antmicro’s comprehensive engineering services for customizing and enhancing RISC-V cores allow us to assist our customers in a variety of use cases, from replacing legacy cores with RISC-V, through developing custom accelerators, to embedding existing cores and augmenting them for specific use cases like Caliptra.

On top of RTL development, we work on improving the verification coverage of RISC-V CPUs through CI-based code quality checks as well as verification and testing pipelines, visualized in custom dashboards.

Leveraging our own, constantly evolving open source ecosystem and the numerous collaborations that help us drive the ASIC design landscape forward, Antmicro provides comprehensive services for modifying, adapting and maintaining RISC-V cores. As members of CHIPS Alliance and RISC-V International, Antmicro is also continuously working on enhancing open source ASIC/FPGA design tools - including enabling open source UVM support in Verilator.

Whether you need to create a new solution from scratch or customize an existing design, Antmicro’s approach enables easier debugging, testing and collaboration at every stage of the development process. To learn more about our engineering services, reach out to us at contact@antmicro.com.