This is the third post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at the design of the virtio-serial device and how to decompose it into modules.
Series table of contents
- Part 1: Overview
- Part 2 - MMIO registers, DMA, and interrupts
- Part 3 - virtio-serial device design (you are here)
- Part 4 - Virtqueue processing
- Part 5 - UART receiver and transmitter
- Part 6 - Writing the RISC-V firmware
The code is available at https://gitlab.com/stefanha/virtio-serial-fpga.
A virtio-serial device is a serial controller, enabling communication with the outside world. The iCESugar FPGA development board has UART rx and tx pins connecting the FPGA to a separate microcontroller that acts as a bridge for USB serial communication. That means the FPGA can wiggle the bits on the UART tx pin to send bytes to a computer connected to the board via USB and you can receive bits from the computer through the UART rx pin. The purpose of the virtio-serial device is to present a VIRTIO device to the PicoRV32 RISC-V CPU inside the FPGA so the software on the CPU can send and receive data.
Device design
The virtio-serial device implements the Console device type defined in the VIRTIO specification and exposes it to the driver running on the CPU via the VIRTIO MMIO Transport. The terms "serial" and "console" are used interchangeably in the VIRTIO community and I will usually use serial unless I'm specifically talking about the Console device type section in the VIRTIO specification.
VIRTIO separates the concept of a device type (like net, block, or console) from the transport that allows the driver to access the device. This architecture allows VIRTIO to be used across a range of different machines, including machines that have a PCI bus, MMIO devices, and so on. Fortunately the VIRTIO MMIO transport is fairly easy to implement from scratch.
The virtio_serial_mmio module implements the virtio-serial device from the following parts:
- VIRTIO MMIO Transport - MMIO device registers conforming to the VIRTIO specification. They allow the CPU to configure the device and initiate data transfers.
- UART reader & virtqueue writer - Incoming data from the UART rx pin is enqueued on the VIRTIO Console receiveq (virtqueue 0) where the driver can receive it.
- Virtqueue reader & UART writer - The VIRTIO Console transmitq (virtqueue 1) lets the driver enqueue data that the device sends over the UART tx pin.
The virtio-serial device interfaces with the outside world through an MMIO interface that the CPU uses to access the device registers, a DMA interface for initiating RAM memory transfers, and the UART rx/tx pins for actually sending and receive data.
Note that both the virtqueue_reader and the virtqueue_writer modules require DMA access, so I reused the spram_mux module that multiplexes the CPU and the virtio-serial device's RAM accesses. spram_mux is used inside virtio_serial_mmio to multiplex access to the single DMA interface.
Reader and writer interfaces
Since the job of the device is to transfer data between the virtqueues and the UART rx/tx pins, it is organized around a module named rdwr_stream that constantly attempts to read data from a source and write it to a destination:
/* Stream data from a reader to a writer */
module rdwr_stream (
input clk,
input resetn,
/* The reader interface */
output reg rd_phase = 0,
input [31:0] rd_data,
input [2:0] rd_data_len,
input rd_ready,
/* The writer interface */
output reg wr_phase = 0,
output reg [31:0] wr_data = 0,
output reg [2:0] wr_data_len = 0,
input wr_ready
);
By implementing the reader and writer interfaces for the virtqueues and UART rx/tx pins, it becomes possible to pump data between them using rdwr_stream. For testing it's also possible to configure virtqueue loopback or UART loopback so that the virtqueue logic or the UART logic can be exercised in isolation.
The reader and writer interfaces that the rdwr_stream module uses are the central abstraction in the virtio-serial device. You might notice that this interface uses a phase bit rather than a valid bit like in the valid/ready interface for MMIO and DMA. Every transfer is initiated by flipping the phase bit from its previous value. I find the phase bit approach easier to work with because it distinguishes back-to-back transfers, whereas interfaces that allow the valid bit to stay 1 for back-to-back transfers are harder to debug. It would be possible to switch to a valid/ready interface though.
To summarize, there are 4 reader or writer implementations that can be connected freely through the rdwr_stream module:
- virtqueue_reader - reads buffers from the transmitq virtqueue (virtqueue 1).
- virtqueue_writer - writes buffers to the receiveq virtqueue (virtqueue 0).
- uart_reader - reads data from the UART rx pin.
- uart_writer - writes data to the UART tx pin.
Conclusion
The virtio-serial device consists of the VIRTIO MMIO Transport device registers plus two rdwr_streams that transfer data between virtqueues and the UART. The next post will look at how virtqueue processing works.
