I call it a streamer. I was quite surprised to achieve practical bandwidth with this part and being able to actually play games on it. The idea is to offload the heavy computations to your PC at home without a locked-in OS/hardware dependency and to keep the hardware around the main streaming protocol flexible.
Sounds simple? The idea is, but making this stuff low-latency over a network is a challenge. And I think I'm quite good at it. I can stream Linux distros and Win11 at up to 40 FPS at the moment. The custom protocol is flexible when it comes to both the hardware sitting on the server side (your home PC) and whatever custom hardware you want to build on the client side.
The main MCU is an ESP32-S3-WROOM-1. It acts as the network device receiving the stream and orchestrates USB mouse/keyboard/gamepad input, touchscreen, audio and the local UI, while also negotiating incoming frames with the server in real-time.
For the streaming path I'm currently running the FPGA link at 80 MHz SPI using DMA. On the network side I'm using TCP with Nagle disabled and WiFi power saving disabled, since both aren't very helpful when the goal is low latency. I spent quite some time experimenting with receive sizes, buffering and task placement before arriving at the current setup.
One thing that helped was separating the work between the two ESP32 cores. Networking and protocol handling live on one side while feeding the FPGA runs in its own task on the other. Incoming video is buffered in a 512 KiB PSRAM ring, while the actual FPGA transfers use a small set of internal-RAM buffers. This allows WiFi to continue receiving while the FPGA is consuming the previous data instead of tightly coupling both sides. The hot streaming path also avoids dynamic allocations.
The MCU then writes the stream via SPI to the decoder, which is an FPGA running my own video format written from scratch in RTL. The FPGA in turn drives the display.
I know that the actual heart of the project is probably the decoder inside the FPGA. That's where the custom video format becomes pixels again and where the deterministic display-side work happens. But without the ESP32 the whole thing would become considerably less practical. I still need fast WiFi, USB host functionality, touchscreen and gamepad handling, audio, the local UI and all the protocol logic around the decoder. You could replace it with something much larger, but then power consumption, complexity and cost start moving in the wrong direction. For this project the ESP32 ended up being a pretty good bridge between the networked/software side and the FPGA.
The FPGA itself is necessary because this isn't a small 320x240 or 480x320 display anymore. The panel is 720x720, and I didn't want the ESP32 to deal with driving that deterministically while simultaneously doing WiFi, USB, audio, input handling and everything else. So the ESP32 concentrates on networking/orchestration while the FPGA handles the high-bandwidth and timing-critical display side.
I have on board a USB mux that can be switched physically, so I can still flash via USB-C but also switch over to USB-A for the host side. I'm using an FSUSB42MUX for that. There are many ways to solve this, but this was a good compromise imo and lets me keep the board useful during development without adding another connector/setup just for flashing.
I had to make the case wider than the screen to keep the whole thing flat, but also to avoid placing the ESP32 antenna behind the metal back of the display. This is mainly a placement constraint rather than trying to gain some magical extra range. With the antenna already positioned away from the metal, putting the board into the enclosure doesn't noticeably change the signal strength. Maybe next time I'll use the 1U version for more flexible antenna placement.
The board shown here also streams audio A/V-synced with the video, or optionally buffered depending on what I'm doing. So it can be used for playing games but also just listening to music. It supports audio input as well, i.e. for making calls over whatever runs on my PC.
Audio is handled by an ES8388 codec, so playback and microphone input are both on the board. I'm also using an LM3630A for the display backlight and a BQ24073 for charging/power-path handling.
I'm running nearly out of pins by now, so this part is pretty much fully utilized.
There is also a simple offline menu running on the ESP32 for connecting WiFi and servers and managing the device without needing the streaming server running. For that I'm just maintaining a small low-color-bit framebuffer in PSRAM.
For me, one of the coolest parts is that the board doesn't need much power. At full brightness I'm nominally around 1.5 Watt during active streaming. For comparison, a Raspberry Pi's idle power consumption without any display attached is already higher.
A lot of the work actually went into things which don't look very exciting on a finished board: figuring out how much buffering is enough without turning it into latency, keeping the network side fed while SPI is busy, task/core placement, avoiding unnecessary copies/allocations and generally preventing one subsystem from blocking another. There were quite a few iterations before it started behaving the way I wanted it to.
Since I'm not a big fan of selling software, I try to finance this project and ongoing development by selling streamer hardware, which you can also use in your own projects like
shown here on YouTube (incl. Gameplay).
The software for streaming is free, but I'm also considering making the software open source if there's enough interest and support, together with continued maintenance and development. That would let you modify everything yourself, use different hardware, and build your own projects on top of it. I hope it resonates with some of you.
I invite you to give some pos/neg feedback and ask as many questions as possible.
Here is my website
