Driver stack and boot tooling for the Cavium CN6640-SNIC10E
(OCTEON II CN6640, PCI 177d:0092), exposing the card as two independent
10 GbE host interfaces (oct0, oct1) over a
reverse-engineered PCIe BAR2 shared-memory datapath. No vendor NDA firmware. It works on
Linux and Windows today, and FreeBSD is in progress.
Status
| OS | Driver | State |
|---|---|---|
| Linux | octnic kernel module (DKMS) |
Working, hardware-validated. Both ports link at 10 G and pass traffic. |
| Windows 10/11 x64 | octnic NDIS 6.20 miniport, test-signed |
Working. Tested on real hardware with Windows 11 24H2: DHCP, ping and internet through the card. Not benchmarked yet. |
| FreeBSD | planned if_octnic |
Work in progress. Nothing runs yet. |
On Linux both ports link at 10 G on a DAC and pass traffic bidirectionally: TX at line rate 10 GbE (9.7 to 9.8 Gb/s, zero-copy datapath) and RX at 8.1 to 8.8 Gb/s. The card boots with no serial cable and the host brings both NICs up automatically at boot. The numbers and how they were measured are on the Linux page.
How it works
The card runs OpenWrt from RAM. The host pushes the image into the card's DRAM over PCIe at every boot, so nothing is flashed, and the one permanent change (the card's u-boot environment) is reversible. The host driver maps the card's BAR2 window, which holds one shared-memory ring set per port: RX is DMA'd by the card's DPI engine, TX is written by the host with PIO.
The protocol is the same on every OS, so the card side is shared and each OS only needs its own host driver. How it was built, and every dead end on the way, is in the write-up.
Requirements
- The card in a PCIe slot, with Secure Boot off. On Linux, kernel lockdown
blocks the
setpciand BAR access this stack needs. On Windows, test-signing cannot be enabled with Secure Boot on. - The prebuilt card image from the releases (the same file for every OS), or an OpenWrt build tree to build it yourself.
- A USB-serial (FT232) adapter for the one-time u-boot provisioning only. Normal operation is serial-free.
- Per OS: a modern Linux with kernel headers, or Windows 10/11 x64. Details on each page.
Install
There is one guided installer per OS. It asks for confirmation before every step that changes something.
| OS | Command |
|---|---|
| Linux | sudo ./install.sh |
| Windows | install.bat (double-click, it asks for Administrator) |
| FreeBSD | not implemented: the installer cancels with a message |
What the installer does:
- Checks that the card is on the PCI bus (
177d:0092). If not, it stops, or asks whether to install the driver anyway. - Installs the driver for your OS.
- Checks whether the card is already provisioned for host boot, that is, whether its u-boot environment was written once. If it is, no serial cable is needed.
- If not, it looks for the USB-serial adapter, tests the link and, only after you confirm,
writes the u-boot environment (
saveenvto NAND, once, reversible withscripts/restore-bootapp.sh). If there is no adapter, it stops and tells you; the driver stays installed and nothing is written to the card. - Boots the card and brings up
oct0andoct1.
Documentation
The reference documentation lives in the repository:
| Page | Read it when |
|---|---|
| FLASHING | Building the modules and the card image, the one-time serial u-boot provisioning, and booting the card with octboot. |
| USAGE | Day-to-day operation: the systemd autostart, octnic module parameters, temperatures and power in sensors, benchmarking, troubleshooting. |
| HARDWARE | The board itself: SoC, PHY, PCIe BARs, the BAR0 window freeze hazard, serial pinout, XAUI ipd_port mapping. |
| PERFORMANCE | Measured throughput, the tuning that got there, and how to reproduce the numbers. |
| ARCHITECTURE | The datapath end to end: the BAR2 shared-memory rings, the card and host modules, the boot flow. |
| DMA-DESIGN | Why RX is card-mastered DPI DMA and TX is host PIO, and the constraints behind that split. |
| winmod/README | The Windows driver: build, installer, signing, the boot task and its logs. |
| bsdmod/README | The plan for the BSD driver. |
FAQ
Does this need vendor firmware or an NDA?
No. Nothing from Cavium's LiquidIO firmware is in the stack. The card runs OpenWrt pushed into its DRAM over PCIe at every boot, and the host talks to it through a shared-memory window the card opens itself.
Is it reversible? What does it write to the card?
One thing, once: the card's u-boot environment, saveenv'd to NAND so the host
can boot it without a serial cable. scripts/restore-bootapp.sh puts the stock
environment back. The OS itself is never flashed: it lives in RAM and is gone at power-off.
Removing the host driver and the autostart is the rest of the rollback.
Which operating systems work?
Linux and Windows 10/11 x64. The Linux driver is hardware-validated; the Windows driver was tested on Windows 11 24H2 and is test-signed, so it needs Secure Boot off. FreeBSD is in progress and nothing runs there yet. See the pages for Linux, Windows and FreeBSD.
Can Linux and Windows share the card on a dual-boot machine?
Yes. On both OSes the boot step does nothing if the card OS is already running, so the two never fight over the card.
Why didn't the stock liquidio driver work?
Its soft-reset plus flash-handshake boot model doesn't fit this OEM board, and out of the box
both PCIe BARs read back all 0xFF. That "BAR wall" is unprogrammed PEM inbound
registers, not a hardware limit: programming them from the card side opens the channel. On
Linux, keep liquidio blacklisted, because it hangs the host probing this board.
What packets-per-second can it do?
At MTU 9000, line-rate TX is roughly 134 k packets/s. Small-frame (64 B) pps is not measured yet, and it is likely the real ceiling of the card: both bottlenecks found while tuning were per-frame CPU costs, which is exactly what small packets stress.
How much power does it draw? Does ASPM work?
The card (estimate) value in sensors is a model
(baseline + per-Gbit/s), not a measurement: the board has no power sensor.
Calibrate the two coefficients against a wall meter if you need a real figure. ASPM is
untested and unlikely to work out of the box.
Can I use the OCTEON's crypto acceleration?
Not through this datapath. The OCTEON II has hardware crypto/HFA, but nothing here wires it up: the path is plain PCIe DMA.
Can the card run standalone, without a host?
That's the next thing to explore: powered on its own, the board is a two-port 10 GbE SFP+ router running OpenWrt, with its own 8-core CPU and 2 GiB of RAM.
Something wedged, what now?
A card that has been driven hard can wedge outright, and because it is bus-powered, recovery is a host reboot. On Linux a single port that stops receiving can often be fixed without one; see Linux troubleshooting.
Questions, or testing on your own board?
Open an issue or a discussion on the repository, or email me@nicologiuliani.site. Results from other boards are especially welcome: everything here was developed against a single card.