Cavium CN6640-SNIC10E — two 10GbE ports from an OEM smart NIC

Out-of-tree Linux driver stack and boot tooling for the Cavium CN6640-SNIC10E (OCTEON II CN6640, PCI 177d:0092), exposing the card as two independent 10 GbE host interfaces (oct0, oct1) over a reverse-engineered PCIe BAR2 shared-memory datapath. No vendor NDA firmware.

Status: working. Both ports link at 10 G on a DAC and pass traffic bidirectionally — TX at line-rate 10 GbE (9.7–9.8 Gb/s, zero-copy datapath) and RX at 8.1–8.8 Gb/s. The card boots with no serial cable and the host brings both NICs up automatically at boot.

The card runs OpenWrt from RAM — nothing is flashed, and the one permanent change (the card's u-boot environment) is reversible. The host side is a kernel module plus scripts. How it was built, and every dead end on the way, is in the write-up.

Performance

port	host → peer	peer → host
`oct0` (xaui0)	9.71 Gb/s	8.10 Gb/s
`oct1` (xaui1)	9.81 Gb/s	8.82 Gb/s

iperf3 -P8, MTU 9000, fresh card boot, one port at a time. Both ports loaded in the same direction aggregate to ~10.5 Gb/s — the host-PIO 64 B-TLP wall on a Gen2 ×4 link. Full duplex is the open front: 13.5 Gb/s aggregate with UDP, but TCP suffers under the RX loss. Numbers, method and the tuning behind them: PERFORMANCE.

Requirements

The card in a PCIe slot, with Secure Boot off — kernel lockdown blocks the setpci and BAR access this stack needs. "Above 4G decoding" is not required.
A modern Linux host (developed on 6.14) with matching kernel headers; dkms recommended so the module survives kernel upgrades.
The prebuilt card image from the releases, or an OpenWrt build tree to build it yourself.
A USB-serial (FT232) adapter for the one-time u-boot provisioning only. Normal operation is serial-free.

Quick start

git clone https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic
cd cavium-cn6640-snic10e-octeon-ii-nic

# 1) the card's OS image, prebuilt — octboot looks for it here in the repo root
curl -LO https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic/releases/latest/download/openwrt-octeon-generic-snic10e-initramfs-kernel.bin

# 2) the host side: builds + installs octnic (DKMS), drops the configs, enables autostart
sudo ./install.sh

# 3) first time on this machine only — persist the card's u-boot env (serial cable, once)
sudo ./scripts/card-prep-hostboot.sh

# 4) every boot, no serial: pushes the image over PCIe, brings up oct0 + oct1
sudo systemctl start cavium-nic
ip -br addr show oct0

Step 4 is what the cavium-nic service runs at every host boot, so after step 3 the card comes up on its own. Steps 1–3 are once per machine. The manual equivalent is sudo ./octboot followed by sudo modprobe octnic ports=2; both ports are then ordinary netdevs.

Documentation

The reference documentation lives in the repository:

Page	Read it when
FLASHING	Building the modules and the card image, the one-time serial u-boot provisioning, and booting the card with `octboot`.
USAGE	Day-to-day operation: the systemd autostart, `octnic` module parameters, temperatures and power in `sensors`, benchmarking, troubleshooting.
HARDWARE	The board itself: SoC, PHY, PCIe BARs, the BAR0 window freeze hazard, serial pinout, XAUI `ipd_port` mapping.
PERFORMANCE	Measured throughput, the tuning that got there, and how to reproduce the numbers.
ARCHITECTURE	The datapath end to end: the BAR2 shared-memory rings, the card and host modules, the boot flow.
DMA-DESIGN	Why RX is card-mastered DPI DMA and TX is host PIO, and the constraints behind that split.

FAQ

Does this need vendor firmware or an NDA?

No. Nothing from Cavium's LiquidIO firmware is in the stack. The card runs OpenWrt pushed into its DRAM over PCIe at every boot, and the host talks to it through a shared-memory window the card opens itself.

Is it reversible? What does it write to the card?

One thing, once: the card's u-boot environment, saveenv'd to NAND so the host can boot it without a serial cable. scripts/restore-bootapp.sh puts the stock environment back. The OS itself is never flashed — it lives in RAM and is gone at power-off. Removing the host module and the service is the rest of the rollback.

Why didn't the stock `liquidio` driver work?

Its soft-reset plus flash-handshake boot model doesn't fit this OEM board, and out of the box both PCIe BARs read back all 0xFF. That "BAR wall" is unprogrammed PEM inbound registers, not a hardware limit — programming them from the card side opens the channel. Keep liquidio blacklisted: it hangs the host probing this board.

What packets-per-second can it do?

At MTU 9000, line-rate TX is roughly 134 k packets/s. Small-frame (64 B) pps is not measured yet, and it is likely the real ceiling of the card: both bottlenecks found while tuning were per-frame CPU costs, which is exactly what small packets stress.

How much power does it draw? Does ASPM work?

The card (estimate) value in sensors is a model (baseline + per-Gbit/s), not a measurement — the board has no power sensor. Calibrate the two coefficients against a wall meter if you need a real figure. ASPM is untested and unlikely to work out of the box.

Can I use the OCTEON's crypto acceleration?

Not through this datapath. The OCTEON II has hardware crypto/HFA, but nothing here wires it up — the path is plain PCIe DMA.

Can the card run standalone, without a host?

That's the next thing to explore: powered on its own, the board is a two-port 10 GbE SFP+ router running OpenWrt, with its own 8-core CPU and 2 GiB of RAM.

Something wedged — what now?

If one port stops receiving, reload the module (rmmod octnic && modprobe octnic ports=2) and re-add the IPs; the rings resync without rebooting the card. A card that has been driven hard can wedge outright, and because it is bus-powered, recovery is a host reboot. Details in USAGE → troubleshooting.

Questions, or testing on your own board?

Open an issue or a discussion on the repository, or email nicolo.giuliani6@studio.unibo.it. Results from other boards are especially welcome — everything here was developed against a single card.