Cavium CN6640-SNIC10E

Status: working. Both ports link at 10 G on a DAC and pass traffic bidirectionally. The card boots with no serial cable and the host brings both NICs up automatically at boot. Hardware-validated on one reference card, never in the field.

This is the Linux page of the Cavium CN6640-SNIC10E docs. The overview, the FAQ and the list of reference documents are on the general page.

Requirements

• The card in a PCIe slot, with Secure Boot off: kernel lockdown blocks the setpci and BAR access this stack needs. "Above 4G decoding" is not required.
• A modern Linux host (developed on 6.14) with matching kernel headers; dkms recommended so the module survives kernel upgrades.
• The prebuilt card image from the releases, or an OpenWrt build tree to build it yourself.
• A USB-serial (FT232) adapter for the one-time u-boot provisioning only. Normal operation is serial-free.

Quick start

git clone https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic
cd cavium-cn6640-snic10e-octeon-ii-nic

# 1) the card's OS image, prebuilt: octboot looks for it here in the repo root
curl -LO https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic/releases/latest/download/openwrt-octeon-generic-snic10e-initramfs-kernel.bin

# 2) the host side: builds + installs octnic (DKMS), drops the configs, enables autostart
sudo ./install.sh

# 3) first time on this machine only: persist the card's u-boot env (serial cable, once)
sudo ./scripts/card-prep-hostboot.sh

# 4) every boot, no serial: pushes the image over PCIe, brings up oct0 + oct1
sudo systemctl start cavium-nic
ip -br addr show oct0

Step 4 is what the cavium-nic service runs at every host boot, so after step 3 the card comes up on its own. Steps 1 to 3 are once per machine. The manual equivalent is sudo ./octboot followed by sudo modprobe octnic ports=2; both ports are then ordinary netdevs.

install.sh is also the guided installer described on the general page: if the card is not provisioned yet it offers step 3 itself and asks before writing anything.

Day to day

• Autostart. system/cavium-nic.service boots the card and brings both ports up at every host boot. It does nothing if the card OS is already running.
• Temperatures and power. Card temperatures and an estimated card power draw show up in plain sensors as cavium_card once octnic is loaded. The power figure is a model, not a measurement.
• Watchdog. octnic checks the card every 2 s. If the card resets or hangs (for example, it overheated because its fan is unplugged) or crosses temp_crit_mc (default 100 °C), it logs a KERN_EMERG alert and drops carrier on oct0 and oct1. The warning level is temp_warn_mc (85 °C). Both are module parameters, writable under /sys/module/octnic/parameters/.

More in USAGE: module parameters, benchmarking and troubleshooting.

Performance

iperf3 -P8, MTU 9000, fresh card boot, one port at a time. Both ports loaded in the same direction aggregate to about 10.5 Gb/s: the host-PIO 64 B-TLP wall on a Gen2 ×4 link. Full duplex is the open front: 13.5 Gb/s aggregate with UDP, but TCP suffers under the RX loss. Numbers, method and the tuning behind them: PERFORMANCE.

Troubleshooting

• One port stops receiving. Reload the module (rmmod octnic && modprobe octnic ports=2) and re-add the IPs: the rings resync without rebooting the card.
• The host hangs when the card is probed. Keep liquidio blacklisted.
• The card wedged. A card that has been driven hard can wedge outright, and because it is bus-powered, recovery is a host reboot.

Details in USAGE, troubleshooting.

Known limitations

• Dual-port simultaneous load. Both ports work bidirectionally, but the card wedges under sustained dual-port traffic, so a fresh boot is needed per test run.
• Forward path. The BAR2 PIO outbound path plateaus at about 3.1 Gb/s. Full 10 G in that direction needs inbound DPI, which wedges the card. Not solved.
• Image rebuild. openwrt/build-openwrt.sh compiles, but it isn't the exact path the shipped release image was verified against.

Questions, or testing on your own board?

Open an issue or a discussion on the repository, or email me@nicologiuliani.site. Results from other boards are especially welcome: everything here was developed against a single card.