[softnpu, mcast] wire up the softnpu backend for multicast table ops by zeeshanlakhani · Pull Request #405 · oxidecomputer/dendrite

zeeshanlakhani · 2026-10-15T13:37:09Z

Previously, SoftNPU didn't interact with multicast. mc_group_create and mc_port_add returned a stubbed OperationUnsupported, and port remove/destroy were no-ops.

With the merge of bitmap-based replication in oxidecomputer/p4rs#240, SoftNPU can now replicate off of port bitmaps carried in the replication table entry itself. On Tofino, membership subscriptions live in the PRE and change independently of match-action state. On SoftNPU, membership is captured at write time, and a membership mutation involves rewriting these affected replication entries.

Includes:

asic/SoftNPU:
- Live replication entries are tracked per group and rewritten when members are added/removed. The replication tracking lock covers both the snapshot and table updates.
- Source filter, mcast NAT (dst-only, per [multicast] Bitmap-based multicast replication for sidecar-lite sidecar-lite#152), egress decap, and mcast src MAC tables now map onto their sidecar-lite tables.
  - egress.asic_id_to_port maps ASIC port IDs to logical port numbers for indexing the decap bitmap (same as on Tofino).
- TableRemove is sent before TableAdd because p4rs drops an add whose key already exists and keeps the older entry (action and params).
dpd/mcast:
- mcast::reset deletes each known group's entries before the table clears. SoftNPU's Table::clear is still a stub.
- del_entries_where skips the stale-entry table sweep when the backend cannot list entries (e.g., in SoftNPU).
deps
- softnpu-lib + p4rs have been updated on main ([multicast] softnpu with bitmap-based replication softnpu#183, [multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2 p4#240). SoftNPU must still loads [multicast] Bitmap-based multicast replication for sidecar-lite sidecar-lite#152 at runtime where used (until its merged).

Previously, SoftNPU didn't interact with multicast. `mc_group_create` and `mc_port_add` returned a stubbed `OperationUnsupported`, and port remove/destroy were no-ops. With bitmap-based replication in p4rs ([#240](oxidecomputer/p4#240)), SoftNPU can replicate off of port bitmaps carried in the replication table entry itself. On the Tofino, membership subscriptions live in the PRE and change independently of match-action state. In the SoftNPU work, membership is captured at write time, and a membership mutation means rewriting the affected replication entries (when triggered). Includes: - *asic/SoftNPU*: - Live replication entries are tracked per group and rewritten when members are added/removed. The replication tracking lock covers both the snapshot and table updates. - Source filter, mcast NAT (tagged and untagged), egress decap, and mcast src MAC tables now map onto their sidecar-lite tables respectively. - `egress.asic_id_to_port` maps ASIC port IDs to logical port numbers for indexing the decap bitmap (same as on Tofino). - `TableRemove` is sent before `TableAdd` because p4rs drops an add whose key already exists and keeps the older entry (action and params). - *dpd/mcast*: - `mcast::reset` deletes each known group's entries before the table clears. SoftNPU's `Table::clear` is still a stub. - `del_entries_where` skips the stale-entry table sweep when the backend cannot list entries (e.g., in SoftNPU). - softnpu-lib is pinned to `zl/multicast` @ softnpu#183, which pulls in p4#240. SoftNPU loads sidecar-lite#152 at runtime.

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 on dendrite up through oxidecomputer/dendrite#405 following RFD 488. We manage 2 bitmaps per underlay group (external and underlay), for which replication is driven by p4rs `replicate` (oxidecomputer/p4#240; we track p4 on main now). Ingress zeroes out at most one of these bitmaps determined by the geneve multicast option tag (`tag=0` zeroes out underlay; `tag=1` zeroes out external), and merges both. Tag 0 (External) also decaps upon ingress. With no tag given, as on external NAT ingress packets, both bitmaps remain in place. Only geneve-encapsulated multicast is replicated. Multicast packets that miss ingress NAT drop on the empty bitmap, matching DPD's `drop_mcastv{4,6}_no_group` semantics. Packet copies that remain encapsulated lose a hop on the outer IPv6 header, as modelled in `MulticastRouter6`. External multicast is NAT-encapsulated into the underlay via `nat_v{4,6}_mcast(_untagged)`, reusing `forward_to_sled`. Tagged and untagged variants exist as separate tables since x4c cannot key directly on header validity. The encap piece now zeroes out the outer src, traffic class, flow label, and inner eth src, like DPD's `add_encap_headers`. Egress decap checks `ff04::/64` and `tag=2`, matching DPD's egress `mcast_tag_check`, with per-port decap from a RID-keyed bitmap (`tbl_decap_ports`) and optional VLAN. `tag=2` decapsulation also rejects inner unicast. `tag=0` copies (already decapped in ingress) use the table for the VLAN. Dst MAC derivation (RFC 1112, RFC 2464) and per-port src MAC rewriting run on every packet copy. Includes: - The parser rejects `ff01` addresses outright (`ff02` *does* still go to the scrimlet). - Multicast packets with a TTL/hop_limit <= 1 get dropped in ingress after the `fwd_from_scrimlet` return, letting scrimlet transmissions pass through. - Both decapsulation paths drop inner TTL <= 1. IPv4 checksums use RFC 1624 (eqn. 3): decrementing the TTL adds 0xfeff in one's complement, which yields 0x0000 rather than 0xffff at the zero boundary, matching a full recompute (XOR is used here due to `~` not being parsable by x4c). - scadm gets add/remove/dump for replication (v6), source filters, mcast NAT, decap ports, src MAC, and asic-id-to-port (for logical mapping). - Unicast modifications: - The router drops TTL/hop_limit <= 1 and decrements remaining packets on routes to nonzero ports, updating the IPv4 checksum. - Port 0 (the scrimlet) skips the `<= 1` check and the decrement, while a TTL/hop_limit `= 0` still drops before route lookup, except on the boundary services decapsulation path. - DPD's Router4/6 still decrement service-bound packets. - The NAT encap zeroing described above is shared with unicast NAT. - Packets with a parsed Geneve header skip ingress NAT, including unicast.

cfzimmerman

Voxel exercises softnpu, right? Can you run a voxel test and post some validation results? I'll stamp once we know it works.

cfzimmerman · 2026-10-28T19:58:41Z

+/// On the production sidecar, the replication table holds a group identifier,
+/// while the replication engine tracks live membership subscriptions, which means
+/// that any port additions do not trigger table rewrites.


Out of curiosity, how did you learn this? Are there docs somewhere that explain? Or is this from reading tofino-backend compiler source?

I started from the TNA doc: §5.7.3 "Packet Replication Engine" (pp. 26-27), Ingress only hands the TM (traffic manager) a group id (mcast_grp_a/b, rid, exclusion ids; see §5.4 "Destinations"), and the control plane configures each group's L1 nodes / L2 ports separately (§5.7.3.1 "Multicast group configuration").

The fields we use are here: https://github.com/barefootnetworks/Open-Tofino/blob/5b6ef19873698fc7d9c49cb33fd54c5fca2ecadd/share/p4c/p4include/tofino1_base.p4#L154-L181. And, we interact with them here:

dendrite/asic/src/tofino_asic/mcast.rs

Lines 65 to 140 in 5076359

bf_mc_mgrp_create(mcast_hdl, dev_id, domain, &mut mgrp_hdl)

.check_error("creating multicast group")?;

}

Ok(mgrp_hdl)

}

fn mgrp_destroy(

mcast_hdl: bf_mc_session_hdl_t,

dev_id: bf_dev_id_t,

mgrp_hdl: bf_mc_mgrp_hdl_t,

) -> AsicResult<()> {

unsafe {

bf_mc_mgrp_destroy(mcast_hdl, dev_id, mgrp_hdl)

.check_error("destroying multicast group")?;

}

Ok(())

}

fn mgrp_get_count(

mcast_hdl: &Handle,

dev_id: bf_dev_id_t,

mut count: u32,

) -> AsicResult<usize> {

unsafe {

bf_mc_mgrp_get_count(mcast_hdl.bf_get().mcast_hdl, dev_id, &mut count)

.check_error("getting total count of multicast groups")?;

}

Ok(count as usize)

}

fn associate_node(

mcast_hdl: bf_mc_session_hdl_t,

dev_id: bf_dev_id_t,

mgrp_hdl: bf_mc_mgrp_hdl_t,

node_hdl: bf_mc_node_hdl_t,

exclusion_id: u16,

) -> AsicResult<()> {

unsafe {

bf_mc_associate_node(

mcast_hdl,

dev_id,

mgrp_hdl,

node_hdl,

exclusion_id != 0,

exclusion_id,

)

.check_error("associating multicast node")?;

}

Ok(())

}

fn dissociate_node(

mcast_hdl: bf_mc_session_hdl_t,

dev_id: bf_dev_id_t,

mgrp_hdl: bf_mc_mgrp_hdl_t,

node_hdl: bf_mc_node_hdl_t,

) -> AsicResult<()> {

unsafe {

bf_mc_dissociate_node(mcast_hdl, dev_id, mgrp_hdl, node_hdl)

.check_error("dissociating multicast node from group")?;

}

Ok(())

}

fn node_create(

mcast_hdl: bf_mc_session_hdl_t,

dev_id: bf_dev_id_t,

repl_id: u16,

portmap: &mut [u8],

lagmap: &mut [u8],

) -> AsicResult<u32> {

let mut node_hdl = 0u32;

unsafe {

bf_mc_node_create(

So, yeah, on the sidecar, a port join never rewrites the table entry. SoftNPU, on the other hand, has no PRE, which is why we insert the bitmap into the entry and rewrite it on membership updates.

zeeshanlakhani · 2026-10-29T07:38:06Z

Voxel exercises softnpu, right? Can you run a voxel test and post some validation results? I'll stamp once we know it works.

Incoming with this line of work. Via oxidecomputer/voxel#75 and past iterations, I demo'ed the prototype set of branches working.

cfzimmerman

Left a reply on the table.rs diff. Otherwise looks good to me once we're confident in e2e voxel testing.

This also replaces ``make_tag(TEST_TAG)` w/ `TEST_TAG`` in two dpd-client mcast tests.

rcgoodfellow · 2026-11-06T17:51:25Z

    pub table_entry_del: TableChaos,
    pub mc_config: AsicMulticastConfig,
+    #[serde(default)]
+    pub entries_unsupported: BTreeSet<TableType>,


What is the purpose of this?

The goal was to make chaos behave more like SoftNPU, where get_entries returns OperationUnsupported. There were tests in the rollback code that checked if we could still clear the group's stored sources one entry at a time without listing the entire table.

But, it's unnecessary. Removing.

rcgoodfellow · 2026-11-06T18:19:23Z

+
+    /// Rewrite the replication entries refreshed after a membership change.
+    ///
+    /// SoftNPU fixes each entry's port bitmaps at write time and has no


I don't think this is correct. I believe an entry add will overwrite an entry with the same keyset.

rcgoodfellow · 2026-11-06T18:40:40Z

+
+/// Multicast group membership and the replication entries derived from it.
+///
+/// Unlike Tofino, SoftNPU has no PRE, leading us to track group membership


It seems odd to me that stateful group tracking is happening at this level, as dendrite is generally considered to be stateless (or more accurately only to have transient soft state). I would have thought that the broader control plane would be tracking group membership and reconciling that down to dendrite. So all dendrite really needs to do is a set-difference style add/remove/update on the replication tables.

rcgoodfellow · 2026-11-06T18:47:49Z

-// All tables are defined to be 1024 entries deep
+/// Write an entry, or replace any entry already associated with the key.
+///
+/// Note: SoftNPU's `TableAdd` retrains an existing entry in place and reports


Suggested change

/// Note: SoftNPU's `TableAdd` retrains an existing entry in place and reports

/// Note: SoftNPU's `TableAdd` retains an existing entry in place and reports

rcgoodfellow · 2026-11-06T18:55:21Z

+        // get removed here, one `TableRemove` per keyset, and every other
+        // table retains its entries.
+        //
+        // Send a `TableClear` once softnpu and propolis support one (the


This has landed, so looks like this can change now?

rcgoodfellow · 2026-11-06T19:02:40Z

-        let keyset_data = keyset_data(match_data.fields, self.type_);
+        let keyset_data = keyset_data(fields, self.type_);
+
+        // Multicast replication is handled ahead of the generic match so the


Similar to my comment above. It's seems odd to be tracking this state at this level. I get that softnpu and tofino are different (and x2 will be different), but it also seems to me that we should be tracking the fundamental group address -> port_bitmaps relationship at a higher level.

Yeah, good point(s).

The next push removes the tracking of each replication entry's external and underlay group IDs, which was used for per-port rewriting. Since dpd already reconciles membership and writes the replication entry afterward (or clears it for an empty group), that should be enough.

The group addr → membership relationship lives in dpd. The SoftNPU backend still needs a group ID → port bitmap (map) for the aal multicast ops, uses this to look up the bitmaps when dpd writes an entry.

rcgoodfellow · 2026-11-06T19:25:01Z

+/// Write an entry, or replace any entry already associated with the key.
+///
+/// Note: SoftNPU's `TableAdd` retrains an existing entry in place and reports
+/// nothing back. The backend key lookup does not return `AsicError::Exists`,


I think the bug here is in p4 codegen. The add entry codegen should use HashSet::replace instead of HashSet::insert as replacement independent of whether or not a matching key already exists in the table is the predominant match-action use case.

Yeah, I was holding off, but I'll just do this.

oxidecomputer/p4#282 @rcgoodfellow.

rcgoodfellow · 2026-11-06T19:27:17Z

-
-        //TODO implement in softnpu
-        Ok(())
+        // SoftNPU does not currently support in-place updates. `entry_add`


Per my comment above, i suspect we should implement this by changing p4rs codegen to use HashSet::replace instead of HashSet::insert for add functions that operate on p4rs::Table::entries.

done (link above)

rcgoodfellow · 2026-11-06T19:29:18Z

+        // Stop tracking a replication entry once it leaves the pipeline so a
+        // later membership change does not resurrect a stale bitmap.
+        if self.type_ == TableType::McastIpv6 {
+            hdl.mc_data.lock().unwrap().forget_repl_entry(&keyset_data);


Similar to above, tracking this at this level feels off.

Yep, fixed as part of #405 (comment).

zeeshanlakhani added this pull request to stack #401 October 15, 2026 13:38

zeeshanlakhani self-assigned this Oct 15, 2026

zeeshanlakhani added the multicast Multicast-related feature/impl label Oct 15, 2026

zeeshanlakhani added 7 commits October 17, 2026 06:29

[merge] merge fwd

776db38

[deps] p4(rs) update

8c742a3

[merge] merge fwd

94ef914

[deps] softnpu and p4(rs) to main

b52e768

[merge] merge fwd

5671a7e

[merge] merge fwd -> drop VLAN dispatch for mcast NAT

90eacb1

[merge] merge fwd

4a52d14

zeeshanlakhani removed this pull request from stack #401 October 21, 2026 01:55

zeeshanlakhani added this pull request to stack #410 October 21, 2026 01:59

zeeshanlakhani requested review from FelixMcFelix, cfzimmerman, rcgoodfellow and taspelund October 21, 2026 02:37

[merge] merge fwd

d664792

cfzimmerman reviewed Oct 28, 2026

View reviewed changes

[merge] merge fwd

61c965a

zeeshanlakhani added 2 commits October 30, 2026 00:13

[review] address review items

1188ce3

[merge] merge fwd

05b4530

zeeshanlakhani requested a review from cfzimmerman October 30, 2026 05:45

zeeshanlakhani added 2 commits October 30, 2026 12:01

[merge] merge fwd

b69f67e

[merge] merge fwd

53f18f3

cfzimmerman approved these changes Oct 30, 2026

View reviewed changes

Comment thread asic/src/softnpu/table.rs Resolved

[review, softnpu] replace entry on add and don't drop on a dup add

563cb14

zeeshanlakhani mentioned this pull request Oct 31, 2026

[mcast, pt1] multicast address, source, and route key types oxidecomputer/maghemite#943

Open

[merge] merge fwd

3019e6f

zeeshanlakhani force-pushed the zl/mcast-softnpu branch from 70ff3bc to 3019e6f Compare November 2, 2026 16:34

zeeshanlakhani added 4 commits November 5, 2026 16:53

[merge] merge fwd

ef57482

[merge] merge-fwd

628e849

[softnpu, mcast] clear known src filters on update/delete + sweep after

eb55743

This also replaces ``make_tag(TEST_TAG)` w/ `TEST_TAG`` in two dpd-client mcast tests.

[merge] merge fwd

50c7013

zeeshanlakhani force-pushed the zl/mcast-softnpu branch from 3562104 to 50c7013 Compare November 6, 2026 06:11

rcgoodfellow reviewed Nov 6, 2026

View reviewed changes

This was referenced Nov 7, 2026

[codegen, fix] Replace (not insert) table entries with matching keys oxidecomputer/p4#282

Open

[ddm, mg-lower, pt6] underlay membership sweep, admin v5, & mrib origination oxidecomputer/maghemite#999

Open

	bf_mc_mgrp_create(mcast_hdl, dev_id, domain, &mut mgrp_hdl)
	.check_error("creating multicast group")?;
	}
	Ok(mgrp_hdl)
	}

	fn mgrp_destroy(
	mcast_hdl: bf_mc_session_hdl_t,
	dev_id: bf_dev_id_t,
	mgrp_hdl: bf_mc_mgrp_hdl_t,
	) -> AsicResult<()> {
	unsafe {
	bf_mc_mgrp_destroy(mcast_hdl, dev_id, mgrp_hdl)
	.check_error("destroying multicast group")?;
	}
	Ok(())
	}

	fn mgrp_get_count(
	mcast_hdl: &Handle,
	dev_id: bf_dev_id_t,
	mut count: u32,
	) -> AsicResult<usize> {
	unsafe {
	bf_mc_mgrp_get_count(mcast_hdl.bf_get().mcast_hdl, dev_id, &mut count)
	.check_error("getting total count of multicast groups")?;
	}

	Ok(count as usize)
	}

	fn associate_node(
	mcast_hdl: bf_mc_session_hdl_t,
	dev_id: bf_dev_id_t,
	mgrp_hdl: bf_mc_mgrp_hdl_t,
	node_hdl: bf_mc_node_hdl_t,
	exclusion_id: u16,
	) -> AsicResult<()> {
	unsafe {
	bf_mc_associate_node(
	mcast_hdl,
	dev_id,
	mgrp_hdl,
	node_hdl,
	exclusion_id != 0,
	exclusion_id,
	)
	.check_error("associating multicast node")?;
	}
	Ok(())
	}

	fn dissociate_node(
	mcast_hdl: bf_mc_session_hdl_t,
	dev_id: bf_dev_id_t,
	mgrp_hdl: bf_mc_mgrp_hdl_t,
	node_hdl: bf_mc_node_hdl_t,
	) -> AsicResult<()> {
	unsafe {
	bf_mc_dissociate_node(mcast_hdl, dev_id, mgrp_hdl, node_hdl)
	.check_error("dissociating multicast node from group")?;
	}
	Ok(())
	}

	fn node_create(
	mcast_hdl: bf_mc_session_hdl_t,
	dev_id: bf_dev_id_t,
	repl_id: u16,
	portmap: &mut [u8],
	lagmap: &mut [u8],
	) -> AsicResult<u32> {
	let mut node_hdl = 0u32;

	unsafe {
	bf_mc_node_create(

	/// Note: SoftNPU's `TableAdd` retrains an existing entry in place and reports
	/// Note: SoftNPU's `TableAdd` retains an existing entry in place and reports

Conversation

zeeshanlakhani commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cfzimmerman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 29, 2026

Uh oh!

cfzimmerman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zeeshanlakhani commented Oct 15, 2026 •

edited

Loading