From nobody Sat Jul 25 03:46:04 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 365FA3B100A for ; Sun, 19 Jul 2026 17:59:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484000; cv=none; b=EDBcRscItcSwFIkmLOdYVjZY4aS1ycqzgKa6pnMGHnaXyxneB4xxa+J4LnaFEE5U4hNfZp9sHozE4HE6ZqRDkP5+KLts3VFNg6pY4SDLMkmUwnouAPbaxsbrJXhejYEi+tirp4zHbkANvMp8LCOYX/Jm1wfWsIrowOP0FE2vsW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484000; c=relaxed/simple; bh=SxgeUFl3a/Kj5MuHYmf7LKaXCbqGDkZ6eIkZB5MHqKs=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VM2NzDXZXD4jbx1bVeBu0tLFNt+hH6daBgQQ4Uu1cxUSZhSev42Xruf+N2+Ej+DT7xUf7iI/NyTkgpjHJl+2R7SeXkVlFEqVu0w4FsXFeYAQeo8ADENFug4BQDUezu0/Bpx4KInBNSkyi6QU/pgnbEME/YZjTsRLYNELjxSDhMc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=owwlH9u/; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="owwlH9u/" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2cc97653887so103554545ad.1 for ; Sun, 19 Jul 2026 10:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784483991; x=1785088791; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=YyqhYngq0RlhZPKjh6r1tvXs2LIUb0LhEqMCu17FH5w=; b=owwlH9u/0Fu67JElRT0sfgxqvdSiHmThPeMC6yGJWcLNCbm4rqMMZEG0UhxhRwwEOZ JutYlW2S17eOzRQYWk6HfRXfYJnW2BdPcM1DdDqyRUBfYIakGIGguuOwUI3P30ezR4uA GmqgMYuIGtECbbkP5TC2T2L4WBfPl9wHw5SmK+qLH8mJIEmqk/lY6TGELFUnwW00BuNM 09K+HnZIKPqjBWL5Bz3rePDxHFZNLtUAYcpxjFPXHKEJwu9nfUwKYrU6SuBro0MmtH+R jO99oMAIJ0qxQQgzkV/m+qwISecWr1WA3WLznrH9wNazHseaBGQsl7oB0q+90WfCmVRr eJlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784483991; x=1785088791; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=YyqhYngq0RlhZPKjh6r1tvXs2LIUb0LhEqMCu17FH5w=; b=tG5aQ9up1ejniUAnH0vHzdg5yVeR8KiC866W2JjBt3zEEUrdXTlL/AKtyHgO6ySS1Z NmvjYePPhU/jZO34AcjNYGbBrkrcQr4xSqyyS4lMtXnJ0Z2KiF8vZ9qcuNsjzmBVr2DL /mUA6v2FjpuLoETQXKslbvssSq4NLG7aD1RHjLZ09nUBIhlUeZt4ZpFYqMuXPEzHlAfM dqW0XWubLF2UVIyi1Ot46ruSLYGnfzmHKbS3HP0c6+u8OwV74A1DeDuGrZQyj/i5Z+Ch oWAf6AFIa7SnhbST/ReE13/uKCrsoRxwfNyS3zAjEL2Yv0KaAgeX/XLhepKtSItKbIPL 94QA== X-Forwarded-Encrypted: i=1; AHgh+Rpk6obSmZOfzKzJ4dURLBmFOADGoay4tO1R8FEh9Yk7DQfyuWdkIERUk+lVdN/RcyRmKGnukh+HEihcdmk=@vger.kernel.org X-Gm-Message-State: AOJu0YwVlYRi4RzgIaA53mlcJ/uPmAGmeQGGfvvfAIIqdgFZyI1ZkT3y MU5faul6KZNA9VOFymAo0hTXe5THTCGW+LkRu97Bi3eocC2sOXk5JpXA X-Gm-Gg: AfdE7clJ+qoRWoW9bhx1vOdXLBGq5zJnRWdf1oGxDx2GcrcKmjPvlWlzvhC2P4lPU8B w7Zwhku0jkHPSVwHFbA9ICQOO14sG171Wx3k86SXqMj6RJYE3zECwhXoW/Wktqzf3c2yDUnW9yK 3prqZKpGZ9XVzX06A2MfKLSPkzK7xlyNQFN+tQ25rcAfVAOYBAyvxbfeCx48+y7IZK2E/J+n36c QCMISG5bd7i0iZGuuJ2I3wt2SW4IBHKRO6EzGOmZBhVhrCTpNrFaIeUxWgSEN6dl8HxAPURrqov JTUD9tXYfO3FdhCCChJeDju9xDevKaBa9xq6Cfulf7O2oV2WB4p2Z4mzG4865bRxUZW+5FBqhl5 ntX7vvB6PJcWEtexgCV5cjs4QTYNZrjyBit+scZ/hSGLyfOlsVdumAq6wKkyh2p3/tg== X-Received: by 2002:a17:903:1987:b0:2ce:9c48:22d3 with SMTP id d9443c01a7336-2cf348840dfmr128578695ad.11.1784483990726; Sun, 19 Jul 2026 10:59:50 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.10.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 10:59:49 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 01/13] net: knod: add uapi and core headers Date: Sun, 19 Jul 2026 17:58:45 +0000 Message-ID: <20260719175857.4071636-2-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the uAPI and core headers for KNOD, an in-kernel network offload device that connects a NIC RX path to an accelerator (e.g. a GPU) for zero-copy packet processing. - include/uapi/linux/knod.h: genetlink uAPI - Documentation/netlink/specs/knod.yaml: netlink spec - include/net/knod.h: core framework types and ops - include/net/spsc_ring.h: lock-free SPSC ring for the data path Signed-off-by: Taehee Yoo (cherry picked from commit d93829acbce14528fcf70e155a0594c0c6151cce) --- Documentation/netlink/specs/knod.yaml | 176 +++++++ include/net/knod.h | 467 +++++++++++++++++++ include/net/spsc_ring.h | 645 ++++++++++++++++++++++++++ include/uapi/linux/knod.h | 67 +++ 4 files changed, 1355 insertions(+) create mode 100644 Documentation/netlink/specs/knod.yaml create mode 100644 include/net/knod.h create mode 100644 include/net/spsc_ring.h create mode 100644 include/uapi/linux/knod.h diff --git a/Documentation/netlink/specs/knod.yaml b/Documentation/netlink/= specs/knod.yaml new file mode 100644 index 000000000000..7c64df467634 --- /dev/null +++ b/Documentation/netlink/specs/knod.yaml @@ -0,0 +1,176 @@ +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Cla= use) +--- +name: knod + +doc: + KNOD (KFD network offload) control plane. + + Bind a NIC netdevice to a GPU/DPU offload accelerator and select which + offload feature (BPF, IPsec) the accelerator runs. + +definitions: + - + type: enum + name: feature + entries: [none, bpf, ipsec] + - + type: enum + name: accel-type + entries: [gpu, dpu] + +attribute-sets: + - + name: accel + attributes: + - + name: id + doc: Accelerator id. + type: u32 + checks: + min: 1 + - + name: name + doc: Accelerator name. + type: string + - + name: type + doc: Accelerator type. + type: u32 + enum: accel-type + - + name: feature-cap + doc: Bitmask of offload features the accelerator supports. + type: u32 + enum: feature + enum-as-flags: true + - + name: feature-ena + doc: Currently active offload feature on the accelerator. + type: u32 + enum: feature + - + name: nic + attributes: + - + name: ifindex + doc: ifindex of the NIC netdevice registered with KNOD. + type: u32 + checks: + min: 1 + - + name: name + doc: Name of the NIC netdevice. + type: string + - + name: dev + attributes: + - + name: nic-ifindex + doc: ifindex of the bound NIC netdevice. + type: u32 + checks: + min: 1 + - + name: accel-id + doc: id of the bound accelerator. + type: u32 + checks: + min: 1 + +operations: + list: + - + name: accel-get + doc: Get / dump accelerators registered on the system. + attribute-set: accel + do: + request: + attributes: + - id + reply: &accel-all + attributes: + - id + - name + - type + - feature-cap + - feature-ena + dump: + reply: *accel-all + - + name: accel-set + doc: Select the active offload feature of an accelerator. + attribute-set: accel + flags: [admin-perm] + do: + request: + attributes: + - id + - feature-ena + reply: + attributes: [] + - + name: nic-get + doc: Get / dump NICs registered with the KNOD framework. + attribute-set: nic + do: + request: + attributes: + - ifindex + reply: &nic-all + attributes: + - ifindex + - name + dump: + reply: *nic-all + - + name: attach + doc: Attach a NIC netdevice to an accelerator. + attribute-set: dev + flags: [admin-perm] + do: + request: + attributes: + - nic-ifindex + - accel-id + reply: + attributes: [] + - + name: detach + doc: Detach a NIC netdevice from its accelerator. + attribute-set: dev + flags: [admin-perm] + do: + request: + attributes: + - nic-ifindex + reply: + attributes: [] + - + name: dev-get + doc: Get / dump active NIC <-> accelerator bindings. + attribute-set: dev + do: + request: + attributes: + - nic-ifindex + reply: &dev-all + attributes: + - nic-ifindex + - accel-id + dump: + reply: *dev-all + - + name: dev-add-ntf + doc: Notification about a new NIC <-> accelerator binding. + notify: dev-get + mcgrp: mgmt + - + name: dev-del-ntf + doc: Notification about a removed NIC <-> accelerator binding. + notify: dev-get + mcgrp: mgmt + +mcast-groups: + list: + - + name: mgmt diff --git a/include/net/knod.h b/include/net/knod.h new file mode 100644 index 000000000000..50333b0e028b --- /dev/null +++ b/include/net/knod.h @@ -0,0 +1,467 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_KNOD_H +#define __NET_KNOD_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct knod_dev; +struct knod_netdev; +struct knod_accel; +struct gen_pool; +struct page_pool; +struct netlink_ext_ack; +struct net_devmem_dmabuf_binding; + +extern struct mutex knod_lock; + +struct spsc_bd { + netmem_ref netmem; + u64 act; + u16 off; + u16 len; + u32 page_idx; + struct page_pool *pp; +}; + +/* + * KNOD action codes for spsc_bd.act + * + * Base actions (compatible with XDP constants for BPF/XDP path): + */ +#define KNOD_ABORTED XDP_ABORTED /* 0 */ +#define KNOD_DROP XDP_DROP /* 1 */ +#define KNOD_PASS XDP_PASS /* 2 */ +#define KNOD_TX XDP_TX /* 3 */ +#define KNOD_REDIRECT XDP_REDIRECT /* 4 */ + +/* + * Extended actions - accel-specific, must not collide with XDP range [0..= 7]. + * The NIC act_handler treats any unknown code as "in-flight to accel": + * stop releasing at that entry and wait for the accel to update bd->act. + */ +#define KNOD_ACT_INFLIGHT 0x80 /* accel processing in progress */ +#define KNOD_IPSEC_INFLIGHT 0x100 /* GPU IPsec dispatch in progress */ +#define KNOD_IPSEC_PASS 0x101 /* GPU IPsec done, recycle page */ +#define KNOD_IPSEC_DROP 0x102 /* GPU IPsec done, drop + recycle */ + +/* + * PASS hand-off descriptor: the DD (NIC act_handler) fills one per PASS bd + * during its single act traversal and hands a batch to accel_ops->pass_co= py. + * @netmem is the source RX page (GPU memory); @off/@len are post-BPF (may + * differ from the bd's original values if the program adjusted head/tail). + */ +struct spsc_pass_bd { + /* source RX page, recycled after the copy lands */ + netmem_ref netmem; + u32 page_idx; /* page index in the queue's dmabuf RX buffer + * (the accel turns this into the GPU src addr; + * the netmem dma_addr is the NIC's, not the + * GPU's) + */ + u16 off; /* packet offset within the page (post-BPF) */ + u16 len; /* packet length (post-BPF) */ +}; + +/* + * Host-page page_pool provider context (GPU->host delivery pools). The + * framework fills the public fields, points page_pool_params.mp_priv at i= t and + * sets .mp_ops to the NOD-private page_pool_hostmem_ops (knod_dev.c); + * ->init() builds @genpool and ->destroy() tears it down. Must outlive t= he + * page_pool; @freed fires once the pool has fully drained. + */ +struct page_pool_hostmem { + struct page **pages; /* owner-supplied, @count real pages */ + unsigned int count; + dma_addr_t base_addr; /* device addr of pages[0] */ + void (*freed)(void *arg); /* called once the pool fully drains */ + void *arg; + struct gen_pool *genpool; /* private: managed by the provider */ +}; + +struct knod_work_priv { + struct dma_buf *dmabuf; + netmem_ref *netmems; + unsigned int *data_lens; + int *data_offs; + int cnt; + int index; + struct napi_struct *napi; + struct spsc_ring spsc_bds; + void *spsc_pool_priv; /* accel driver priv for spsc pool memory */ + u64 spsc_pool_gaddr; /* device-visible address of spsc pool */ + /* framework-owned delivery pool */ + struct page_pool *pass_pool; + /* provider ctx (owner storage) */ + struct page_pool_hostmem pass_hm; + /* d2h: SDMA-issued, awaiting drain */ + struct spsc_ring pass_pending; +} ____cacheline_aligned_in_smp; + +static inline void knod_napi_kick(struct knod_work_priv *wpriv) +{ + struct napi_struct *napi; + + rcu_read_lock(); + napi =3D READ_ONCE(wpriv->napi); + if (napi) + napi_schedule(napi); + rcu_read_unlock(); +} + +#define KNOD_DEFAULT_PASS_SLOTS 64 + +/* + * GPU->host delivery descriptor: a packet at @off (preserved headroom) for + * @len bytes within @netmem, a page from the framework delivery pool. The + * d2h path fills these into the per-queue pass_pending ring; knod_d2h_dra= in() + * turns each into an skb once its SDMA copy lands and hands it to the sta= ck. + */ +struct knod_pass_desc { + u16 len; /* head_frag length (ipsec: inner_len) */ + u16 off; /* head_frag offset (ipsec: inner_off) */ + netmem_ref netmem; /* dst: framework delivery-pool page */ + netmem_ref src; /* src: RX page recycled once the copy lands, or + * 0 when the producer recycles it elsewhere + * (ipsec: NIC act handler recycles via the bd) + */ + /* SDMA fence to await before delivery (async) */ + u32 fence_val; + u8 sdma_idx; /* which accel SDMA queue's fence to await */ + /* Feature finalisation context, consumed by knod_dev->post_copy. */ + struct { + u32 sa_slot; /* ipsec SA table slot */ + u32 seq_lo; /* ESP sequence low 32 bits */ + u32 seq_hi; /* ESN high 32 bits (0 if !ESN) */ + u8 mode; /* XFRM_MODE_TRANSPORT / _TUNNEL */ + u8 next_hdr; /* ESP trailer next-header */ + u8 family; /* AF_INET / AF_INET6 */ + } feat; +}; + +struct knod_accel_xdp_ops { + /* init/exit: permanent per-attach setup (attach/detach). */ + int (*init)(struct knod_dev *knodev); + void (*exit)(struct knod_dev *knodev); + /* activate/deactivate: feature resource alloc/free (feature select). */ + int (*activate)(struct knod_dev *knodev); + void (*deactivate)(struct knod_dev *knodev); + /* + * true while user XDP progs/maps are still bound (blocks feature + * switch). + */ + bool (*busy)(struct knod_dev *knodev); + int (*xdp_offload_init)(struct knod_dev *knodev); + void (*xdp_offload_uninit)(struct knod_dev *knodev); + int (*xdp_install)(struct knod_dev *knodev, + struct netdev_bpf *bpf); + int (*rx_netmem)(struct knod_dev *knodev, netmem_ref netmem, + unsigned int data_len, int data_offset, int index); + int (*rx_netmem_bulk)(struct knod_dev *knodev, + struct knod_work_priv *wpriv); + /* Direct dispatch */ + int (*dispatch)(struct knod_dev *knodev, int index); + /* Direct finish */ + int (*finish)(struct knod_dev *knodev, int index); + void (*start)(struct knod_dev *knodev); + void (*stop)(struct knod_dev *knodev); +}; + +struct xfrm_state; +struct xfrm_policy; +struct netlink_ext_ack; + +struct knod_accel_ipsec_ops { + /* init/exit: permanent per-attach setup (attach/detach). */ + int (*init)(struct knod_dev *knodev); + void (*exit)(struct knod_dev *knodev); + /* activate/deactivate: feature resource alloc/free (feature select). */ + int (*activate)(struct knod_dev *knodev); + void (*deactivate)(struct knod_dev *knodev); + /* + * true while offloaded xfrm SAs are still bound (blocks feature + * switch). + */ + bool (*busy)(struct knod_dev *knodev); + /* + * start/stop: worker/dispatcher start + GPU drain (interface + * up/down). + */ + void (*start)(struct knod_dev *knodev); + void (*stop)(struct knod_dev *knodev); + int (*xdo_dev_state_add)(struct knod_dev *knodev, + struct xfrm_state *x, + struct netlink_ext_ack *extack); + void (*xdo_dev_state_delete)(struct knod_dev *knodev, + struct xfrm_state *x); + void (*xdo_dev_state_free)(struct knod_dev *knodev, + struct xfrm_state *x); + bool (*xdo_dev_offload_ok)(struct knod_dev *knodev, + struct sk_buff *skb, + struct xfrm_state *x); + void (*xdo_dev_state_advance_esn)(struct knod_dev *knodev, + struct xfrm_state *x); + void (*xdo_dev_state_update_stats)(struct knod_dev *knodev, + struct xfrm_state *x); + int (*xdo_dev_policy_add)(struct knod_dev *knodev, + struct xfrm_policy *x, + struct netlink_ext_ack *extack); + void (*xdo_dev_policy_delete)(struct knod_dev *knodev, + struct xfrm_policy *x); + void (*xdo_dev_policy_free)(struct knod_dev *knodev, + struct xfrm_policy *x); +}; + +struct knod_accel_ops { + int (*attach)(struct knod_dev *knodev); + void (*pre_detach)(struct knod_dev *knodev); + void (*detach)(struct knod_dev *knodev); + /* interface up/down (NIC driver knod_dev_start/stop): worker only. */ + void (*dev_start)(struct knod_dev *knodev); + void (*dev_stop)(struct knod_dev *knodev); + void *(*alloc_mem)(struct knod_dev *knodev, size_t size, + u64 *gaddr, struct page ***pages, void **priv); + void (*free_mem)(struct knod_dev *knodev, void *priv); + /* + * Map dmabuf RX BOs into the GPU VM; must run after + * knod_dmabuf_attach(). + */ + int (*mp_map)(struct knod_dev *knodev); + /* + * Device->host copy primitives, used by the common knod_d2h_copy / + * knod_d2h_drain delivery path. The accel owns the SDMA engine (and + * its fence/ring); the framework owns the pending ring and dst pool. + * d2h_submit: queue one GPU->host copy. Returns a monotonic fence + * position to tag the descriptor with, or 0 if the SDMA + * ring is full (caller drops -- backpressure). + * d2h_kick: publish the batch (fence + doorbell). + * d2h_fence: current completed fence position of an SDMA queue + * (drain compares the descriptor's tag against this). + */ + u32 (*d2h_submit)(struct knod_dev *knodev, u64 dst, int queue, + u32 page_idx, u16 off, u32 len); + void (*d2h_kick)(struct knod_dev *knodev); + u32 (*d2h_fence)(struct knod_dev *knodev, int sdma_idx); + struct knod_accel_xdp_ops *xdp_ops; + struct knod_accel_ipsec_ops *ipsec_ops; + + /* control plane (knod genetlink) feature select */ + int (*feature_get)(struct knod_accel *accel, u32 *ena, u32 *cap); + int (*feature_set)(struct knod_accel *accel, u32 feature, + struct netlink_ext_ack *extack); +}; + +struct knod_nic_ops { + int (*attach)(struct knod_dev *knodev); + int (*detach)(struct knod_dev *knodev); + int (*tx_handler)(struct knod_dev *knodev, struct spsc_bd **bds, + int cnt, int napi_index, void *priv); + int (*redir_handler)(struct knod_dev *knodev, netmem_ref *netmems, + u16 *lens, int cnt, int napi_index, + struct net_device *target_dev, void *priv); + int (*drop_handler)(struct knod_dev *knodev, netmem_ref *netmems, + u16 *lens, int cnt, void *priv); +}; + +struct knod_dev_stats { + u64_stats_t tx_packets; + u64_stats_t tx_bytes; + struct u64_stats_sync syncp; + u32 tx_dropped; + u32 tx_errors; +}; + +#define __NOD_FLAGS_XDP 0 +#define __NOD_FLAGS_IPSEC 2 +#define __NOD_FLAGS_KTLS 3 +#define __NOD_FLAGS_MAX (__NOD_FLAGS_KTLS + 1) +#define KNOD_FLAGS_XDP (1 << __NOD_FLAGS_XDP) +#define KNOD_FLAGS_IPSEC (1 << __NOD_FLAGS_IPSEC) +#define KNOD_FLAGS_KTLS (1 << __NOD_FLAGS_KTLS) + +#define KNOD_TYPE_GPU 0 +#define KNOD_TYPE_DPU 1 +#define KNOD_TYPE_MAX (KNOD_TYPE_DPU + 1) + +#define KNOD_SPSC_MAX 32 +#define KNOD_SPSC_ELEMS_MAX 8192 + +/* Per-RX-queue GPU->host delivery pages (in-flight cap; sized for the dee= pest + * feature pipeline, independent of any per-feature descriptor ring size). + * Must cover the worst case where one RSS-concentrated flow lands every + * in-flight work on a single queue: ipsec holds a full batch of delivery + * pages per work (alloc precedes the SDMA copy into them), so the cap nee= ds + * KNOD_IPSEC_NR_WORK * KNOD_IPSEC_PKT_BATCH (=3D 4 * 512) plus the pass_p= ending + * ring depth. Sized to absorb a full bd ring (KNOD_SPSC_ELEMS_MAX =3D 819= 2) + * worth of decrypted-but-undelivered packets plus the in-flight works. The + * backing is GTT, sized nqueues * KNOD_PASS_SLOTS * PAGE_SIZE (2 GiB at t= he + * 32-queue cap), which is why the alloc size path is size_t rather than i= nt. + */ +#define KNOD_PASS_SLOTS 16384 + +#define KNOD_STATUS_FREE 0 +#define KNOD_STATUS_USED 1 + +struct knod_netdev { + struct list_head list; + struct net_device *dev; + struct knod_dev *knodev; + struct knod_accel *accel; + struct knod_nic_ops *nic_ops; + struct module *owner; + int flags; + int status; + void *priv; +}; + +struct knod_accel_xdp { + struct bpf_offload_dev *bpf_dev; + struct xdp_attachment_info xdp; + struct xdp_attachment_info xdp_hw; + struct bpf_prog *bpf_offloaded; + struct list_head bound_maps; + void *priv; +}; + +struct knod_accel { + struct list_head list; + struct knod_accel_ops *accel_ops; + struct module *owner; + struct knod_accel_xdp xdp; + struct knod_dev *knodev; + struct knod_netdev *knetdev; + int status; + int type; + int flags; + int id; + void *priv; + char name[16]; +}; + +struct knod_dev { + struct list_head list; + struct knod_netdev *knetdev; + struct knod_accel *accel; + struct net_devmem_dmabuf_binding *bindings[KNOD_SPSC_MAX]; + struct mutex lock; + struct net_device *netdev; + struct knod_dev_stats *stats __percpu; + + struct knod_nic_ops *nic_ops; + struct knod_accel_ops *accel_ops; + /* per-queue priv data */ + struct knod_work_priv *wpriv; + bool started; + + /* IPsec proxy: original NIC xfrmdev_ops/feature state saved at attach, + * restored at detach so a NIC's native offload is not clobbered. + */ + const struct xfrmdev_ops *ipsec_orig_xfrmdev_ops; + bool ipsec_added_hw_esp; + + /* framework-owned GPU->host delivery (default pass): drain barrier */ + /* accel handle for the delivery buffer */ + void *pass_priv; + atomic_t pp_live; /* live delivery page_pools */ + struct completion pp_drained; /* all pools drained (teardown) */ + /* + * device->host (d2h) delivery: the accel owns the fence counter (its + * SDMA ring position); this lock just serialises the shared SDMA + * submit path across the per-queue NAPIs that drive knod_d2h_copy. + */ + spinlock_t d2h_lock; + + /* + * Feature delivery hook, set by the active feature on activate (NULL + * for bpf/none). knod_d2h_drain calls it after building the head_frag + * skb to run feature-specific finalisation (ipsec: SA/replay/secpath); + * it returns false to drop the packet. + */ + bool (*post_copy)(struct knod_dev *knodev, struct sk_buff *skb, + const struct knod_pass_desc *desc, int queue_idx); +}; + +static inline bool knod_dev_active(struct knod_dev *knodev) +{ + return !!knodev->accel->xdp.xdp_hw.prog; +} + +static inline struct bpf_prog * +knod_dev_offloaded(struct knod_dev *knodev) +{ + return knodev->accel->xdp.bpf_offloaded; +} + +static inline void knod_dev_offload(struct knod_dev *knodev, + struct bpf_prog *bpf_offloaded) +{ + knodev->accel->xdp.bpf_offloaded =3D bpf_offloaded; +} + +static inline bool knod_dev_map_empty(struct knod_dev *knodev) +{ + return list_empty(&knodev->accel->xdp.bound_maps); +} + +void knod_netdev_register(struct knod_netdev *netdev); +void knod_netdev_unregister(struct knod_netdev *netdev); +void knod_accel_register(struct knod_accel *accel); +void knod_accel_unregister(struct knod_accel *accel); +void knod_dev_start(struct knod_dev *knodev); +void knod_dev_stop(struct knod_dev *knodev); +int knod_dev_xdp_install(struct knod_dev *knodev, + struct netdev_bpf *xdp); +void knod_dev_get_stats64(struct knod_dev *knodev, + struct rtnl_link_stats64 *stats); +void knod_dev_lock(void); +void knod_dev_unlock(void); +struct sk_buff *knod_pass_build_skb(netmem_ref netmem, u16 off, u16 len, + struct page_pool *pool, bool napi); +int knod_d2h_copy(struct knod_dev *knodev, int napi_index, + struct spsc_pass_bd *bds, int cnt); +int knod_d2h_drain(struct knod_dev *knodev, int napi_index, + struct napi_struct *napi, int budget); + +extern struct list_head knod_dev_list; +extern struct list_head knod_netdev_list; +extern struct list_head knod_accel_list; + +#define for_each_xdev(d) \ + list_for_each_entry(d, &knod_dev_list, list) +#define for_each_xdev_safe(d) \ + list_for_each_entry_safe(d, n, &knod_dev_list, list) +#define for_each_nodev(d) \ + list_for_each_entry(d, &knod_netdev_list, list) +#define for_each_nodev_safe(d, n) \ + list_for_each_entry_safe(d, n, &knod_netdev_list, list) +#define for_each_accel(d) \ + list_for_each_entry(d, &knod_accel_list, list) +#define for_each_accel_safe(d, n) \ + list_for_each_entry_safe(d, n, &knod_accel_list, list) + +/* IPsec proxy functions */ +#if IS_ENABLED(CONFIG_XFRM_OFFLOAD) +int knod_ipsec_attach(struct knod_dev *knodev); +void knod_ipsec_detach(struct knod_dev *knodev); +#else +#endif + +/* XDP PASS drain - called from NIC NAPI poll */ +int knod_dev_xdp_drain_pass(struct knod_dev *knodev, + struct napi_struct *napi, + int queue_idx, int budget); + +#endif diff --git a/include/net/spsc_ring.h b/include/net/spsc_ring.h new file mode 100644 index 000000000000..1eccadc927d6 --- /dev/null +++ b/include/net/spsc_ring.h @@ -0,0 +1,645 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * spsc_ring.h - Lock-free SPSC Ring Buffer with embedded element pool + * + * Single-Producer Single-Consumer ring buffer. The ring owns a + * contiguous page-backed memory pool. Each slot is permanently bound + * to a cacheline-aligned element inside that pool. + * + * Producer calls spsc_produce() -> receives a pointer to a free + * element, writes data into it, then calls spsc_produce_commit() + * to publish. + * + * Consumer calls spsc_acquire() / spsc_pop() -> receives a pointer + * to a filled element. After spsc_release() the slot becomes + * available to the producer again - the element pointer is reused + * automatically because it is fixed to the slot. + * + * Two consumer modes (do NOT mix on the same instance): + * + * Mode 1 - Sliding window (2-step consumer): + * spsc_produce / commit -> peek -> acquire -> release + * + * Mode 2 - Simple queue: + * spsc_push -> pop + * + * Return convention: + * 0 success + * -ENOSPC ring full (producer side) + * -ENOENT ring empty (consumer side) + * -EINVAL bad parameter + * -ENOMEM allocation failure + * + * Memory ordering: + * Producer: write data -> smp_store_release(head) + * Consumer: smp_load_acquire(head) -> read data + * Consumer: done -> smp_store_release(tail) + * Producer: smp_load_acquire(tail) -> write data + * + * Capacity is always a power of two. + */ + +#ifndef _SPSC_RING_H +#define _SPSC_RING_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct spsc_ring { + void **slots; /* pointer-per-slot into pool */ + unsigned int mask; /* capacity - 1 */ + + /* Producer side */ + unsigned int head; /* next slot to publish */ + + /* Consumer side */ + unsigned int tail; /* oldest unconsumed slot */ + unsigned int acquired; /* sliding window read cursor + * tail <=3D acquired <=3D head + */ + + /* Element pool */ + struct page *pool_page; /* compound page backing elements */ + unsigned int pool_order; /* page order */ + unsigned int elem_stride; /* cacheline-aligned element size */ +} ____cacheline_aligned_in_smp; + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ +/* Init / Destroy */ +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ + +/** + * __spsc_init - initialize ring with a pre-allocated element pool + * @r: pointer to caller-allocated spsc_ring + * @elem_size: size of each element (rounded up to cacheline) + * @capacity: number of elements (rounded up to power of 2) + * @pool: pre-allocated pool memory (must be at least stride * capaci= ty) + * @gfp: allocation flags (for slots array only) + * + * The caller owns the pool memory; spsc_destroy will NOT free it. + * Returns 0 on success, negative errno on failure. + */ +static inline int __spsc_init(struct spsc_ring *r, unsigned int elem_size, + unsigned int capacity, void *pool, gfp_t gfp) +{ + unsigned int stride =3D ALIGN(elem_size, SMP_CACHE_BYTES); + unsigned int i; + char *base =3D pool; + + if (elem_size =3D=3D 0 || capacity =3D=3D 0 || !pool) + return -EINVAL; + + capacity =3D roundup_pow_of_two(capacity); + + r->slots =3D kcalloc(capacity, sizeof(void *), gfp); + if (!r->slots) + return -ENOMEM; + + for (i =3D 0; i < capacity; i++) + r->slots[i] =3D base + (unsigned long)stride * i; + + r->mask =3D capacity - 1; + r->head =3D 0; + r->tail =3D 0; + r->acquired =3D 0; + r->pool_page =3D NULL; + r->pool_order =3D 0; + r->elem_stride =3D stride; + + return 0; +} + +/** + * spsc_init - allocate ring and element pool + * @r: pointer to caller-allocated spsc_ring + * @elem_size: size of each element (rounded up to cacheline) + * @capacity: number of elements (rounded up to power of 2) + * @gfp: allocation flags + * + * Returns 0 on success, negative errno on failure. + */ +static inline int spsc_init(struct spsc_ring *r, unsigned int elem_size, + unsigned int capacity, gfp_t gfp) +{ + unsigned int stride =3D ALIGN(elem_size, SMP_CACHE_BYTES); + unsigned long pool_bytes; + unsigned int order; + struct page *page; + + if (elem_size =3D=3D 0 || capacity =3D=3D 0) + return -EINVAL; + + capacity =3D roundup_pow_of_two(capacity); + pool_bytes =3D (unsigned long)stride * capacity; + order =3D get_order(pool_bytes); + + if (pool_bytes > (PAGE_SIZE << order)) + return -EOVERFLOW; + + /* Allocate element pool */ + page =3D alloc_pages(gfp | __GFP_COMP | __GFP_ZERO, order); + if (!page) + return -ENOMEM; + + if (__spsc_init(r, elem_size, capacity, page_address(page), gfp)) { + __free_pages(page, order); + return -ENOMEM; + } + + r->pool_page =3D page; + r->pool_order =3D order; + + return 0; +} + +/** + * spsc_destroy - free ring and element pool + */ +static inline void spsc_destroy(struct spsc_ring *r) +{ + if (r->pool_page) { + __free_pages(r->pool_page, r->pool_order); + r->pool_page =3D NULL; + } + kfree(r->slots); + r->slots =3D NULL; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ +/* Status helpers */ +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ + +static inline unsigned int spsc_capacity(const struct spsc_ring *r) +{ + return r->mask + 1; +} + +static inline unsigned int spsc_elem_size(const struct spsc_ring *r) +{ + return r->elem_stride; +} + +/** spsc_count - total unconsumed entries (including acquired) */ +static inline unsigned int spsc_count(const struct spsc_ring *r) +{ + /* acquire head so a concurrent producer's slot writes are observed */ + return smp_load_acquire(&((struct spsc_ring *)r)->head) - r->tail; +} + +static inline bool spsc_empty(const struct spsc_ring *r) +{ + return spsc_count(r) =3D=3D 0; +} + +static inline bool spsc_full(const struct spsc_ring *r) +{ + return spsc_count(r) > r->mask; +} + +/** spsc_pending - entries acquired but not yet released */ +static inline unsigned int spsc_pending(const struct spsc_ring *r) +{ + return r->acquired - r->tail; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ +/* Producer API (single thread only, shared by both modes) */ +/* */ +/* Two-phase produce: */ +/* 1) spsc_produce() -> get pointer to free element */ +/* 2) caller writes data */ +/* 3) spsc_produce_commit() -> publish to consumer */ +/* */ +/* Or one-shot: spsc_push() for pre-filled elements. */ +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ + +/** + * spsc_produce - reserve one slot and return its element pointer + * @r: ring buffer + * @out: receives pointer to the element to write into + * + * The slot is NOT yet visible to the consumer. Caller must write + * data into *out and then call spsc_produce_commit(). + * + * Returns 0 on success, -ENOSPC if full. + */ +static inline int spsc_produce(struct spsc_ring *r, void **out) +{ + unsigned int head =3D r->head; + unsigned int tail; + + /* acquire tail to observe the slots the consumer has released */ + tail =3D smp_load_acquire(&r->tail); + + if (head - tail > r->mask) + return -ENOSPC; + + *out =3D r->slots[head & r->mask]; + + return 0; +} + +/** + * spsc_produce_commit - publish the previously reserved slot + * @r: ring buffer + * + * Must be called exactly once after each successful spsc_produce(). + */ +static inline void spsc_produce_commit(struct spsc_ring *r) +{ + /* wmb() (sfence on x86) is needed when the pool backing memory + * is mapped Write-Combining (e.g. GPU GTT). WC stores are not + * ordered by x86 TSO, so smp_store_release (compiler barrier) + * alone cannot guarantee the element writes are visible before + * the head update reaches the consumer. + */ + wmb(); + /* release: publish the reserved slot; pairs with the head acquire */ + smp_store_release(&r->head, r->head + 1); +} + +/** + * spsc_produce_n - reserve up to @n slots + * @r: ring buffer + * @out: destination array for element pointers + * @n: max slots to reserve + * @cnt: out - number actually reserved (may be NULL) + * + * Caller must write data into each out[i] and then call + * spsc_produce_commit_n(r, *cnt). + * + * Returns 0 on success, -ENOSPC if zero could be reserved. + */ +static inline int spsc_produce_n(struct spsc_ring *r, void **out, + unsigned int n, unsigned int *cnt) +{ + /* acquire tail to observe the slots the consumer has released */ + unsigned int tail =3D smp_load_acquire(&r->tail); + unsigned int head =3D r->head; + unsigned int free; + unsigned int i; + + free =3D (r->mask + 1) - (head - tail); + + n =3D min(n, free); + if (n =3D=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOSPC; + } + + for (i =3D 0; i < n; i++) + out[i] =3D r->slots[(head + i) & r->mask]; + + if (cnt) + *cnt =3D n; + return 0; +} + +/** + * spsc_produce_commit_n - publish @n previously reserved slots + * @r: ring buffer + * @n: number of slots to publish (must match produce_n count) + */ +static inline void spsc_produce_commit_n(struct spsc_ring *r, unsigned int= n) +{ + /* drain WC element stores before the head update (see commit above) */ + wmb(); + /* release: publish the reserved slots; pairs with the head acquire */ + smp_store_release(&r->head, r->head + n); +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ +/* Mode 1: Sliding window consumer (single thread only) */ +/* */ +/* peek -> read from acquired cursor, no cursor movement */ +/* acquire -> advance acquired cursor, return element pointers */ +/* release -> advance tail, slots become reusable by producer */ +/* rewind -> reset acquired back to tail */ +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ + +/** + * spsc_peek - read element pointers from acquired cursor (read-only) + * @r: ring buffer + * @out: destination array for element pointers + * @max: max entries to peek + * @cnt: out - number of entries peeked (may be NULL) + * + * Does NOT move any cursor. + * + * Returns 0 on success, -ENOENT if nothing to peek. + */ +static inline int spsc_peek(struct spsc_ring *r, void **out, unsigned int = max, + unsigned int *cnt) +{ + /* acquire the producer's head; slots it published are now visible */ + unsigned int head =3D smp_load_acquire(&r->head); + unsigned int acq =3D r->acquired; + unsigned int avail; + unsigned int i; + + avail =3D head - acq; + avail =3D min(avail, max); + + if (avail =3D=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOENT; + } + + for (i =3D 0; i < avail; i++) + out[i] =3D r->slots[(acq + i) & r->mask]; + + if (cnt) + *cnt =3D avail; + return 0; +} + +/** + * spsc_peek_at - peek starting at an offset past the acquired cursor + * @r: ring buffer + * @skip: number of entries to skip past r->acquired + * @out: destination array for element pointers + * @max: max entries to peek + * @cnt: out - number of entries peeked (may be NULL) + * + * Lets the consumer stage a second batch past entries that have been + * read by a previous peek but not yet committed via spsc_acquire. + * Does NOT move any cursor. The caller is responsible for tracking + * the cumulative skip across staged batches; when those batches are + * eventually released to the producer via spsc_acquire, pass the same + * count so r->acquired catches up. + * + * Returns 0 on success, -ENOENT if nothing to peek at that offset. + */ +static inline int spsc_peek_at(struct spsc_ring *r, unsigned int skip, + void **out, unsigned int max, unsigned int *cnt) +{ + /* acquire the producer's head; slots it published are now visible */ + unsigned int head =3D smp_load_acquire(&r->head); + unsigned int pos =3D r->acquired + skip; + unsigned int avail; + unsigned int i; + + if ((int)(head - pos) <=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOENT; + } + + avail =3D min(head - pos, max); + for (i =3D 0; i < avail; i++) + out[i] =3D r->slots[(pos + i) & r->mask]; + + if (cnt) + *cnt =3D avail; + return 0; +} + +/** + * spsc_acquire - advance acquired cursor (step 1) + * @r: ring buffer + * @out: destination array for element pointers, or NULL to skip + * @max: max entries to acquire + * @cnt: out - number acquired (may be NULL) + * + * Returns 0 on success, -ENOENT if nothing to acquire. + */ +static inline int spsc_acquire(struct spsc_ring *r, void **out, + unsigned int max, unsigned int *cnt) +{ + /* acquire the producer's head; slots it published are now visible */ + unsigned int head =3D smp_load_acquire(&r->head); + unsigned int acq =3D r->acquired; + unsigned int avail; + unsigned int i; + + avail =3D head - acq; + avail =3D min(avail, max); + + if (avail =3D=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOENT; + } + + if (out) { + for (i =3D 0; i < avail; i++) + out[i] =3D r->slots[(acq + i) & r->mask]; + } + + /* Publish the window to the releasing consumer (a different CPU than + * this acquirer): pair with the smp_load_acquire() in spsc_release() + * so it cannot observe the advanced cursor before the element stores + * (e.g. an accel verdict) those slots now point at. + */ + smp_store_release(&r->acquired, acq + avail); + + if (cnt) + *cnt =3D avail; + return 0; +} + +/** + * spsc_acquire_all - acquire all available entries + */ +static inline int spsc_acquire_all(struct spsc_ring *r, void **out, + unsigned int *cnt) +{ + return spsc_acquire(r, out, UINT_MAX, cnt); +} + +/** + * spsc_release - get element pointers of acquired entries (release step 1) + * @r: ring buffer + * @out: destination array for element pointers, or NULL to skip + * @n: number of entries to release (<=3D pending) + * @cnt: out - number of entries prepared for release (may be NULL) + * + * Returns element pointers for the oldest @n acquired entries but + * does NOT advance tail - the producer still cannot reuse these slots. + * Caller processes the elements, then calls spsc_release_commit() to + * actually free them. + * + * Returns 0 on success, -ENOENT if nothing to release. + */ +static inline int spsc_release(struct spsc_ring *r, void **out, unsigned i= nt n, + unsigned int *cnt) +{ + unsigned int tail =3D r->tail; + unsigned int pending; + unsigned int i; + + /* Pairs with smp_store_release(&r->acquired) in spsc_acquire(): once + * we see the advanced cursor we are guaranteed to see the element + * stores (e.g. the accel verdict) for the slots it exposes. + */ + pending =3D smp_load_acquire(&r->acquired) - tail; + + n =3D min(n, pending); + if (n =3D=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOENT; + } + + if (out) { + for (i =3D 0; i < n; i++) + out[i] =3D r->slots[(tail + i) & r->mask]; + } + + if (cnt) + *cnt =3D n; + return 0; +} + +/** + * spsc_release_commit - advance tail, free slots for producer (release st= ep 2) + * @r: ring buffer + * @n: number of entries to commit (must match prior spsc_release count) + * + * After this call the producer may reuse these slots. + */ +static inline void spsc_release_commit(struct spsc_ring *r, unsigned int n) +{ + /* release: hand the consumed slots back to the producer */ + smp_store_release(&r->tail, r->tail + n); +} + +/** + * spsc_release_all - get all acquired entries' pointers (release step 1) + * @r: ring buffer + * @out: destination array for element pointers, or NULL to skip + * @cnt: out - number of entries prepared for release (may be NULL) + * + * Convenience for spsc_release(r, out, pending, cnt). + * Caller must still call spsc_release_commit(r, *cnt) afterward. + * + * Returns 0 on success, -ENOENT if nothing pending. + */ +static inline int spsc_release_all(struct spsc_ring *r, void **out, + unsigned int *cnt) +{ + return spsc_release(r, out, r->acquired - r->tail, cnt); +} + +/** + * spsc_rewind - undo acquires, reset acquired cursor to tail + */ +static inline void spsc_rewind(struct spsc_ring *r) +{ + r->acquired =3D r->tail; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ +/* Mode 2: Simple push / pop (single thread per side) */ +/* */ +/* One-shot convenience wrappers. */ +/* Do NOT mix with Mode 1 acquire/release on the same instance. */ +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D */ + +/** + * spsc_push - reserve, let caller fill, and publish in one shot + * @r: ring buffer + * @out: receives pointer to the element to write into + * + * Unlike produce/commit, the slot is published immediately. + * Caller must fill *out BEFORE this function returns if another + * thread could consume it - but since this is SPSC with push/pop + * the typical pattern is: + * + * spsc_push(&r, &elem); + * fill(elem); // safe: consumer hasn't seen it yet? + * + * NO - push publishes immediately. Use produce/commit if you need + * to fill before publishing. push is an alias for produce+commit. + * + * Returns 0 on success, -ENOSPC if full. + */ +static inline int spsc_push(struct spsc_ring *r, void **out) +{ + int ret; + + ret =3D spsc_produce(r, out); + if (ret) + return ret; + + spsc_produce_commit(r); + return 0; +} + +/** + * spsc_pop - dequeue one element + * @r: ring buffer + * @out: receives pointer to the consumed element + * + * The element pointer remains valid until the next spsc_push() or + * spsc_produce() reuses that slot. + * + * Returns 0 on success, -ENOENT if empty. + */ +static inline int spsc_pop(struct spsc_ring *r, void **out) +{ + /* acquire the producer's head; slots it published are now visible */ + unsigned int head =3D smp_load_acquire(&r->head); + unsigned int tail =3D r->tail; + + if (tail =3D=3D head) + return -ENOENT; + + *out =3D r->slots[tail & r->mask]; + + r->acquired =3D tail + 1; + /* release: hand the consumed slot back to the producer */ + smp_store_release(&r->tail, tail + 1); + + return 0; +} + +/** + * spsc_pop_n - dequeue up to @n elements + * @r: ring buffer + * @out: destination array for element pointers + * @n: max entries to dequeue + * @cnt: out - number dequeued (may be NULL) + * + * Returns 0 on success, -ENOENT if empty. + */ +static inline int spsc_pop_n(struct spsc_ring *r, void **out, unsigned int= n, + unsigned int *cnt) +{ + /* acquire the producer's head; slots it published are now visible */ + unsigned int head =3D smp_load_acquire(&r->head); + unsigned int tail =3D r->tail; + unsigned int avail; + unsigned int i; + + avail =3D head - tail; + n =3D min(n, avail); + + if (n =3D=3D 0) { + if (cnt) + *cnt =3D 0; + return -ENOENT; + } + + for (i =3D 0; i < n; i++) + out[i] =3D r->slots[(tail + i) & r->mask]; + + r->acquired =3D tail + n; + /* release: hand the consumed slots back to the producer */ + smp_store_release(&r->tail, tail + n); + + if (cnt) + *cnt =3D n; + return 0; +} + +#endif /* _SPSC_RING_H */ diff --git a/include/uapi/linux/knod.h b/include/uapi/linux/knod.h new file mode 100644 index 000000000000..57602685c14e --- /dev/null +++ b/include/uapi/linux/knod.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Cl= ause) */ +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/knod.yaml */ +/* YNL-GEN uapi header */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#ifndef _UAPI_LINUX_KNOD_H +#define _UAPI_LINUX_KNOD_H + +#define KNOD_FAMILY_NAME "knod" +#define KNOD_FAMILY_VERSION 1 + +enum knod_feature { + KNOD_FEATURE_NONE, + KNOD_FEATURE_BPF, + KNOD_FEATURE_IPSEC, +}; + +enum knod_accel_type { + KNOD_ACCEL_TYPE_GPU, + KNOD_ACCEL_TYPE_DPU, +}; + +enum { + KNOD_A_ACCEL_ID =3D 1, + KNOD_A_ACCEL_NAME, + KNOD_A_ACCEL_TYPE, + KNOD_A_ACCEL_FEATURE_CAP, + KNOD_A_ACCEL_FEATURE_ENA, + + __KNOD_A_ACCEL_MAX, + KNOD_A_ACCEL_MAX =3D (__KNOD_A_ACCEL_MAX - 1) +}; + +enum { + KNOD_A_NIC_IFINDEX =3D 1, + KNOD_A_NIC_NAME, + + __KNOD_A_NIC_MAX, + KNOD_A_NIC_MAX =3D (__KNOD_A_NIC_MAX - 1) +}; + +enum { + KNOD_A_DEV_NIC_IFINDEX =3D 1, + KNOD_A_DEV_ACCEL_ID, + + __KNOD_A_DEV_MAX, + KNOD_A_DEV_MAX =3D (__KNOD_A_DEV_MAX - 1) +}; + +enum { + KNOD_CMD_ACCEL_GET =3D 1, + KNOD_CMD_ACCEL_SET, + KNOD_CMD_NIC_GET, + KNOD_CMD_ATTACH, + KNOD_CMD_DETACH, + KNOD_CMD_DEV_GET, + KNOD_CMD_DEV_ADD_NTF, + KNOD_CMD_DEV_DEL_NTF, + + __KNOD_CMD_MAX, + KNOD_CMD_MAX =3D (__KNOD_CMD_MAX - 1) +}; + +#define KNOD_MCGRP_MGMT "mgmt" + +#endif /* _UAPI_LINUX_KNOD_H */ --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 608BC3B19DC for ; Sun, 19 Jul 2026 18:00:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484009; cv=none; b=aYOfHSciFnxot4NkX1ozT65+F8FO7i9VZ30EGI2VU2ExSqRUxevRwp5NRMCc8+0A7NyTpHflgVGEdLxbd9CtZvHBGArjwvK+PNEkutxTz9nXXT2DXgVcsaUZO+ylUI92i5xy2EXSQ4gdCG6g7Q0PyPDbVvDxNFWD248gjQbXpQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484009; c=relaxed/simple; bh=zKcST3unMT1n6mugJyCc2BBN/+rFXNV5kcDB9rOeHQs=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SPkePwCAOJ0Ll1xC/6yZzDnfB5RnbDujixI1RXpUWGCis21nQ9T2uQKIA23woaew/R6tmrQkTAqpZB6uGjTCZqbranVNMEuU4LcD10cJ53zzZrXHpDxAOGfzuIt09AuypiTabPJoapd/MGNS5AZIbhvxUDMh1TJ77OainN2K/h4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CEkWZ+lg; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CEkWZ+lg" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2ccf2360620so53569215ad.3 for ; Sun, 19 Jul 2026 11:00:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484002; x=1785088802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=93nCpaEZziHyTudSghuLV0J77ekvur0UGewCgMEBZ+0=; b=CEkWZ+lgo++ISB7LSkrV6C0dedvr8ITiVW2vH5afyQovwqomD8i+10NVCQcteixeDy tM4dct1qgJoyehVIF0kLafas9HnAt7i6m9mnje952NJsSCaZ29d1qgCGBubP1gCPe2OC EUGbcLiwdCWQtykJvS/jlsXdQNq360YRZ9EECpm7RCYhnuVjcisGTP2QlrI745wMe1dG /VK6EVW94T73uI07D16ipCObjNL5wvSUlqV8g7d+707yecoAVcC/tIPAhXUMEp0/ejuW aHY4tZshinDuPBMcYkgtpprrXDE2bbsfZRpINWVS7bM386T7NisZyoEqPSOrZK1hP4h4 DzTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484002; x=1785088802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=93nCpaEZziHyTudSghuLV0J77ekvur0UGewCgMEBZ+0=; b=YNEbgn4cA7mDcKrbxZ9NBaCHZBV6C84wZLrpmCxpjjgXfp1eygCVbGjeYB5oCF7/Y/ xslcZhMFTuXwW9E9xGiPyT0/ylDVqwy8ll7BaPNt+3TJgrVohiK2dQ+qo2rKTf4Ydic2 aGoPB+bsQ0L82A2Cg/GInbssnJT5FVymEinDW4y36rNLJgpLXzGnFhTtgCmWPERCzC5R Ss9vZPcyVDSPnJXwk9bBrfZOoBZkoY+f+4//PKCOifGBDW31VE1St0HEzL5NVC4grxjh p5Iqut65YR/9T78iBfz9JX9G7FgUvU3z3BJB0keJxNSjEnDMAVnLqHJP9yDRS3qAP0Qy N4eQ== X-Forwarded-Encrypted: i=1; AHgh+RpX2rNLQcbsp17r9wwTfIj0sk0U/e61fWy2aZ7nzCs/Kkr+Uz6OTYnv6ViRUlHLeRs0laz+TdnKKFq6rcg=@vger.kernel.org X-Gm-Message-State: AOJu0YzfcU6bocPwqZyY4CLTSWaBasES7D7y5svT+svRJY1BaiqPHng7 FvVJKirMet2UVFP/6ZwHnStjMlATqrpsZfbC8foxI/V9k6UCVjeaEcXh X-Gm-Gg: AfdE7ckWZf7EY3G0f3ccltHLDERRBbrMQf3JvM4sWZ+v7WKsCFhivDh0sk8dMCmuhC9 U4+yPkCSvIxmLZdDW6xBiMXishrpMre/an7S1Md8apIi2sWjyX6rDOaQHXzr/dUSLH0hykaPtJg CDI44fzIn5vuhB+QadLwsyZTgQQ/WH9ivkZLc1eNotvL1D69RLjBIJTrk4RPGMxwQnHuRYUxTQs eikf36FVV+byUFD0urDNP5j8FoGfiT0zZ5LOKm+yQb9ty/1iWZmp6tvjCGLir+aWb1ywvIQKrqj o+2dOWj6OwIdKpT5XW1yy/E/aMURuKZSC5AaKKcxXkQfFpIri4BYlwJQ4bsoeEZXQghysQ10/G/ yzkB/1woghZTM8inG9C9iPoDKcuRczCUyU63plWui6UpsXoBl6xLPjr5ZOaZirfkBSQ== X-Received: by 2002:a17:903:46ce:b0:2ca:b300:43b7 with SMTP id d9443c01a7336-2cf348507f5mr113252025ad.4.1784484001910; Sun, 19 Jul 2026 11:00:01 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.10.59.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:00:01 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 02/13] net: devmem: extend memory provider for knod Date: Sun, 19 Jul 2026 17:58:46 +0000 Message-ID: <20260719175857.4071636-3-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend the devmem memory-provider path so a knod accelerator can back a NIC page_pool with accelerator-exported memory (dma-buf), letting the NIC DMA received packets directly into accelerator memory. Signed-off-by: Taehee Yoo (cherry picked from commit d511a8cb3e229f8f5cf060985880d45bd384db87) --- include/net/devmem.h | 58 +++++++++++++ include/net/netmem.h | 9 +++ include/net/page_pool/memory_provider.h | 4 + include/net/page_pool/types.h | 23 +++++- net/core/devmem.c | 103 +++++++++++++++++++----- net/core/devmem.h | 7 +- net/core/page_pool.c | 22 ++++- 7 files changed, 198 insertions(+), 28 deletions(-) create mode 100644 include/net/devmem.h diff --git a/include/net/devmem.h b/include/net/devmem.h new file mode 100644 index 000000000000..f1c3895d7833 --- /dev/null +++ b/include/net/devmem.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Device memory TCP support + * + * Authors: Mina Almasry + * Willem de Bruijn + * Kaiyuan Zhang + * + */ +#ifndef _NET_DEVMEM_H +#define _NET_DEVMEM_H + +#include +#include +#include + +struct device; +struct dma_buf; +struct dma_buf_attach_ops; +struct net_device; +struct net_devmem_dmabuf_binding; +struct netlink_ext_ack; + +#if defined(CONFIG_NET_DEVMEM) +struct net_devmem_dmabuf_binding * +__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev, + struct dma_buf *dmabuf, + enum dma_data_direction direction, + const struct dma_buf_attach_ops *importer_ops, + struct netlink_ext_ack *extack); +int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq= _idx, + struct net_devmem_dmabuf_binding *binding); +void net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *bin= ding); +#else +static inline struct net_devmem_dmabuf_binding * +__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev, + struct dma_buf *dmabuf, + enum dma_data_direction direction, + const struct dma_buf_attach_ops *importer_ops, + struct netlink_ext_ack *extack) +{ + return ERR_PTR(-EOPNOTSUPP); +} + +static inline int +net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx, + struct net_devmem_dmabuf_binding *binding) +{ + return -EOPNOTSUPP; +} + +static inline void +net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding) +{ +} +#endif + +#endif /* _NET_DEVMEM_H */ diff --git a/include/net/netmem.h b/include/net/netmem.h index bccacd21b6c3..3ddfbd37500f 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -127,6 +127,15 @@ static inline void net_iov_init(struct net_iov *niov, niov->type =3D type; } =20 +/* Global page index within the dma-buf, accounting for multi-chunk + * scatter-gather layouts where each chunk owner's niovs start at 0. + */ +static inline unsigned int net_iov_binding_idx(const struct net_iov *niov) +{ + return (net_iov_owner(niov)->base_virtual >> PAGE_SHIFT) + + net_iov_idx(niov); +} + /* netmem */ =20 /** diff --git a/include/net/page_pool/memory_provider.h b/include/net/page_poo= l/memory_provider.h index 255ce4cfd975..4b58a9702fb7 100644 --- a/include/net/page_pool/memory_provider.h +++ b/include/net/page_pool/memory_provider.h @@ -23,6 +23,10 @@ bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_= addr_t addr); void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *nio= v); void net_mp_niov_clear_page_pool(struct net_iov *niov); =20 +void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netm= em, + dma_addr_t addr); +void page_pool_clear_pp_info(netmem_ref netmem); + int netif_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, const struct pp_memory_provider_params *p, struct netlink_ext_ack *extack); diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 03da138722f5..3e866f249768 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -31,8 +31,16 @@ */ #define PP_FLAG_ALLOW_UNREADABLE_NETMEM BIT(3) =20 +/* Driver-managed pool with a directly-supplied memory provider, not bound= to a + * netdev rx queue. Setting this flag requires page_pool_params.mp_ops and + * .mp_priv to both be set. + */ +#define PP_FLAG_CUSTOM_MEMORY_PROVIDER BIT(4) + #define PP_FLAG_ALL (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \ - PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM) + PP_FLAG_SYSTEM_POOL | \ + PP_FLAG_ALLOW_UNREADABLE_NETMEM | \ + PP_FLAG_CUSTOM_MEMORY_PROVIDER) =20 /* Index limit to stay within PP_DMA_INDEX_BITS for DMA indices */ #define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1) @@ -54,11 +62,11 @@ * would have to take a slower code path. */ #if PAGE_SIZE >=3D SZ_64K -#define PP_ALLOC_CACHE_REFILL 4 +#define PP_ALLOC_CACHE_REFILL 256 #elif PAGE_SIZE >=3D SZ_16K -#define PP_ALLOC_CACHE_REFILL 16 +#define PP_ALLOC_CACHE_REFILL 1024 #else -#define PP_ALLOC_CACHE_REFILL 64 +#define PP_ALLOC_CACHE_REFILL 4096 #endif =20 #define PP_ALLOC_CACHE_SIZE (PP_ALLOC_CACHE_REFILL * 2) @@ -67,6 +75,8 @@ struct pp_alloc_cache { netmem_ref cache[PP_ALLOC_CACHE_SIZE]; }; =20 +struct memory_provider_ops; + /** * struct page_pool_params - page pool parameters * @fast: params accessed frequently on hotpath @@ -83,6 +93,9 @@ struct pp_alloc_cache { * @queue_idx: queue idx this page_pool is being created for. * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL, * PP_FLAG_ALLOW_UNREADABLE_NETMEM. + * @mp_ops: driver-supplied memory provider for a pool not bound to a + * netdev rx queue (NULL to use rxq->mp_params instead) + * @mp_priv: context passed to @mp_ops */ struct page_pool_params { struct_group_tagged(page_pool_params_fast, fast, @@ -99,6 +112,8 @@ struct page_pool_params { struct net_device *netdev; unsigned int queue_idx; unsigned int flags; + const struct memory_provider_ops *mp_ops; + void *mp_priv; /* private: used by test code only */ void (*init_callback)(netmem_ref netmem, void *arg); void *init_arg; diff --git a/net/core/devmem.c b/net/core/devmem.c index 957d6b96216b..9e21cffc9643 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -121,12 +121,9 @@ void net_devmem_free_dmabuf(struct net_iov *niov) gen_pool_free(binding->chunk_pool, dma_addr, PAGE_SIZE); } =20 -void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) +static void +net_devmem_binding_unpublish(struct net_devmem_dmabuf_binding *binding) { - struct netdev_rx_queue *rxq; - unsigned long xa_idx; - unsigned int rxq_idx; - xa_erase(&net_devmem_dmabuf_bindings, binding->id); =20 /* Ensure no tx net_devmem_lookup_dmabuf() are in flight after the @@ -136,6 +133,15 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf= _binding *binding) =20 if (binding->list.next) list_del(&binding->list); +} + +void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) +{ + struct netdev_rx_queue *rxq; + unsigned long xa_idx; + unsigned int rxq_idx; + + net_devmem_binding_unpublish(binding); =20 xa_for_each(&binding->bound_rxqs, xa_idx, rxq) { const struct pp_memory_provider_params mp_params =3D { @@ -151,6 +157,47 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf= _binding *binding) percpu_ref_kill(&binding->ref); } =20 +/* Bind/unbind variants for in-kernel offload importers that drive the rx + * queue lifecycle themselves. The mp_params are poked directly, without t= he + * tcp-data-split/XDP guards or the queue reconfigure that the netlink con= trol + * plane applies through netif_mp_open_rxq()/netif_mp_close_rxq(). + */ +int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq= _idx, + struct net_devmem_dmabuf_binding *binding) +{ + struct netdev_rx_queue *rxq; + u32 xa_idx; + int err; + + rxq =3D __netif_get_rx_queue(dev, rxq_idx); + rxq->mp_params.mp_priv =3D binding; + rxq->mp_params.mp_ops =3D &dmabuf_devmem_ops; + + err =3D xa_alloc(&binding->bound_rxqs, &xa_idx, rxq, xa_limit_32b, + GFP_KERNEL); + if (err) { + rxq->mp_params.mp_priv =3D NULL; + rxq->mp_params.mp_ops =3D NULL; + } + + return err; +} + +void net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *bin= ding) +{ + struct netdev_rx_queue *rxq; + unsigned long xa_idx; + + net_devmem_binding_unpublish(binding); + + xa_for_each(&binding->bound_rxqs, xa_idx, rxq) { + rxq->mp_params.mp_priv =3D NULL; + rxq->mp_params.mp_ops =3D NULL; + } + + percpu_ref_kill(&binding->ref); +} + int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, struct net_devmem_dmabuf_binding *binding, struct netlink_ext_ack *extack) @@ -188,12 +235,7 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *v= dev, struct netlink_ext_ack *extack) { struct net_devmem_dmabuf_binding *binding; - static u32 id_alloc_next; - struct scatterlist *sg; struct dma_buf *dmabuf; - unsigned int sg_idx, i; - unsigned long virtual; - int err; =20 if (!dma_dev) { NL_SET_ERR_MSG(extack, "Device doesn't support DMA"); @@ -204,15 +246,39 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *= vdev, if (IS_ERR(dmabuf)) return ERR_CAST(dmabuf); =20 + binding =3D __net_devmem_binding_create(dev, dma_dev, dmabuf, direction, + NULL, extack); + if (IS_ERR(binding)) { + dma_buf_put(dmabuf); + return binding; + } + + binding->vdev =3D vdev; + list_add(&binding->list, &priv->bindings); + + return binding; +} + +struct net_devmem_dmabuf_binding * +__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev, + struct dma_buf *dmabuf, + enum dma_data_direction direction, + const struct dma_buf_attach_ops *importer_ops, + struct netlink_ext_ack *extack) +{ + struct net_devmem_dmabuf_binding *binding; + static u32 id_alloc_next; + struct scatterlist *sg; + unsigned int sg_idx, i; + unsigned long virtual; + int err; + binding =3D kzalloc_node(sizeof(*binding), GFP_KERNEL, dev_to_node(&dev->dev)); - if (!binding) { - err =3D -ENOMEM; - goto err_put_dmabuf; - } + if (!binding) + return ERR_PTR(-ENOMEM); =20 binding->dev =3D dev; - binding->vdev =3D vdev; xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC); =20 err =3D percpu_ref_init(&binding->ref, @@ -226,7 +292,8 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vd= ev, binding->dmabuf =3D dmabuf; binding->direction =3D direction; =20 - binding->attachment =3D dma_buf_attach(binding->dmabuf, dma_dev); + binding->attachment =3D dma_buf_dynamic_attach(binding->dmabuf, dma_dev, + importer_ops, binding); if (IS_ERR(binding->attachment)) { err =3D PTR_ERR(binding->attachment); NL_SET_ERR_MSG(extack, "Failed to bind dmabuf to device"); @@ -325,8 +392,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vd= ev, if (err < 0) goto err_free_chunks; =20 - list_add(&binding->list, &priv->bindings); - return binding; =20 err_free_chunks: @@ -344,8 +409,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vd= ev, percpu_ref_exit(&binding->ref); err_free_binding: kfree(binding); -err_put_dmabuf: - dma_buf_put(dmabuf); return ERR_PTR(err); } =20 diff --git a/net/core/devmem.h b/net/core/devmem.h index 3852a56036cb..6b57837ab454 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -7,9 +7,10 @@ * Kaiyuan Zhang * */ -#ifndef _NET_DEVMEM_H -#define _NET_DEVMEM_H +#ifndef _NET_CORE_DEVMEM_H +#define _NET_CORE_DEVMEM_H =20 +#include #include #include =20 @@ -240,4 +241,4 @@ net_devmem_iov_binding(const struct net_iov *niov) } #endif =20 -#endif /* _NET_DEVMEM_H */ +#endif /* _NET_CORE_DEVMEM_H */ diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 21dc4a9c8714..fd7444943b9f 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -272,7 +272,17 @@ static int page_pool_init(struct page_pool *pool, =20 xa_init_flags(&pool->dma_mapped, XA_FLAGS_ALLOC1); =20 - if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { + if (pool->slow.flags & PP_FLAG_CUSTOM_MEMORY_PROVIDER) { + /* Driver-managed pool with a directly-supplied memory + * provider, not bound to a netdev rx queue. + */ + if (WARN_ON(!pool->slow.mp_ops || !pool->slow.mp_priv)) { + err =3D -EINVAL; + goto free_ptr_ring; + } + pool->mp_priv =3D pool->slow.mp_priv; + pool->mp_ops =3D pool->slow.mp_ops; + } else if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { netdev_assert_locked(pool->slow.netdev); rxq =3D __netif_get_rx_queue(pool->slow.netdev, pool->slow.queue_idx); @@ -725,6 +735,16 @@ void page_pool_clear_pp_info(netmem_ref netmem) netmem_set_pp(netmem, NULL); } =20 +void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netm= em, + dma_addr_t addr) +{ + netmem_to_nmdesc(netmem)->pp_magic =3D 0; + netmem_to_nmdesc(netmem)->pp =3D NULL; + atomic_long_set(&netmem_to_nmdesc(netmem)->pp_ref_count, 0); + page_pool_set_pp_info(pool, netmem); + page_pool_set_dma_addr_netmem(netmem, addr); +} + static __always_inline void __page_pool_release_netmem_dma(struct page_poo= l *pool, netmem_ref netmem) { --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04C603B2FFB for ; Sun, 19 Jul 2026 18:00:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484019; cv=none; b=VQbWplzu/PPxJDHG4lB6tIrY+fzWRBT42704xH/WnNMGXgzX7QFNIoFbVhWq6u4sm3zJGOwkrfgdw6VX2U/OHqcARy5znzOGJhz06kmYVSqd3ISFP4OSPKL9tqAk6OTKh2yHYKSB21TXvimMb+CVBFqFdNXPdF97o2b/AoDxRYM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484019; c=relaxed/simple; bh=IWl/rQwPGM5RLLR3ae5FmKgiapEO5Ii89mk7y4WpFec=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jNiQyD23eAe9j3DnFeJC9uQDQpozv9PvMYB+GR9a78m2UuanSPGmnDcnlcnk3QrAK/K1CE2EKXOGT2OChh7F2sNo2ynB76QwNdxAbferWAvnjKxaz8Ez70q2+IbLXRnidtwJebYcumoPXd3XO+l5yGRdWZKnpTYcxEs1Ts84w7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gRSyNP3n; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gRSyNP3n" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2ce87c7e3bbso104946805ad.1 for ; Sun, 19 Jul 2026 11:00:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484013; x=1785088813; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=OjFgwGrb1ClX/kve0vswPTA8/2SHmC5jDANL46Jtr8I=; b=gRSyNP3nbOy92ndFyGq6aP6JDaV8aAmOvPKk6h4ZEMEh+9n1MXRhN/KjKJ00rxs7Hd KLoCPSoxZwgxWI8xls30M7eYHu0FOR0MiRmNjhrmI4shqvn0WxeheUew5dXh0r/H5jan 0IxTL/ohLXUqGZMkpFHvdBOTcinVnPoIVaOcMaRr2IPYLFLYYFgLlqHPsqeAwbnsHO1S VZUwdmWN9uuM7msnE8QYvaQyhoc1V2It5aYl8p4y/k/4i9I9UWou3ISu6o61KBH4lxv8 l8qlx0+WDF36IK4OuqpytHCzwq2lJEnG6i620BEfTNIsK2p9RjAtq1NK4kTU4dfFluTF oNqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484013; x=1785088813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=OjFgwGrb1ClX/kve0vswPTA8/2SHmC5jDANL46Jtr8I=; b=K/qhPdod1uEspNgujOsTmU1fnESJ4fmgoZ9mDOurWZkBOrZAMxzBKyUlTJSxSHG+vg clKZUJ6zYLqW7Cgcs42O2uX34/3Xzx1zvTvPnf0YAplXhcNWLG0iKaNShYDYBC0HrAQq t6pwuaZPjCMSPn/roPsxySSG77Yf0tqPiLGR2SXR9kP2bT9G3AgRPJz8+RiX/R41m6nv Sy76ziVicI+Zr1EsdgsQqOaBTjHombORbmUAAh+N0hFSDdm5VjuIRd4qMG1LvXAFoiCX YU5FFJpm4hWTET96dGIJydmTPm/kCxJYVUBTtEE85jgRapChOUCs7/s56roZqZ30rKwh mLmw== X-Forwarded-Encrypted: i=1; AHgh+Rr3z3H3gWhmL/RenCbo8cCVr76pV/FtUNVMa6boaA6WW8B1HfhbkZK/+2MnxB/9eS+W0bj4hSbQd1AQ4AM=@vger.kernel.org X-Gm-Message-State: AOJu0YxbYy8LNxtR2DsihVC5sh/tb5c/vQ8F9km1tbrSikLBBWhuvpqd laPzfcurSO8AeHNOFaUbjfZTJqYaFgTfPUmFhIWz3BRwAE5hjbaySu8D X-Gm-Gg: AfdE7cnwRyxj4LtALlUDg6+6/kgA5fktkliVu9m194sxPh3jVd0ZaPnV65v0lpFTiuC OM8dRn5WQRP6ZUFXHFx7n1JcutsZ+89+rKhosNtuGbW9Kru1TkN7VWwgVycCZgBYPMFf/ezBHNa 7EgbcU9n8JdnyzodO70IFunLjE1St2Qea6JmmkdAz0OPycMctWD5eoKocCIS5AxgVU+2Vg0sCN3 dRQwit9RqPHff8qNXF92SWiGVkBgDLimUpQPhD5Rb4FbGlQtOn8cRp5YzOZuWasTl+YH7jCKh6z axfWxe1tBUFBLyyapkA9WFP4oQvaXhGORmBDtEFkea0Au+96SmsFbOF13v5YrzPsq4wWwF4AE6H DHcJUiZJqCENP6LRWDVY7b4X3fls0MLAJsN59pk1WNPwTYTWdagbF8wfbFwl32FeENw== X-Received: by 2002:a17:903:988:b0:2cc:f5aa:9513 with SMTP id d9443c01a7336-2cf34821a87mr130472895ad.10.1784484013524; Sun, 19 Jul 2026 11:00:13 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.00.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:00:13 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 03/13] net: core: add XDP_MODE_HW offload hook for knod Date: Sun, 19 Jul 2026 17:58:47 +0000 Message-ID: <20260719175857.4071636-4-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Route XDP program install/uninstall to the knod accelerator when a device is attached, so BPF programs run on the accelerator instead of the host. Signed-off-by: Taehee Yoo (cherry picked from commit 9689730acf7d6e4d34cd677e4615be10856c709c) --- net/core/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index 7c21bc0a1e34..5ba1c969029b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -10334,7 +10334,7 @@ static int dev_xdp_install(struct net_device *dev, = enum bpf_xdp_mode mode, return -EBUSY; } =20 - if (dev_get_min_mp_channel_count(dev)) { + if (dev_get_min_mp_channel_count(dev) && mode !=3D XDP_MODE_HW) { NL_SET_ERR_MSG(extack, "unable to install XDP to device using memory pro= vider"); return -EBUSY; } --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E71C93B2FDB for ; Sun, 19 Jul 2026 18:00:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484039; cv=none; b=lzT3WA5PEJpZ+3bkIyTuPDrhiJTaKJVwLGdTGuJ+IAc9nfiecZMj16x1YDLSBs/2TEf3557BfwXepsV1onCbXV6gXcHin5SM4DmZ4By+83BSR/cRxEAmK6N/fqSNgsEo78iWf4W2scpZPzGCi8t+BHOqqZ87rSjJLVx3qM8y6oU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484039; c=relaxed/simple; bh=64dr6/EhjExhAai+doRKHdZG800ipBjWqecG8I8FlME=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KEOFGwYcxqVkLMC/BrjZFv4pK0aD717BUZnKZ6JrLGCh6pm7iZZUTWyWFw468fl6hmnq3v13md2B2ZJmVMbwYXiijRbCsHKCi+tih5PCr6BuIENDtSs6b2W1+eJ7cpIj4N5nvIxYhL+UnTYEzRAFmX6IqvZkR5xBNe7gN2gWpxc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KjBPSF+5; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KjBPSF+5" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2ceb096e675so90503535ad.0 for ; Sun, 19 Jul 2026 11:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484026; x=1785088826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=Oa/inmGTRuGYQ1CvN1g/UdrfMqnsy164WzZKu4NRiN8=; b=KjBPSF+5ZQMdu/70C3DvJm+cM7b5mmWWXenGKJ5RDjgOFF+FPZNRwDXTl2yS6kHChv 8MtWrbC5rSwXGIAQMfuun58H6OKgKnTvt6SSw0GijSE2x5F0/pzadjtkp9PgUzYf8I8J DU7ror25SbH3EGF14+Ht0I3KXm44AzH2SPAWa7oo7y0V2QMJQ2QUpNAiHtbaOgtYHrcC XTK/4yGomcXYjqq4tvaXCzWCEMsfbJIIyNuErl1ePVbRM1dEanSYh/qSXMI8gyQEolje D/lDmTyUB2wODad7UT4KxwfdB19wR2J39/doAlw/zy7APttxnQh3QdUUERTcPXrvyvRQ 6Fdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484026; x=1785088826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=Oa/inmGTRuGYQ1CvN1g/UdrfMqnsy164WzZKu4NRiN8=; b=tGAdO1opRgMClB7773FUpuQbJ9+ALCNvuMO11KVuV7AK039O8o0q2ShUg7/bCr9cJ4 fsFTpW7J0HIDRZPjokmSlsADkpI9aHc2RK0hlqD57rkiynuy4tzeeQSP43URA0+J8rnX Ro570m1I89HGj94s7OyGZoIiwokWDE3U/jRo5Rq0Q5DY0CnHyH+mwNiK0O57T7vo8R58 +eh1bwaWV/3uU1i4bcBpse0LgwjkE7JrCvzmeX5/eu3MKu1Qn3RCX6hmiTweQ+XkXIeK ZEG5vbPtOuAkIslIoSahM+N0gszwYP2e37G4JzF01k/LxMpFyzUbwp7Di9adiweFhARM 51Sg== X-Forwarded-Encrypted: i=1; AHgh+RrMvSo8Bop0cyoJZIfSpNl342kBSaHO/w37n9PjfNuOaUmaWm4dWp0PtCkpbecYkB17I5TDkWQL+g1//oY=@vger.kernel.org X-Gm-Message-State: AOJu0YyMDOceuhfYOtMQ3vX+EREZZsvqcL+G7jLG4QPluowoo3Kj+b3S bHFQTbRCWPuvyZoH2gcL6ayC2xrPruv29NemQSVDK6T59U2Z4Pdc4aRO X-Gm-Gg: AfdE7clyukqV4KadRND+uPuYHySXIfgxvrPwKIMbWvfMAz400gGTrlMY6HdDfNOCy6S rPUSkeviK64nhbRYzMPy2kk81dDRjgRNu7yEPhHYzu8Utpzt8FegZbLymaPGIONgpOOArkVSBN9 eAyyAyNgpX/+Ovi/DD4Cs5QoZpolr75A9x3voWY6MkSOQFaxEdmU+HB3tY+9EIscbbkMuUUcwrH t6DTynbofWi5VaZBN6xxWzakJfILfAu1cZuxVLcIW62XNkoKrqmhZtmLBhkDXpYBekcj9+ojOYy Wv70VxLy7YS+Vjx0Lj6o+YQnQGBsPQ4GetHNXm/zIpivfAG0Wg/ZHAgYirG03cMw2Y6WyGQpHQ4 eOTP2rqZBvAIL1jKJxLNxILuCl8+AnQbLmj2zOhq9RUrjyQArfVokiYUBH4HRMX6mfQ== X-Received: by 2002:a17:902:d490:b0:2ce:faa6:7cbb with SMTP id d9443c01a7336-2cf348325bdmr124065555ad.4.1784484025887; Sun, 19 Jul 2026 11:00:25 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.00.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:00:25 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 04/13] net: knod: add offload device core and control plane Date: Sun, 19 Jul 2026 17:58:48 +0000 Message-ID: <20260719175857.4071636-5-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the KNOD core framework and its genetlink control plane. The core maintains three object types - a NIC-side netdev, an accelerator, and a dynamically created offload device that binds them - and drives the RX data path over per-queue SPSC rings backed by a page_pool of accelerator memory. The genetlink family ("knod") exposes attach/detach, device/nic/accel inventory, and per-accelerator feature selection as a stable uAPI. Signed-off-by: Taehee Yoo (cherry picked from commit f17f82a39be852aa8c520aea7177ce0086f7e811) --- net/Kconfig | 2 + net/Makefile | 1 + net/knod/Kconfig | 5 + net/knod/Makefile | 8 + net/knod/knod-nl-gen.c | 121 ++++ net/knod/knod-nl-gen.h | 31 + net/knod/knod.h | 26 + net/knod/knod_core.c | 1231 ++++++++++++++++++++++++++++++++++++++++ net/knod/knod_nl.c | 406 +++++++++++++ 9 files changed, 1831 insertions(+) create mode 100644 net/knod/Kconfig create mode 100644 net/knod/Makefile create mode 100644 net/knod/knod-nl-gen.c create mode 100644 net/knod/knod-nl-gen.h create mode 100644 net/knod/knod.h create mode 100644 net/knod/knod_core.c create mode 100644 net/knod/knod_nl.c diff --git a/net/Kconfig b/net/Kconfig index e38477393551..e7a2200e92cc 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -72,6 +72,8 @@ config NET_DEVMEM depends on DMA_SHARED_BUFFER depends on PAGE_POOL =20 +source "net/knod/Kconfig" + config NET_SHAPER bool =20 diff --git a/net/Makefile b/net/Makefile index 5b2dd7f07a85..ea4ef5c5906d 100644 --- a/net/Makefile +++ b/net/Makefile @@ -19,6 +19,7 @@ obj-$(CONFIG_TLS) +=3D tls/ obj-$(CONFIG_XFRM) +=3D xfrm/ obj-$(CONFIG_UNIX) +=3D unix/ obj-$(CONFIG_INET_PSP) +=3D psp/ +obj-$(CONFIG_KNOD) +=3D knod/ obj-y +=3D ipv6/ obj-$(CONFIG_PACKET) +=3D packet/ obj-$(CONFIG_NET_KEY) +=3D key/ diff --git a/net/knod/Kconfig b/net/knod/Kconfig new file mode 100644 index 000000000000..d5a6d76d33e3 --- /dev/null +++ b/net/knod/Kconfig @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +config KNOD + def_bool y + depends on NET_DEVMEM diff --git a/net/knod/Makefile b/net/knod/Makefile new file mode 100644 index 000000000000..73abe5c99f8c --- /dev/null +++ b/net/knod/Makefile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Makefile for the KNOD (KFD network offload) control plane. +# + +obj-$(CONFIG_KNOD) +=3D knod.o + +knod-y :=3D knod_core.o knod_nl.o knod-nl-gen.o diff --git a/net/knod/knod-nl-gen.c b/net/knod/knod-nl-gen.c new file mode 100644 index 000000000000..96042f0c15c9 --- /dev/null +++ b/net/knod/knod-nl-gen.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Cl= ause) +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/knod.yaml */ +/* YNL-GEN kernel source */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#include +#include + +#include "knod-nl-gen.h" + +#include + +/* KNOD_CMD_ACCEL_GET - do */ +static const struct nla_policy knod_accel_get_nl_policy[KNOD_A_ACCEL_ID + = 1] =3D { + [KNOD_A_ACCEL_ID] =3D NLA_POLICY_MIN(NLA_U32, 1), +}; + +/* KNOD_CMD_ACCEL_SET - do */ +static const struct nla_policy knod_accel_set_nl_policy[KNOD_A_ACCEL_FEATU= RE_ENA + 1] =3D { + [KNOD_A_ACCEL_ID] =3D NLA_POLICY_MIN(NLA_U32, 1), + [KNOD_A_ACCEL_FEATURE_ENA] =3D NLA_POLICY_MAX(NLA_U32, 2), +}; + +/* KNOD_CMD_NIC_GET - do */ +static const struct nla_policy knod_nic_get_nl_policy[KNOD_A_NIC_IFINDEX += 1] =3D { + [KNOD_A_NIC_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), +}; + +/* KNOD_CMD_ATTACH - do */ +static const struct nla_policy knod_attach_nl_policy[KNOD_A_DEV_ACCEL_ID += 1] =3D { + [KNOD_A_DEV_NIC_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), + [KNOD_A_DEV_ACCEL_ID] =3D NLA_POLICY_MIN(NLA_U32, 1), +}; + +/* KNOD_CMD_DETACH - do */ +static const struct nla_policy knod_detach_nl_policy[KNOD_A_DEV_NIC_IFINDE= X + 1] =3D { + [KNOD_A_DEV_NIC_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), +}; + +/* KNOD_CMD_DEV_GET - do */ +static const struct nla_policy knod_dev_get_nl_policy[KNOD_A_DEV_NIC_IFIND= EX + 1] =3D { + [KNOD_A_DEV_NIC_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), +}; + +/* Ops table for knod */ +static const struct genl_split_ops knod_nl_ops[] =3D { + { + .cmd =3D KNOD_CMD_ACCEL_GET, + .doit =3D knod_nl_accel_get_doit, + .policy =3D knod_accel_get_nl_policy, + .maxattr =3D KNOD_A_ACCEL_ID, + .flags =3D GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_ACCEL_GET, + .dumpit =3D knod_nl_accel_get_dumpit, + .flags =3D GENL_CMD_CAP_DUMP, + }, + { + .cmd =3D KNOD_CMD_ACCEL_SET, + .doit =3D knod_nl_accel_set_doit, + .policy =3D knod_accel_set_nl_policy, + .maxattr =3D KNOD_A_ACCEL_FEATURE_ENA, + .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_NIC_GET, + .doit =3D knod_nl_nic_get_doit, + .policy =3D knod_nic_get_nl_policy, + .maxattr =3D KNOD_A_NIC_IFINDEX, + .flags =3D GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_NIC_GET, + .dumpit =3D knod_nl_nic_get_dumpit, + .flags =3D GENL_CMD_CAP_DUMP, + }, + { + .cmd =3D KNOD_CMD_ATTACH, + .doit =3D knod_nl_attach_doit, + .policy =3D knod_attach_nl_policy, + .maxattr =3D KNOD_A_DEV_ACCEL_ID, + .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_DETACH, + .doit =3D knod_nl_detach_doit, + .policy =3D knod_detach_nl_policy, + .maxattr =3D KNOD_A_DEV_NIC_IFINDEX, + .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_DEV_GET, + .doit =3D knod_nl_dev_get_doit, + .policy =3D knod_dev_get_nl_policy, + .maxattr =3D KNOD_A_DEV_NIC_IFINDEX, + .flags =3D GENL_CMD_CAP_DO, + }, + { + .cmd =3D KNOD_CMD_DEV_GET, + .dumpit =3D knod_nl_dev_get_dumpit, + .flags =3D GENL_CMD_CAP_DUMP, + }, +}; + +static const struct genl_multicast_group knod_nl_mcgrps[] =3D { + [KNOD_NLGRP_MGMT] =3D { "mgmt", }, +}; + +struct genl_family knod_nl_family __ro_after_init =3D { + .name =3D KNOD_FAMILY_NAME, + .version =3D KNOD_FAMILY_VERSION, + .netnsok =3D true, + .parallel_ops =3D true, + .module =3D THIS_MODULE, + .split_ops =3D knod_nl_ops, + .n_split_ops =3D ARRAY_SIZE(knod_nl_ops), + .mcgrps =3D knod_nl_mcgrps, + .n_mcgrps =3D ARRAY_SIZE(knod_nl_mcgrps), +}; diff --git a/net/knod/knod-nl-gen.h b/net/knod/knod-nl-gen.h new file mode 100644 index 000000000000..a12b6a174bed --- /dev/null +++ b/net/knod/knod-nl-gen.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Cl= ause) */ +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/knod.yaml */ +/* YNL-GEN kernel header */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#ifndef _LINUX_KNOD_GEN_H +#define _LINUX_KNOD_GEN_H + +#include +#include + +#include + +int knod_nl_accel_get_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_accel_get_dumpit(struct sk_buff *skb, struct netlink_callback = *cb); +int knod_nl_accel_set_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_nic_get_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_nic_get_dumpit(struct sk_buff *skb, struct netlink_callback *c= b); +int knod_nl_attach_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_detach_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info); +int knod_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback *c= b); + +enum { + KNOD_NLGRP_MGMT, +}; + +extern struct genl_family knod_nl_family; + +#endif /* _LINUX_KNOD_GEN_H */ diff --git a/net/knod/knod.h b/net/knod/knod.h new file mode 100644 index 000000000000..eaff0e82ca23 --- /dev/null +++ b/net/knod/knod.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef _NET_KNOD_KNOD_H +#define _NET_KNOD_KNOD_H + +#include + +#include "knod-nl-gen.h" + +struct net_device; + +/* knod_core.c */ +struct knod_netdev *knod_netdev_lookup(struct net_device *dev); +struct knod_dev *knod_dev_lookup(struct net_device *dev); +struct knod_accel *knod_accel_lookup(int id); +int knod_dev_attach(struct knod_netdev *knetdev, + struct knod_accel *accel); +int knod_dev_detach(struct knod_dev *knodev); + +/* knod_nl.c */ +void knod_nl_notify_dev(struct knod_dev *knodev, u32 cmd); + +#endif /* _NET_KNOD_KNOD_H */ diff --git a/net/knod/knod_core.c b/net/knod/knod_core.c new file mode 100644 index 000000000000..af50a69bca65 --- /dev/null +++ b/net/knod/knod_core.c @@ -0,0 +1,1231 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "knod.h" + +DEFINE_MUTEX(knod_lock); +LIST_HEAD(knod_dev_list); +EXPORT_SYMBOL_GPL(knod_dev_list); +LIST_HEAD(knod_netdev_list); +EXPORT_SYMBOL_GPL(knod_netdev_list); +LIST_HEAD(knod_accel_list); +EXPORT_SYMBOL_GPL(knod_accel_list); + +void knod_dev_lock(void) +{ + mutex_lock(&knod_lock); +} +EXPORT_SYMBOL_GPL(knod_dev_lock); + +void knod_dev_unlock(void) +{ + mutex_unlock(&knod_lock); +} +EXPORT_SYMBOL_GPL(knod_dev_unlock); + +void knod_netdev_register(struct knod_netdev *knetdev) +{ + mutex_lock(&knod_lock); + knetdev->status =3D KNOD_STATUS_FREE; + list_add(&knetdev->list, &knod_netdev_list); + mutex_unlock(&knod_lock); +} +EXPORT_SYMBOL(knod_netdev_register); + +void knod_netdev_unregister(struct knod_netdev *knetdev) +{ + mutex_lock(&knod_lock); + list_del(&knetdev->list); + mutex_unlock(&knod_lock); +} +EXPORT_SYMBOL(knod_netdev_unregister); + +void knod_accel_register(struct knod_accel *accel) +{ + mutex_lock(&knod_lock); + accel->status =3D KNOD_STATUS_FREE; + list_add(&accel->list, &knod_accel_list); + mutex_unlock(&knod_lock); +} +EXPORT_SYMBOL(knod_accel_register); + +void knod_accel_unregister(struct knod_accel *accel) +{ + mutex_lock(&knod_lock); + list_del(&accel->list); + mutex_unlock(&knod_lock); +} +EXPORT_SYMBOL(knod_accel_unregister); + +struct knod_netdev *knod_netdev_lookup(struct net_device *dev) +{ + struct knod_netdev *knetdev; + + list_for_each_entry(knetdev, &knod_netdev_list, list) + if (knetdev->dev =3D=3D dev) + return knetdev; + + return NULL; +} + +struct knod_dev *knod_dev_lookup(struct net_device *dev) +{ + struct knod_dev *knodev; + + list_for_each_entry(knodev, &knod_dev_list, list) + if (knodev->netdev =3D=3D dev) + return knodev; + + return NULL; +} + +struct knod_accel *knod_accel_lookup(int id) +{ + struct knod_accel *accel; + + list_for_each_entry(accel, &knod_accel_list, list) + if (accel->id =3D=3D id) + return accel; + + return NULL; +} + +void knod_dev_start(struct knod_dev *knodev) +{ + /* Interface up: start the active feature's worker. */ + knodev->started =3D true; + if (knodev->accel_ops->dev_start) + knodev->accel_ops->dev_start(knodev); +} +EXPORT_SYMBOL(knod_dev_start); + +void knod_dev_stop(struct knod_dev *knodev) +{ + /* Interface down: stop the worker + drain the GPU in-flight. */ + if (knodev->accel_ops->dev_stop) + knodev->accel_ops->dev_stop(knodev); + knodev->started =3D false; +} +EXPORT_SYMBOL(knod_dev_stop); + +int knod_dev_xdp_install(struct knod_dev *knodev, struct netdev_bpf *xdp) +{ + if (!knodev->accel_ops->xdp_ops || + !knodev->accel_ops->xdp_ops->xdp_install) + return -EOPNOTSUPP; + return knodev->accel_ops->xdp_ops->xdp_install(knodev, xdp); +} +EXPORT_SYMBOL(knod_dev_xdp_install); + +void knod_dev_get_stats64(struct knod_dev *knodev, + struct rtnl_link_stats64 *stats) +{ + struct knod_dev_stats *p; + u32 tx_dropped =3D 0, tx_errors =3D 0; + u64 tx_packets, tx_bytes; + unsigned int start; + int i; + + for_each_possible_cpu(i) { + p =3D per_cpu_ptr(knodev->stats, i); + do { + start =3D u64_stats_fetch_begin(&p->syncp); + tx_packets =3D u64_stats_read(&p->tx_packets); + tx_bytes =3D u64_stats_read(&p->tx_bytes); + } while (u64_stats_fetch_retry(&p->syncp, start)); + + stats->tx_packets +=3D tx_packets; + stats->tx_bytes +=3D tx_bytes; + tx_dropped +=3D READ_ONCE(p->tx_dropped); + tx_errors +=3D READ_ONCE(p->tx_errors); + } + stats->tx_dropped +=3D tx_dropped; + stats->tx_errors +=3D tx_errors; +} +EXPORT_SYMBOL(knod_dev_get_stats64); + +/* + * Wrap a delivery-pool page as a zero-copy head_frag skb: the packet sits= at + * @off (preserved headroom) for @len bytes. Building it linear keeps + * skb->data on the packet so callers can edit headers in place. The page + * recycles to @pool when the skb is freed. On oversize or alloc failure = the + * page is returned to @pool and NULL is returned. @napi selects the NAPI= skb + * cache; callers outside softirq pass false. + */ +struct sk_buff *knod_pass_build_skb(netmem_ref netmem, u16 off, u16 len, + struct page_pool *pool, bool napi) +{ + struct page *pg =3D netmem_to_page(netmem); + struct sk_buff *skb; + + /* Must fit alongside skb_shared_info at the page tail; MTU is capped + * below this, so drop the rare oversized outlier. + */ + if (off + len > SKB_WITH_OVERHEAD(PAGE_SIZE)) { + if (pool) + page_pool_put_full_netmem(pool, netmem, false); + return NULL; + } + + if (napi) + skb =3D napi_build_skb(page_address(pg), PAGE_SIZE); + else + skb =3D build_skb(page_address(pg), PAGE_SIZE); + if (unlikely(!skb)) { + if (pool) + page_pool_put_full_netmem(pool, netmem, false); + return NULL; + } + + skb_mark_for_recycle(skb); + skb_reserve(skb, off); + skb_put(skb, len); + return skb; +} +EXPORT_SYMBOL(knod_pass_build_skb); + +/* + * Device->host copy: the NIC DD hands the PASS bds it accumulated during = its + * single act traversal. Allocate a delivery page per packet, issue the a= ccel + * SDMA (GPU -> page) asynchronously, and queue a descriptor on the per-qu= eue + * pending ring tagged with this batch's fence; knod_d2h_drain delivers th= em + * once the fence lands and recycles the source. Sources that cannot be q= ueued + * (bad len / ring full / pool empty) are recycled here. Returns the count + * queued. Producer and the drain consumer run on the same per-queue NAPI= , so + * the ring is single-threaded; @d2h_lock only guards the shared SDMA subm= it. + */ +int knod_d2h_copy(struct knod_dev *knodev, int napi_index, + struct spsc_pass_bd *bds, int cnt) +{ + struct knod_accel_ops *ops =3D knodev->accel_ops; + struct knod_work_priv *wpriv; + struct page_pool *pool; + bool submitted =3D false; + int i, produced =3D 0; + + if (napi_index < 0 || napi_index >=3D KNOD_SPSC_MAX || !ops->d2h_submit) + goto drop_all; + wpriv =3D &knodev->wpriv[napi_index]; + pool =3D READ_ONCE(wpriv->pass_pool); + if (!pool || !wpriv->pass_pending.slots) + goto drop_all; + + spin_lock(&knodev->d2h_lock); + + for (i =3D 0; i < cnt; i++) { + netmem_ref src =3D bds[i].netmem; + struct knod_pass_desc *desc; + netmem_ref dst; + void *ptr; + u32 fv; + u16 off =3D bds[i].off; + u16 len =3D bds[i].len; + + if (!len || off + len > SKB_WITH_OVERHEAD(PAGE_SIZE)) + goto drop; + if (spsc_produce(&wpriv->pass_pending, &ptr)) + goto drop; + dst =3D page_pool_dev_alloc_netmems(pool); + if (!dst) + goto drop; /* slot left uncommitted, reused next */ + + fv =3D ops->d2h_submit(knodev, + page_pool_get_dma_addr_netmem(dst) + off, + napi_index, bds[i].page_idx, off, len); + if (!fv) { /* SDMA ring full: backpressure drop */ + page_pool_put_full_netmem(pool, dst, false); + goto drop; + } + submitted =3D true; + + desc =3D ptr; + desc->netmem =3D dst; + desc->src =3D src; + desc->off =3D off; + desc->len =3D len; + desc->fence_val =3D fv; + desc->sdma_idx =3D 0; + spsc_produce_commit(&wpriv->pass_pending); + produced++; + continue; +drop: + page_pool_recycle_direct_netmem(netmem_get_pp(src), src); + } + + if (submitted) + ops->d2h_kick(knodev); + spin_unlock(&knodev->d2h_lock); + + /* The copies are async; re-arm the NAPI so the drain runs once the + * SDMA lands -- at low rate, RX traffic alone may not poll again soon. + */ + if (submitted) + knod_napi_kick(wpriv); + return produced; + +drop_all: + for (i =3D 0; i < cnt; i++) + page_pool_recycle_direct_netmem(netmem_get_pp(bds[i].netmem), + bds[i].netmem); + return 0; +} +EXPORT_SYMBOL(knod_d2h_copy); + +/* + * Drain the per-queue pending ring: deliver every descriptor whose batch + * fence has landed (accel_ops->d2h_fence) as a zero-copy head_frag skb fr= om + * the delivery page, recycling the source RX page. Stops at the first + * not-yet-landed descriptor -- the ring is in fence order. Runs on the N= IC + * NAPI, the same thread as the knod_d2h_copy producer. + */ +int knod_d2h_drain(struct knod_dev *knodev, int napi_index, + struct napi_struct *napi, int budget) +{ + void *ptrs[KNOD_DEFAULT_PASS_SLOTS]; + struct knod_work_priv *wpriv; + struct knod_pass_desc *d0; + struct page_pool *pool; + LIST_HEAD(deliver_list); + unsigned int got =3D 0, i, n =3D 0; + int delivered =3D 0; + u32 cur_fence; + + if (napi_index < 0 || napi_index >=3D KNOD_SPSC_MAX || + !knodev->accel_ops->d2h_fence) + return 0; + wpriv =3D &knodev->wpriv[napi_index]; + if (!wpriv->pass_pending.slots) + return 0; + pool =3D READ_ONCE(wpriv->pass_pool); + + if (spsc_peek(&wpriv->pass_pending, ptrs, + min_t(unsigned int, budget, KNOD_DEFAULT_PASS_SLOTS), + &got) < 0 || got =3D=3D 0) + return 0; + + d0 =3D ptrs[0]; + cur_fence =3D knodev->accel_ops->d2h_fence(knodev, d0->sdma_idx); + + for (i =3D 0; i < got; i++) { + struct knod_pass_desc *desc =3D ptrs[i]; + struct sk_buff *skb; + + if ((s32)(cur_fence - desc->fence_val) < 0) + break; /* not landed yet; later descs are newer */ + + /* bpf carries the RX page as src for post-copy recycle; ipsec + * leaves it 0 (the NIC act handler recycles via the bd). + */ + if (desc->src) + page_pool_recycle_direct_netmem( + netmem_get_pp(desc->src), desc->src); + skb =3D knod_pass_build_skb(desc->netmem, desc->off, desc->len, + pool, true); + n++; + if (!skb) + continue; + if (knodev->post_copy) { + if (!knodev->post_copy(knodev, skb, desc, napi_index)) { + kfree_skb(skb); + continue; + } + } else if (likely(skb->len >=3D ETH_HLEN)) { + skb->protocol =3D eth_type_trans(skb, knodev->netdev); + } else { + kfree_skb(skb); + continue; + } + list_add_tail(&skb->list, &deliver_list); + delivered++; + } + + if (n) { + spsc_acquire(&wpriv->pass_pending, NULL, n, NULL); + spsc_release_commit(&wpriv->pass_pending, n); + } + + /* Descriptors whose copy has not landed yet remain queued; re-arm so + * we poll again instead of waiting for the next RX event. + */ + if (spsc_count(&wpriv->pass_pending)) + knod_napi_kick(wpriv); + + if (!list_empty(&deliver_list)) + netif_receive_skb_list(&deliver_list); + return delivered; +} +EXPORT_SYMBOL(knod_d2h_drain); + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * NOD IPsec proxy - bridges standard kernel xfrmdev_ops to + * knod_accel_ipsec_ops on the GPU accelerator. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +#if IS_ENABLED(CONFIG_XFRM_OFFLOAD) + +static struct knod_dev *knod_ipsec_find_xdev(struct net_device *dev) +{ + struct knod_dev *knodev; + + list_for_each_entry(knodev, &knod_dev_list, list) { + if (knodev->netdev =3D=3D dev) + return knodev; + } + return NULL; +} + +static int knod_ipsec_xdo_state_add(struct net_device *dev, + struct xfrm_state *x, + struct netlink_ext_ack *extack) +{ + struct knod_dev *knodev =3D knod_ipsec_find_xdev(dev); + + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_state_add) + return -EOPNOTSUPP; + return knodev->accel_ops->ipsec_ops->xdo_dev_state_add(knodev, x, + extack); +} + +static void knod_ipsec_xdo_state_delete(struct net_device *dev, + struct xfrm_state *x) +{ + struct knod_dev *knodev =3D knod_ipsec_find_xdev(dev); + + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_state_delete) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_state_delete(knodev, x); +} + +static void knod_ipsec_xdo_state_free(struct net_device *dev, + struct xfrm_state *x) +{ + struct knod_dev *knodev =3D knod_ipsec_find_xdev(dev); + + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_state_free) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_state_free(knodev, x); +} + +static bool knod_ipsec_xdo_offload_ok(struct sk_buff *skb, + struct xfrm_state *x) +{ + struct knod_dev *knodev =3D knod_ipsec_find_xdev(x->xso.dev); + + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_offload_ok) + return false; + return knodev->accel_ops->ipsec_ops->xdo_dev_offload_ok(knodev, skb, x); +} + +static void knod_ipsec_xdo_state_advance_esn(struct xfrm_state *x) +{ + struct knod_dev *knodev; + + if (!x->xso.dev) + return; + knodev =3D knod_ipsec_find_xdev(x->xso.dev); + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_state_advance_esn) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_state_advance_esn(knodev, x); +} + +static void knod_ipsec_xdo_state_update_stats(struct xfrm_state *x) +{ + struct knod_dev *knodev; + + if (!x->xso.dev) + return; + knodev =3D knod_ipsec_find_xdev(x->xso.dev); + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_state_update_stats) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_state_update_stats(knodev, x); +} + +static int knod_ipsec_xdo_policy_add(struct xfrm_policy *xp, + struct netlink_ext_ack *extack) +{ + struct knod_dev *knodev; + + if (!xp->xdo.dev) + return -EOPNOTSUPP; + knodev =3D knod_ipsec_find_xdev(xp->xdo.dev); + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_policy_add) + return -EOPNOTSUPP; + return knodev->accel_ops->ipsec_ops->xdo_dev_policy_add(knodev, xp, + extack); +} + +static void knod_ipsec_xdo_policy_delete(struct xfrm_policy *xp) +{ + struct knod_dev *knodev; + + if (!xp->xdo.dev) + return; + knodev =3D knod_ipsec_find_xdev(xp->xdo.dev); + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_policy_delete) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_policy_delete(knodev, xp); +} + +static void knod_ipsec_xdo_policy_free(struct xfrm_policy *xp) +{ + struct knod_dev *knodev; + + if (!xp->xdo.dev) + return; + knodev =3D knod_ipsec_find_xdev(xp->xdo.dev); + if (!knodev || !knodev->accel_ops->ipsec_ops || + !knodev->accel_ops->ipsec_ops->xdo_dev_policy_free) + return; + knodev->accel_ops->ipsec_ops->xdo_dev_policy_free(knodev, xp); +} + +static const struct xfrmdev_ops knod_ipsec_xfrmdev_ops =3D { + .xdo_dev_state_add =3D knod_ipsec_xdo_state_add, + .xdo_dev_state_delete =3D knod_ipsec_xdo_state_delete, + .xdo_dev_state_free =3D knod_ipsec_xdo_state_free, + .xdo_dev_offload_ok =3D knod_ipsec_xdo_offload_ok, + .xdo_dev_state_advance_esn =3D knod_ipsec_xdo_state_advance_esn, + .xdo_dev_state_update_stats =3D knod_ipsec_xdo_state_update_stats, + .xdo_dev_policy_add =3D knod_ipsec_xdo_policy_add, + .xdo_dev_policy_delete =3D knod_ipsec_xdo_policy_delete, + .xdo_dev_policy_free =3D knod_ipsec_xdo_policy_free, +}; + +int knod_dev_xdp_drain_pass(struct knod_dev *knodev, struct napi_struct *n= api, + int queue_idx, int budget) +{ + if (!knodev || !knodev->accel_ops) + return 0; + /* Common device->host delivery drain. PASS bds were SDMA-copied by + * knod_d2h_copy from the NIC act handler; deliver the ones whose copy + * has landed. + */ + return knod_d2h_drain(knodev, queue_idx, napi, budget); +} +EXPORT_SYMBOL_GPL(knod_dev_xdp_drain_pass); + +int knod_ipsec_attach(struct knod_dev *knodev) +{ + struct knod_accel_ipsec_ops *ops; + + if (!knodev->accel_ops->ipsec_ops) + return 0; + + ops =3D knodev->accel_ops->ipsec_ops; + if (ops->activate) { + int err =3D ops->activate(knodev); + + if (err) { + pr_err("nod: IPsec activate failed on %s (%d)\n", + netdev_name(knodev->netdev), err); + return err; + } + } + + /* Take over xfrmdev_ops, saving the NIC's original so detach can + * restore it (and only clear NETIF_F_HW_ESP if we added it). + */ + knodev->ipsec_orig_xfrmdev_ops =3D knodev->netdev->xfrmdev_ops; + knodev->ipsec_added_hw_esp =3D + !(knodev->netdev->features & NETIF_F_HW_ESP); + knodev->netdev->xfrmdev_ops =3D &knod_ipsec_xfrmdev_ops; + knodev->netdev->features |=3D NETIF_F_HW_ESP; + knodev->netdev->hw_enc_features |=3D NETIF_F_HW_ESP; + + pr_info("nod: IPsec offload enabled on %s\n", + netdev_name(knodev->netdev)); + return 0; +} +EXPORT_SYMBOL(knod_ipsec_attach); + +void knod_ipsec_detach(struct knod_dev *knodev) +{ + struct knod_accel_ipsec_ops *ops =3D knodev->accel_ops->ipsec_ops; + + if (!ops) + return; + if (knodev->netdev->xfrmdev_ops !=3D &knod_ipsec_xfrmdev_ops) + return; + + if (knodev->ipsec_added_hw_esp) { + knodev->netdev->features &=3D ~NETIF_F_HW_ESP; + knodev->netdev->hw_enc_features &=3D ~NETIF_F_HW_ESP; + } + knodev->netdev->xfrmdev_ops =3D knodev->ipsec_orig_xfrmdev_ops; + + if (ops->deactivate) + ops->deactivate(knodev); + + pr_info("nod: IPsec offload disabled on %s\n", + netdev_name(knodev->netdev)); +} +EXPORT_SYMBOL(knod_ipsec_detach); + +#endif /* CONFIG_XFRM_OFFLOAD */ + +/* + * Host-page page_pool memory provider for GPU->host delivery (NOD-private= ). + * Unlike the devmem/dma-buf providers (which hand out unreadable net_iov)= , this + * returns real host-readable pages (a GPU GTT buffer also mapped into sys= tem + * memory), so delivered data becomes skbs on the normal receive path. The + * pages stay owned by the accel and are never returned to the buddy; the + * gen_pool is only the slow-path backing store (page_pool's cache/ring ab= sorb + * the per-packet churn), and the device address it hands out doubles as t= he + * netmem dma_addr (the worker's SDMA destination). Selected via + * page_pool_params.mp_ops in knod_pass_attach(). + */ +static int hostmem_pp_init(struct page_pool *pool) +{ + struct page_pool_hostmem *hm =3D pool->mp_priv; + int err; + + if (pool->p.order !=3D 0) + return -E2BIG; + if (!hm || !hm->pages || !hm->count) + return -EINVAL; + + hm->genpool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); + if (!hm->genpool) + return -ENOMEM; + + err =3D gen_pool_add(hm->genpool, (unsigned long)hm->base_addr, + (size_t)hm->count * PAGE_SIZE, NUMA_NO_NODE); + if (err) { + gen_pool_destroy(hm->genpool); + hm->genpool =3D NULL; + return err; + } + + /* Device addresses are pre-set; page_pool must not DMA-sync them. */ + pool->dma_sync =3D false; + pool->dma_sync_for_cpu =3D false; + + return 0; +} + +static netmem_ref hostmem_pp_alloc_netmems(struct page_pool *pool, gfp_t g= fp) +{ + struct page_pool_hostmem *hm =3D pool->mp_priv; + unsigned long addr; + netmem_ref netmem; + unsigned int idx; + + addr =3D gen_pool_alloc(hm->genpool, PAGE_SIZE); + if (!addr) + return 0; + + idx =3D (addr - hm->base_addr) >> PAGE_SHIFT; + if (WARN_ON_ONCE(idx >=3D hm->count)) { + gen_pool_free(hm->genpool, addr, PAGE_SIZE); + return 0; + } + + netmem =3D page_to_netmem(hm->pages[idx]); + page_pool_provider_set_netmem(pool, netmem, addr); + + pool->pages_state_hold_cnt++; + trace_page_pool_state_hold(pool, netmem, pool->pages_state_hold_cnt); + + return netmem; +} + +static bool hostmem_pp_release_netmem(struct page_pool *pool, netmem_ref n= etmem) +{ + struct page_pool_hostmem *hm =3D pool->mp_priv; + unsigned long addr =3D page_pool_get_dma_addr_netmem(netmem); + + page_pool_clear_pp_info(netmem); + gen_pool_free(hm->genpool, addr, PAGE_SIZE); + + /* Pages are accel-owned: never put_page() them to the buddy. */ + return false; +} + +static void hostmem_pp_destroy(struct page_pool *pool) +{ + struct page_pool_hostmem *hm =3D pool->mp_priv; + + gen_pool_destroy(hm->genpool); + hm->genpool =3D NULL; + + /* The pool has fully drained (inflight =3D=3D 0); tell the owner it may = now + * release the backing pages. The struct itself is owner-allocated. + */ + if (hm->freed) + hm->freed(hm->arg); +} + +static int hostmem_pp_nl_fill(void *mp_priv, struct sk_buff *rsp, + struct netdev_rx_queue *rxq) +{ + /* Nothing provider-specific to report; page_pool_user.c calls this + * unconditionally once mp_ops is set, so it must not be NULL. + */ + return 0; +} + +static const struct memory_provider_ops page_pool_hostmem_ops =3D { + .init =3D hostmem_pp_init, + .alloc_netmems =3D hostmem_pp_alloc_netmems, + .release_netmem =3D hostmem_pp_release_netmem, + .destroy =3D hostmem_pp_destroy, + .nl_fill =3D hostmem_pp_nl_fill, + /* + * .uninstall is only invoked for rxq-bound pools and is + * NULL-checked. + */ +}; + +/* Provider drain callback: fires once a delivery pool has no inflight pag= es. */ +static void knod_pass_drained(void *arg) +{ + struct knod_dev *knodev =3D arg; + + if (atomic_dec_and_test(&knodev->pp_live)) + complete(&knodev->pp_drained); +} + +/* + * Create one GPU->host delivery page_pool per RX queue, backed by a single + * accel-allocated GTT buffer. Mirrors the spsc-pool setup: alloc_mem() h= ands + * back the buffer (kaddr/gaddr/pages/priv); the framework owns the page_p= ools + * and the drain barrier. The hostmem provider hands out the real GTT pag= es + * with their device address as dma_addr, so the worker's SDMA lands direc= tly + * in the page that later becomes the skb frag. page_pool inflight accoun= ting + * then keeps the buffer alive until every delivered skb has drained. + */ +static int knod_pass_attach(struct knod_dev *knodev) +{ + struct page_pool_params pp =3D { + .order =3D 0, + .pool_size =3D KNOD_PASS_SLOTS, + .nid =3D NUMA_NO_NODE, + .dev =3D knodev->netdev->dev.parent, + .dma_dir =3D DMA_FROM_DEVICE, + .max_len =3D PAGE_SIZE, + .flags =3D PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | + PP_FLAG_CUSTOM_MEMORY_PROVIDER, + .mp_ops =3D &page_pool_hostmem_ops, + }; + unsigned int nqueues =3D min(knodev->netdev->num_rx_queues, + KNOD_SPSC_MAX); + struct page **pages; + void *pass_priv; + u64 base_gaddr; + size_t total; + void *kaddr; + int qi; + + /* Accels without GPU memory simply run without zero-copy delivery. */ + if (!knodev->accel_ops->alloc_mem) + return 0; + + total =3D nqueues * KNOD_PASS_SLOTS * PAGE_SIZE; + kaddr =3D knodev->accel_ops->alloc_mem(knodev, total, &base_gaddr, &pages, + &pass_priv); + if (!kaddr) { + pr_err("%s: delivery alloc_mem failed\n", __func__); + return -ENOMEM; + } + if (!pages) { + knodev->accel_ops->free_mem(knodev, pass_priv); + return -ENOMEM; + } + + init_completion(&knodev->pp_drained); + atomic_set(&knodev->pp_live, 0); + spin_lock_init(&knodev->d2h_lock); + + for (qi =3D 0; qi < nqueues; qi++) { + struct knod_work_priv *wpriv =3D &knodev->wpriv[qi]; + int base =3D qi * KNOD_PASS_SLOTS; + + if (spsc_init(&wpriv->pass_pending, + sizeof(struct knod_pass_desc), + KNOD_PASS_SLOTS, GFP_KERNEL)) + goto err_destroy; + + wpriv->pass_hm.pages =3D &pages[base]; + wpriv->pass_hm.count =3D KNOD_PASS_SLOTS; + wpriv->pass_hm.base_addr =3D base_gaddr + (u64)base * PAGE_SIZE; + wpriv->pass_hm.freed =3D knod_pass_drained; + wpriv->pass_hm.arg =3D knodev; + pp.mp_priv =3D &wpriv->pass_hm; + + wpriv->pass_pool =3D page_pool_create(&pp); + if (IS_ERR(wpriv->pass_pool)) { + wpriv->pass_pool =3D NULL; + spsc_destroy(&wpriv->pass_pending); + goto err_destroy; + } + atomic_inc(&knodev->pp_live); + } + + knodev->pass_priv =3D pass_priv; + return 0; + +err_destroy: + while (qi-- > 0) { + spsc_destroy(&knodev->wpriv[qi].pass_pending); + page_pool_destroy(knodev->wpriv[qi].pass_pool); + knodev->wpriv[qi].pass_pool =3D NULL; + } + if (atomic_read(&knodev->pp_live)) + wait_for_completion_timeout(&knodev->pp_drained, + msecs_to_jiffies(5000)); + knodev->accel_ops->free_mem(knodev, pass_priv); + return -ENOMEM; +} + +/* + * Drain a queue's pass_pending ring on teardown. The interface is down s= o no + * new copies are submitted; wait for each queued copy to land, then retur= n its + * pages to the pool instead of delivering. Without this, page_pool_destr= oy() + * stalls on the pages a detach raced against in-flight d2h copies. + */ +static void knod_pass_flush(struct knod_dev *knodev, unsigned int qi) +{ + struct knod_work_priv *wpriv =3D &knodev->wpriv[qi]; + struct page_pool *pool =3D wpriv->pass_pool; + void *ptrs[KNOD_DEFAULT_PASS_SLOTS]; + unsigned int got, i; + + if (!wpriv->pass_pending.slots || !pool) + return; + + while (spsc_peek(&wpriv->pass_pending, ptrs, + KNOD_DEFAULT_PASS_SLOTS, &got) >=3D 0 && got) { + for (i =3D 0; i < got; i++) { + struct knod_pass_desc *desc =3D ptrs[i]; + int spins =3D 1000000; + + while (knodev->accel_ops->d2h_fence && spins-- && + (s32)(knodev->accel_ops->d2h_fence(knodev, + desc->sdma_idx) - desc->fence_val) < 0) + cpu_relax(); + + WARN_ONCE(knodev->accel_ops->d2h_fence && + (s32)(knodev->accel_ops->d2h_fence(knodev, + desc->sdma_idx) - desc->fence_val) < 0, + "knod: d2h fence timeout on pass flush q%u idx%u\n", + qi, desc->sdma_idx); + + if (desc->src) { + struct page_pool *src_pp =3D + netmem_get_pp(desc->src); + + page_pool_recycle_direct_netmem(src_pp, + desc->src); + } + page_pool_put_full_netmem(pool, desc->netmem, false); + } + spsc_acquire(&wpriv->pass_pending, NULL, got, NULL); + spsc_release_commit(&wpriv->pass_pending, got); + } +} + +/* + * Tear down the delivery pools and free the backing buffer. page_pool_de= stroy + * is async, so wait for every pool to drain (knod_pass_drained) before ha= nding + * the buffer back to the accel -- otherwise an inflight skb frag would ou= tlive + * the BO. Idempotent: a no-op for accels that never allocated. + */ +static void knod_pass_detach(struct knod_dev *knodev) +{ + unsigned int qi; + + if (!knodev->pass_priv) + return; + + /* Iterate the full array (like the spsc teardown): destroy every pool + * that was created, so the drain barrier is guaranteed to reach zero. + */ + for (qi =3D 0; qi < KNOD_SPSC_MAX; qi++) { + if (!knodev->wpriv[qi].pass_pool) + continue; + /* Return any queued (now-landed) d2h pages before destroying + * the pool, or page_pool_destroy() stalls on the inflight + * count a detach raced against in-flight d2h copies. + */ + knod_pass_flush(knodev, qi); + spsc_destroy(&knodev->wpriv[qi].pass_pending); + page_pool_destroy(knodev->wpriv[qi].pass_pool); + knodev->wpriv[qi].pass_pool =3D NULL; + } + if (atomic_read(&knodev->pp_live)) + wait_for_completion_timeout(&knodev->pp_drained, + msecs_to_jiffies(5000)); + knodev->accel_ops->free_mem(knodev, knodev->pass_priv); + knodev->pass_priv =3D NULL; +} + +static void knod_dmabuf_move_notify(struct dma_buf_attachment *attach) +{ +} + +static const struct dma_buf_attach_ops knod_dmabuf_attach_ops =3D { + .allow_peer2peer =3D true, + .invalidate_mappings =3D knod_dmabuf_move_notify, +}; + +static int knod_dmabuf_attach(struct knod_dev *knodev) +{ + struct net_device *dev =3D knodev->netdev; + struct net_devmem_dmabuf_binding *binding; + struct dma_buf *dmabuf; + int i, err; + + for (i =3D 0; i < min(dev->num_rx_queues, KNOD_SPSC_MAX); i++) { + dmabuf =3D knodev->wpriv[i].dmabuf; + if (!dmabuf) + continue; + + /* The binding owns a dmabuf reference that + * __net_devmem_dmabuf_binding_free() drops on unbind, mirroring + * the dma_buf_get(fd) in the fd-based net_devmem_bind_dmabuf(). + * We hand it the BO's dmabuf directly, so take that reference + * here -- otherwise the unbind put races the BO free's put and + * underflows the dmabuf file refcount. + */ + get_dma_buf(dmabuf); + binding =3D __net_devmem_binding_create(dev, dev->dev.parent, + dmabuf, DMA_BIDIRECTIONAL, + &knod_dmabuf_attach_ops, + NULL); + if (IS_ERR(binding)) { + dma_buf_put(dmabuf); + err =3D PTR_ERR(binding); + goto err_unwind; + } + + err =3D net_devmem_bind_dmabuf_to_queue_direct(dev, i, binding); + if (err) { + net_devmem_unbind_dmabuf_direct(binding); + goto err_unwind; + } + + knodev->bindings[i] =3D binding; + } + + return 0; + +err_unwind: + while (i-- > 0) { + if (knodev->bindings[i]) { + net_devmem_unbind_dmabuf_direct(knodev->bindings[i]); + knodev->bindings[i] =3D NULL; + } + } + return err; +} + +static void knod_dmabuf_detach(struct knod_dev *knodev) +{ + int i; + + for (i =3D 0; i < KNOD_SPSC_MAX; i++) { + if (!knodev->bindings[i]) + continue; + + net_devmem_unbind_dmabuf_direct(knodev->bindings[i]); + knodev->bindings[i] =3D NULL; + } +} + +int knod_dev_attach(struct knod_netdev *knetdev, struct knod_accel *accel) +{ + struct knod_dev *knodev; + int err =3D -EINVAL, i; + + if (knetdev->status =3D=3D KNOD_STATUS_USED || + accel->status =3D=3D KNOD_STATUS_USED) { + pr_err("knod: %s already attached\n", + netdev_name(knetdev->dev)); + return -EINVAL; + } + + knodev =3D kzalloc(sizeof(struct knod_dev), GFP_KERNEL); + if (!knodev) + return -ENOMEM; + + if (!try_module_get(knetdev->owner)) { + pr_err("knod: NIC driver for %s is unloading\n", + netdev_name(knetdev->dev)); + kfree(knodev); + return -ENODEV; + } + if (!try_module_get(accel->owner)) { + pr_err("knod: accelerator driver is unloading\n"); + module_put(knetdev->owner); + kfree(knodev); + return -ENODEV; + } + + knetdev->accel =3D accel; + knetdev->knodev =3D knodev; + accel->knetdev =3D knetdev; + accel->knodev =3D knodev; + knodev->knetdev =3D knetdev; + knodev->accel =3D accel; + knodev->netdev =3D knetdev->dev; + knodev->accel_ops =3D accel->accel_ops; + knodev->nic_ops =3D knetdev->nic_ops; + mutex_init(&knodev->lock); + + knodev->stats =3D netdev_alloc_pcpu_stats(struct knod_dev_stats); + if (!knodev->stats) { + err =3D -ENOMEM; + pr_err("knod: failed to allocate stats for %s\n", + netdev_name(knetdev->dev)); + goto free_xdev; + } + + knodev->wpriv =3D kmalloc_array(KNOD_SPSC_MAX, + sizeof(struct knod_work_priv), + GFP_KERNEL | __GFP_ZERO); + if (!knodev->wpriv) { + pr_err("knod: failed to allocate work priv for %s\n", + netdev_name(knetdev->dev)); + err =3D -ENOMEM; + goto free_percpu; + } + + netdev_lock(knodev->netdev); + err =3D knodev->nic_ops->attach(knodev); + if (err) { + err =3D -ENOMEM; + pr_err("knod: NIC attach failed on %s\n", + netdev_name(knetdev->dev)); + goto unlock; + } + + err =3D knodev->accel_ops->attach(knodev); + if (err) { + err =3D -ENOMEM; + pr_err("knod: accelerator attach failed on %s\n", + netdev_name(knetdev->dev)); + goto nic_detach; + } + + { + unsigned int nqueues =3D min(knodev->netdev->num_rx_queues, + KNOD_SPSC_MAX); + unsigned int stride =3D ALIGN(sizeof(struct spsc_bd), + SMP_CACHE_BYTES); + unsigned int cap =3D roundup_pow_of_two(KNOD_SPSC_ELEMS_MAX); + size_t pool_size =3D (size_t)stride * cap; + + if (knodev->accel_ops->alloc_mem) { + size_t total =3D pool_size * nqueues; + u64 base_gaddr; + void *base_pool; + void *pool_priv; + + base_pool =3D knodev->accel_ops->alloc_mem(knodev, total, + &base_gaddr, NULL, &pool_priv); + if (!base_pool) { + err =3D -ENOMEM; + pr_err("%s: alloc_mem failed\n", __func__); + goto free_spsc; + } + memset(base_pool, 0, total); + + /* First queue owns the BO, others reference it */ + knodev->wpriv[0].spsc_pool_priv =3D pool_priv; + for (i =3D 0; i < nqueues; i++) { + void *pool =3D base_pool + + (unsigned long)i * pool_size; + + knodev->wpriv[i].spsc_pool_gaddr =3D + base_gaddr + (u64)i * pool_size; + err =3D __spsc_init(&knodev->wpriv[i].spsc_bds, + sizeof(struct spsc_bd), + KNOD_SPSC_ELEMS_MAX, pool, + GFP_KERNEL); + if (err) { + pr_err("%s: spsc_init failed q%d\n", + __func__, i); + goto free_spsc; + } + } + } else { + for (i =3D 0; i < nqueues; i++) { + err =3D spsc_init(&knodev->wpriv[i].spsc_bds, + sizeof(struct spsc_bd), + KNOD_SPSC_ELEMS_MAX, + GFP_KERNEL); + if (err) { + pr_err("%s: spsc_init failed q%d\n", + __func__, i); + goto free_spsc; + } + } + } + } + goto spsc_done; + +free_spsc: + for (i--; i >=3D 0; i--) + spsc_destroy(&knodev->wpriv[i].spsc_bds); + if (knodev->wpriv[0].spsc_pool_priv) + knodev->accel_ops->free_mem(knodev, + knodev->wpriv[0].spsc_pool_priv); + knodev->accel_ops->detach(knodev); + goto nic_detach; +spsc_done: + + err =3D knod_pass_attach(knodev); + if (err) { + pr_err("%s: knod_pass_attach failed (%d)\n", __func__, err); + goto accel_detach; + } + + err =3D knod_dmabuf_attach(knodev); + if (err) { + err =3D -ENOMEM; + pr_err("knod: dmabuf attach failed on %s\n", + netdev_name(knetdev->dev)); + goto accel_detach; + } + + pr_info("knod: %s attached to accel %d\n", + netdev_name(knodev->netdev), accel->id); + list_add(&knodev->list, &knod_dev_list); + knetdev->status =3D KNOD_STATUS_USED; + accel->status =3D KNOD_STATUS_USED; + + if (knodev->accel_ops->xdp_ops && knodev->accel_ops->xdp_ops->init) { + err =3D knodev->accel_ops->xdp_ops->init(knodev); + if (err) { + pr_err("knod: XDP init failed on %s\n", + netdev_name(knetdev->dev)); + goto xdp_err; + } + } + + /* + * Feature offloads (BPF, IPsec) allocate their GPU resources and + * advertise their netdev capabilities only when the feature is + * selected via knod_accel_feature_set(), not at attach. + */ + netdev_unlock(knodev->netdev); + + if (knodev->accel_ops->mp_map) { + err =3D knodev->accel_ops->mp_map(knodev); + if (err) { + pr_err("knod: mp_map failed (%d)\n", err); + goto dmabuf_detach; + } + } + + return err; + +dmabuf_detach: + netdev_lock(knodev->netdev); +xdp_err: + list_del(&knodev->list); + knetdev->status =3D KNOD_STATUS_FREE; + accel->status =3D KNOD_STATUS_FREE; + knod_dmabuf_detach(knodev); +accel_detach: + knod_pass_detach(knodev); + for (i =3D 0; i < KNOD_SPSC_MAX; i++) + spsc_destroy(&knodev->wpriv[i].spsc_bds); + if (knodev->wpriv[0].spsc_pool_priv) + knodev->accel_ops->free_mem(knodev, + knodev->wpriv[0].spsc_pool_priv); + knodev->accel_ops->detach(knodev); +nic_detach: + knodev->nic_ops->detach(knodev); +unlock: + netdev_unlock(knodev->netdev); + kfree(knodev->wpriv); +free_percpu: + free_percpu(knodev->stats); +free_xdev: + /* Drop the accel<->knetdev<->knodev links set above before freeing + * knodev, or a reader (e.g. knod_default_worker via accel->knodev) + * dereferences a dangling pointer after a failed attach. + */ + accel->knodev =3D NULL; + accel->knetdev =3D NULL; + knetdev->knodev =3D NULL; + knetdev->accel =3D NULL; + module_put(accel->owner); + module_put(knetdev->owner); + kfree(knodev); + return err; +} + +int knod_dev_detach(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_netdev *knetdev =3D knodev->knetdev; + int i; + + if (netif_running(knodev->netdev)) { + pr_err("knod_dev: interface is up\n"); + return -EINVAL; + } + + netdev_lock(knodev->netdev); + /* + * pre_detach() runs knod_feature_stop()+knod_feature_deactivate(), + * which tears down whatever feature is active and frees its + * resources, so no per-feature teardown is needed here. + */ + if (knodev->accel_ops->pre_detach) + knodev->accel_ops->pre_detach(knodev); + list_del(&knodev->list); + knod_dmabuf_detach(knodev); + knod_pass_detach(knodev); + for (i =3D 0; i < KNOD_SPSC_MAX; i++) + spsc_destroy(&knodev->wpriv[i].spsc_bds); + if (knodev->wpriv[0].spsc_pool_priv) + knodev->accel_ops->free_mem(knodev, + knodev->wpriv[0].spsc_pool_priv); + knodev->nic_ops->detach(knodev); + knodev->accel_ops->detach(knodev); + netdev_unlock(knodev->netdev); + knetdev->status =3D KNOD_STATUS_FREE; + knetdev->accel =3D NULL; + knetdev->knodev =3D NULL; + accel->knetdev =3D NULL; + accel->knodev =3D NULL; + accel->status =3D KNOD_STATUS_FREE; + kfree(knodev->wpriv); + free_percpu(knodev->stats); + kfree(knodev); + + module_put(accel->owner); + module_put(knetdev->owner); + return 0; +} + +static int __init knod_dev_init(void) +{ + return genl_register_family(&knod_nl_family); +} + +core_initcall(knod_dev_init); + diff --git a/net/knod/knod_nl.c b/net/knod/knod_nl.c new file mode 100644 index 000000000000..77799202abcf --- /dev/null +++ b/net/knod/knod_nl.c @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#include +#include +#include +#include +#include + +#include "knod-nl-gen.h" +#include "knod.h" + +/* ---- accel ---- */ + +static int knod_nl_accel_fill(struct knod_accel *accel, + struct sk_buff *rsp, const struct genl_info *info) +{ + u32 ena =3D 0, cap =3D 0; + void *hdr; + + if (accel->accel_ops->feature_get) + accel->accel_ops->feature_get(accel, &ena, &cap); + + hdr =3D genlmsg_iput(rsp, info); + if (!hdr) + return -EMSGSIZE; + + if (nla_put_u32(rsp, KNOD_A_ACCEL_ID, accel->id) || + nla_put_string(rsp, KNOD_A_ACCEL_NAME, accel->name) || + nla_put_u32(rsp, KNOD_A_ACCEL_TYPE, accel->type) || + nla_put_u32(rsp, KNOD_A_ACCEL_FEATURE_CAP, cap) || + nla_put_u32(rsp, KNOD_A_ACCEL_FEATURE_ENA, ena)) { + genlmsg_cancel(rsp, hdr); + return -EMSGSIZE; + } + + genlmsg_end(rsp, hdr); + return 0; +} + +int knod_nl_accel_get_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_accel *accel; + struct sk_buff *rsp; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_ACCEL_ID)) + return -EINVAL; + + rsp =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) + return -ENOMEM; + + mutex_lock(&knod_lock); + accel =3D knod_accel_lookup(nla_get_u32(info->attrs[KNOD_A_ACCEL_ID])); + if (!accel) { + mutex_unlock(&knod_lock); + err =3D -ENODEV; + goto err_free; + } + err =3D knod_nl_accel_fill(accel, rsp, info); + mutex_unlock(&knod_lock); + if (err) + goto err_free; + + return genlmsg_reply(rsp, info); + +err_free: + nlmsg_free(rsp); + return err; +} + +int knod_nl_accel_get_dumpit(struct sk_buff *rsp, struct netlink_callback = *cb) +{ + struct knod_accel *accel; + int idx =3D 0, s_idx =3D cb->args[0]; + + mutex_lock(&knod_lock); + list_for_each_entry(accel, &knod_accel_list, list) { + if (idx < s_idx) + goto cont; + if (knod_nl_accel_fill(accel, rsp, genl_info_dump(cb)) < 0) + break; +cont: + idx++; + } + mutex_unlock(&knod_lock); + + cb->args[0] =3D idx; + return rsp->len; +} + +int knod_nl_accel_set_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_accel *accel; + u32 feature; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_ACCEL_ID) || + GENL_REQ_ATTR_CHECK(info, KNOD_A_ACCEL_FEATURE_ENA)) + return -EINVAL; + + feature =3D nla_get_u32(info->attrs[KNOD_A_ACCEL_FEATURE_ENA]); + + /* + * rtnl serialises the feature switch against attach/detach (which + * also take rtnl) and against the NIC driver's interface up/down + * (knod_dev_start/stop), and lets feature_set touch netdev->features + * and run an expedited synchronize_net(). knod_lock still guards the + * accel list walk (lock order: rtnl -> knod_lock). + */ + rtnl_lock(); + mutex_lock(&knod_lock); + accel =3D knod_accel_lookup(nla_get_u32(info->attrs[KNOD_A_ACCEL_ID])); + if (!accel) { + err =3D -ENODEV; + goto unlock; + } + if (!accel->accel_ops->feature_set) { + err =3D -EOPNOTSUPP; + goto unlock; + } + err =3D accel->accel_ops->feature_set(accel, feature, info->extack); +unlock: + mutex_unlock(&knod_lock); + rtnl_unlock(); + return err; +} + +/* ---- nic ---- */ + +static int knod_nl_nic_fill(struct knod_netdev *knetdev, + struct sk_buff *rsp, const struct genl_info *info) +{ + void *hdr; + + hdr =3D genlmsg_iput(rsp, info); + if (!hdr) + return -EMSGSIZE; + + if (nla_put_u32(rsp, KNOD_A_NIC_IFINDEX, knetdev->dev->ifindex) || + nla_put_string(rsp, KNOD_A_NIC_NAME, netdev_name(knetdev->dev))) { + genlmsg_cancel(rsp, hdr); + return -EMSGSIZE; + } + + genlmsg_end(rsp, hdr); + return 0; +} + +int knod_nl_nic_get_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_netdev *knetdev; + struct net_device *dev; + struct sk_buff *rsp; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_NIC_IFINDEX)) + return -EINVAL; + + rsp =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) + return -ENOMEM; + + rtnl_lock(); + mutex_lock(&knod_lock); + dev =3D __dev_get_by_index(genl_info_net(info), + nla_get_u32(info->attrs[KNOD_A_NIC_IFINDEX])); + knetdev =3D dev ? knod_netdev_lookup(dev) : NULL; + if (!knetdev) { + mutex_unlock(&knod_lock); + rtnl_unlock(); + err =3D -ENODEV; + goto err_free; + } + err =3D knod_nl_nic_fill(knetdev, rsp, info); + mutex_unlock(&knod_lock); + rtnl_unlock(); + if (err) + goto err_free; + + return genlmsg_reply(rsp, info); + +err_free: + nlmsg_free(rsp); + return err; +} + +int knod_nl_nic_get_dumpit(struct sk_buff *rsp, struct netlink_callback *c= b) +{ + struct net *net =3D sock_net(rsp->sk); + struct knod_netdev *knetdev; + int idx =3D 0, s_idx =3D cb->args[0]; + + rtnl_lock(); + mutex_lock(&knod_lock); + list_for_each_entry(knetdev, &knod_netdev_list, list) { + if (idx < s_idx) + goto cont; + if (dev_net(knetdev->dev) !=3D net) + goto cont; + if (knod_nl_nic_fill(knetdev, rsp, genl_info_dump(cb)) < 0) + break; +cont: + idx++; + } + mutex_unlock(&knod_lock); + rtnl_unlock(); + + cb->args[0] =3D idx; + return rsp->len; +} + +/* ---- dev (NIC <-> accel binding) ---- */ + +static int knod_nl_dev_fill(struct knod_dev *knodev, struct sk_buff *rsp, + const struct genl_info *info) +{ + void *hdr; + + hdr =3D genlmsg_iput(rsp, info); + if (!hdr) + return -EMSGSIZE; + + if (nla_put_u32(rsp, KNOD_A_DEV_NIC_IFINDEX, knodev->netdev->ifindex) || + nla_put_u32(rsp, KNOD_A_DEV_ACCEL_ID, knodev->accel->id)) { + genlmsg_cancel(rsp, hdr); + return -EMSGSIZE; + } + + genlmsg_end(rsp, hdr); + return 0; +} + +void knod_nl_notify_dev(struct knod_dev *knodev, u32 cmd) +{ + struct net *net =3D dev_net(knodev->netdev); + struct genl_info info; + struct sk_buff *ntf; + + if (!genl_has_listeners(&knod_nl_family, net, KNOD_NLGRP_MGMT)) + return; + + ntf =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!ntf) + return; + + genl_info_init_ntf(&info, &knod_nl_family, cmd); + if (knod_nl_dev_fill(knodev, ntf, &info)) { + nlmsg_free(ntf); + return; + } + + genlmsg_multicast_netns(&knod_nl_family, net, ntf, 0, + KNOD_NLGRP_MGMT, GFP_KERNEL); +} + +int knod_nl_attach_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_netdev *knetdev; + struct knod_accel *accel; + struct net_device *dev; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_DEV_NIC_IFINDEX) || + GENL_REQ_ATTR_CHECK(info, KNOD_A_DEV_ACCEL_ID)) + return -EINVAL; + + rtnl_lock(); + mutex_lock(&knod_lock); + + dev =3D __dev_get_by_index(genl_info_net(info), + nla_get_u32( + info->attrs[KNOD_A_DEV_NIC_IFINDEX])); + if (!dev) { + NL_SET_ERR_MSG(info->extack, "no such netdevice"); + err =3D -ENODEV; + goto unlock; + } + + knetdev =3D knod_netdev_lookup(dev); + if (!knetdev) { + NL_SET_ERR_MSG(info->extack, "netdevice not registered with knod"); + err =3D -ENODEV; + goto unlock; + } + + if (netif_running(dev)) { + NL_SET_ERR_MSG(info->extack, "bring the netdevice down first"); + err =3D -EBUSY; + goto unlock; + } + + accel =3D knod_accel_lookup( + nla_get_u32(info->attrs[KNOD_A_DEV_ACCEL_ID])); + if (!accel) { + NL_SET_ERR_MSG(info->extack, "no such accelerator"); + err =3D -ENODEV; + goto unlock; + } + + err =3D knod_dev_attach(knetdev, accel); + if (!err) + knod_nl_notify_dev(knetdev->knodev, KNOD_CMD_DEV_ADD_NTF); + +unlock: + mutex_unlock(&knod_lock); + rtnl_unlock(); + return err; +} + +int knod_nl_detach_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_dev *knodev; + struct net_device *dev; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_DEV_NIC_IFINDEX)) + return -EINVAL; + + rtnl_lock(); + + dev =3D __dev_get_by_index(genl_info_net(info), + nla_get_u32( + info->attrs[KNOD_A_DEV_NIC_IFINDEX])); + knodev =3D dev ? knod_dev_lookup(dev) : NULL; + if (!knodev) { + NL_SET_ERR_MSG(info->extack, "netdevice not attached"); + err =3D -ENODEV; + goto unlock; + } + + if (netif_running(dev)) { + NL_SET_ERR_MSG(info->extack, "bring the netdevice down first"); + err =3D -EBUSY; + goto unlock; + } + + knod_nl_notify_dev(knodev, KNOD_CMD_DEV_DEL_NTF); + err =3D knod_dev_detach(knodev); + +unlock: + rtnl_unlock(); + return err; +} + +int knod_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct knod_dev *knodev; + struct net_device *dev; + struct sk_buff *rsp; + int err; + + if (GENL_REQ_ATTR_CHECK(info, KNOD_A_DEV_NIC_IFINDEX)) + return -EINVAL; + + rsp =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) + return -ENOMEM; + + rtnl_lock(); + dev =3D __dev_get_by_index(genl_info_net(info), + nla_get_u32( + info->attrs[KNOD_A_DEV_NIC_IFINDEX])); + knodev =3D dev ? knod_dev_lookup(dev) : NULL; + if (!knodev) { + rtnl_unlock(); + err =3D -ENODEV; + goto err_free; + } + err =3D knod_nl_dev_fill(knodev, rsp, info); + rtnl_unlock(); + if (err) + goto err_free; + + return genlmsg_reply(rsp, info); + +err_free: + nlmsg_free(rsp); + return err; +} + +int knod_nl_dev_get_dumpit(struct sk_buff *rsp, struct netlink_callback *c= b) +{ + struct net *net =3D sock_net(rsp->sk); + struct knod_dev *knodev; + int idx =3D 0, s_idx =3D cb->args[0]; + + rtnl_lock(); + list_for_each_entry(knodev, &knod_dev_list, list) { + if (idx < s_idx) + goto cont; + if (dev_net(knodev->netdev) !=3D net) + goto cont; + if (knod_nl_dev_fill(knodev, rsp, genl_info_dump(cb)) < 0) + break; +cont: + idx++; + } + rtnl_unlock(); + + cb->args[0] =3D idx; + return rsp->len; +} --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CEE13AFCEC for ; Sun, 19 Jul 2026 18:00:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484043; cv=none; b=J0jAC1lQJ5kOuqyw6XhTBj0uR+L0XHgGQtIP0MhRG9U4VRTnIjLVjGvlM2SP6S41VjSZBkpVQezqcj9lmWUN1lrf4WyfpYXXu4VDfo0HYn7eDz7k3k+tlIfvEzuVXXnFNAoQOf9M0ZZ3SjF4NEhyBpMq9pH0q10UILouVZ2OJ64= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484043; c=relaxed/simple; bh=3ZUCrrDmOM7VZSnp6sdBGE5jCnwQNOc5dOPQ5kzHMhY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EYdOZf8LlKbt/KAe9HsPZVdC6T5AwyLRofFbpmyrpqanKfKnhqYYN3uFL3WGspyTKn5EY9q3as8dm1H9rOhZC8Pct35TGlpeBiZmF6u5fo1o/ekM3x6jBTLmHe45g7wpO+jK5Ke1N0rvt1WzagkYOBbD2XbOonAo3zYdDqWlAIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ExhUftIV; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ExhUftIV" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cab973140bso115378345ad.3 for ; Sun, 19 Jul 2026 11:00:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484037; x=1785088837; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=gQVLUSh0YdJH3i5yQlMay6TXyEhugldDrvm9uMupsBE=; b=ExhUftIVu07CRaWkA9btZNonG/ISkzf7n8v+5OSZpmTtSD+vyFNTnHW5vo2MxaRJ2Y 2NFSCdUYT4ZCtPWwf2RYXdU+9Jc5QVP6RyonM+yB72H9vEeeFHKExnMphZBd6YFlzDwj CbASnd+Y0hvBf6wh+bebN+yIauWXQfo/ayG2+LYL2PhyWfH37vvxQ9ecEa/A6JUhX5/u n1+V6Ecc3I67uEAIVYJfhYBkHf736a82LAX7Og+YiwQd1bKONylK9q1J+FU/tkZJ0lI2 jh0qzZ6ZPvCCPZjaWibdfFjz0Npu+Q5WrPhDnVu50XsQe6Nj4QaAwWsZGbY6PhcyCFym l0Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484037; x=1785088837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=gQVLUSh0YdJH3i5yQlMay6TXyEhugldDrvm9uMupsBE=; b=pnQL8UDkHSvWMXEOfVoVmWijWS9QMzNST9tTxI/5zYxDZWdwroGodQ905x+iXN8g9L /EQIsKRI+nXM196mtszXlp0uWqs3w0URnIIUByP+BR3S/CmHjCGXtDUsVaGSvADSUOrM EqmZ137bGhCndROUB9u2CwQEM1zqrlwRzDscNrIERcfe8m578xOraD4ZkdSpHHwaIbjR qYaiiB5EcgaMT/Ioa19nez1vh+HWGoAHSC4BGBUyFOnS5D8oOjicZkUcotELWes5u4oc abIIpJZ7CVxzc8HlpMvr1P0ECkjcy5jt4S9hlBNjZq1L96r0DIvc7eqwWsvHT9fSuEPs lywQ== X-Forwarded-Encrypted: i=1; AHgh+RpBsc/+sVpprYNbPLVuW4pgBzr6ISlIwXujWzQ7hHYd7k0Rk5Jt6qzc1Qh0yfyprWwAGF3Ex/ZJZNPgIa0=@vger.kernel.org X-Gm-Message-State: AOJu0Ywd1t/zuwxSGcZC0nSc8LOnm+qkfIzwGKHz1TDfEP879XP2Q6+F hONOyQvY8kJ6SIeOZh7GOx+QXiyCC9VggkVIU3N9rrQsvtjsl4ickUMl X-Gm-Gg: AfdE7cn5j4ML4BBrl5+EsMbpn7qNVo/oX1sm1Qy5XKhuWlBaNb7kZbDaP0u5l1vTWa0 mgtyOE/3nYZ8l3BKFFi1xIlYR4nUKCSZgUIf536FRZ6QyI67HlimRlpVs3A/2IlmRP/GsdZl7pQ whKol1X5epa5YHh9ck2HM18vMrd4bLliQA1CaWusbUuRbDV7+E6OsNkKX1bfwBYRevWFNwgZy1X 5c5tFn9sU7UBmboE95iOpMlgmicwsC89DC4ckWZ+NXKryBfQTw4UlhjdNI5Dewk+19rfEbmBTMa rPEtyFuvgfVMB9zRsS7LKV2ZEtv3SHKd7H3q88C5HI67g4gY0eY5SME2KjonIa1ZrSs+ZbfldNG KjXMkD02qzs9BlPZF6UckoiRCL/o2a101BqooCwnXD35/VN+Y9LsaSA4IZX6mhLDjHg== X-Received: by 2002:a17:902:fc44:b0:2cc:b480:28ba with SMTP id d9443c01a7336-2cf34a25ab6mr120497085ad.44.1784484037543; Sun, 19 Jul 2026 11:00:37 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.00.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:00:36 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 05/13] bpf: offload: allow PERCPU_ARRAY maps for offloaded programs Date: Sun, 19 Jul 2026 17:58:49 +0000 Message-ID: <20260719175857.4071636-6-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The knod BPF offload keeps hot per-CPU statistics in a PERCPU_ARRAY map that is mirrored into accelerator memory and aggregated back on the host. Permit this map type on the offload path. Signed-off-by: Taehee Yoo (cherry picked from commit c447c9012940d48c1ec20a8ebac8d779e7d5a6fa) --- kernel/bpf/offload.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 0d6f5569588c..36e747a12cbe 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -510,7 +510,8 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_att= r *attr) if (!capable(CAP_SYS_ADMIN)) return ERR_PTR(-EPERM); if (attr->map_type !=3D BPF_MAP_TYPE_ARRAY && - attr->map_type !=3D BPF_MAP_TYPE_HASH) + attr->map_type !=3D BPF_MAP_TYPE_HASH && + attr->map_type !=3D BPF_MAP_TYPE_PERCPU_ARRAY) return ERR_PTR(-EINVAL); =20 offmap =3D bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE); --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B2B03B6356 for ; Sun, 19 Jul 2026 18:00:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484058; cv=none; b=FtznMPhjt/AYzOpC47iKZBxpxwWLTTcnVR6qD2fgjb9LE1XQhBWsYoqpiL5XPuTNXfz/R8j4Hakhe/c8L/6o3YY74c60P3G04K3QEFdfK5NXXkgZp37kdpXfDEF1xhSH68TaBkzbzOZa3yQW63a1L2lC7T8OCtoqXzWTwQKiZgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484058; c=relaxed/simple; bh=NrtEpgxQUFrRc1uTeZZtwhQVm6kBTpOeb6Zgg5an5FE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sMC7RSmz9871g2obp/db8dGNh25i88tfi0HRATI6tNcHCma5dXDDsH19ZDMvEG7Ws0Je/FtoTTDzR+4aisYsqbNdZjcFePNNqsBtuLs2YhIasHjLf1RMkRa+bl71qjTXtC6Dv+jtic1IC+sU3mf26Dn5lDx4CMh4D9eCVUfkg+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UfIbyMaw; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UfIbyMaw" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2ce7d2adef4so87724285ad.3 for ; Sun, 19 Jul 2026 11:00:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484050; x=1785088850; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=xWOKjN21n342CNCWn9TzH0J09vI9mPVEGuCL2DJh7/4=; b=UfIbyMawCJcA7TNUIk4vhveVttM+K31M83v+ZIRteW5hIg/GlEXNM2souZCurveLo8 DjGDBZosXPgqLhymMbkj8xAiMnR+XuRzYKojTMzQWtOy/pjo22PV1+PsXD7Th7h6FqX4 sAMyFPSpwTrAksv7B5K692GtnAr64r8t5S1FXOJX1/aJmsusMY85Pt6kC3lA1wcnUWmv UmLuwk6Acq1B5FUbXHFgYfSfm+BiMwpKNw3dKNjnhf4Ai8LsIv5FH/0vToWVH+g91B6q nUBKW8Yj6z5trdH6n4UCSF0al6dEMa50I+VNS9N4GIgjO6kXJSc72b/vXeDnkQ7PC2ds mogw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484050; x=1785088850; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=xWOKjN21n342CNCWn9TzH0J09vI9mPVEGuCL2DJh7/4=; b=UT4HtMsznvaVAqmJk1XXFxWmWtNsJOkp01s2Beo/3qIHX7oJYgW8oYGKikIfUL8+md 5a6A16KZzyG6lbejQ2CS0lSwfwrPtCbwMnTkC87nx/0MZKfuVr+t5QB8FU34PXFUKZ1F jn3hYUTvXJn8u6pLat/Z0SpezB5txOjjK4PRJkK66EJXVvtv4c7GHcdlbfpdKGBUDZ0d y0lmWgpGMpPFqF8rO3KbIGyZO33RkZOXp/kMacscDYxcqnj/G3XL5AC2PseRq/wnWxMn giOTz8g2Fz54CttW45Fmqv/M2Kf5YPgUE7qjgCR9JPGCYUz3a6esxYIpOwThvR6fF61C sAQg== X-Forwarded-Encrypted: i=1; AHgh+RoGyYRgBiSAfyVpkElPS/sCHT4mbCqlYWN7cJWiy6d7chGdcBpgJU8KRR5oR7Q6yZHVcqElyEJ5QUg4lag=@vger.kernel.org X-Gm-Message-State: AOJu0YxCHUKdp2AywialmyhzX+nksngvUfAHzmAwNzGNV5joFYs74iHM 9Tki50r5vDT7Fq0g98OBJCEBzg9MCNVqjD6O5X0PxYpyTULSdyNNOQfv X-Gm-Gg: AfdE7cl1KVxUca60hBI3pDbJXGefx5dBfZMX5MowR0KC0woY8bFT7tdxSIoY8yEb9/b LudfDvUmo+hgNvssQE5D8a9qa9aElY6cZQkZ0w6AmZdTnbeYEtgnrVzG/V6rY3zqMe09+V0tUAO q1R3hQeE4lE/4EtKll7UMwajUkYY2tckTE/YS15LF+01d1Qrz8mkfJLskDzPIHoM6QfRUSTEzrd xpD2W0Ejsy7Y0TbDqvRWUxv4mDisnia5AJOYnsmkGYTa/wnmdzTXnevp7wpnDOuyFQeECsGnXOD 1eS5uvV3LQNOEujvzktb0GP7RofWaF4+s9azYD03huoiBBnxZcjy5bvpyN/JQBboKIkmTUaD1tM zj41Ei6aigHEEyj4nio3ew/6amkxYLQm/Zxic4YHCo8wGJqBuPkq0s7QkAiBmF3RD/g== X-Received: by 2002:a17:902:c411:b0:2c9:97a9:2096 with SMTP id d9443c01a7336-2cf349edab6mr112140895ad.42.1784484049397; Sun, 19 Jul 2026 11:00:49 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.00.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:00:48 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 06/13] drm/amdkfd: prepare kfd core for the knod provider Date: Sun, 19 Jul 2026 17:58:50 +0000 Message-ID: <20260719175857.4071636-7-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Expose the kfd internals the knod provider needs to build and run GPU queues on behalf of the kernel: kernel-side event waiting, GPUVM allocation for VRAM/GTT, doorbell access, and the HSA queue/packet layout (kfd_hsa.h). No functional change for existing user-mode queue users; the provider itself is added in a following patch. Signed-off-by: Taehee Yoo (cherry picked from commit 7d2c7cbfbb2a4a80052b5793841f6a229271973d) --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 13 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 65 +++ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 + drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 117 ++--- drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 20 +- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 112 ++++- drivers/gpu/drm/amd/amdkfd/kfd_events.h | 4 + drivers/gpu/drm/amd/amdkfd/kfd_hsa.h | 451 ++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_module.c | 1 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 34 ++ drivers/gpu/drm/amd/amdkfd/kfd_process.c | 26 +- 11 files changed, 778 insertions(+), 68 deletions(-) create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_hsa.h diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/a= md/amdgpu/amdgpu_amdkfd.h index e443a7277299..cf906b32eacb 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -328,6 +328,8 @@ int amdgpu_amdkfd_gpuvm_sync_memory( struct amdgpu_device *adev, struct kgd_mem *mem, bool intr); int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem, void **kptr, uint64_t *size); +int amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel(struct kgd_mem *mem, + void **kptr, uint64_t *size); void amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(struct kgd_mem *mem); =20 int amdgpu_amdkfd_map_gtt_bo_to_gart(struct amdgpu_bo *bo, struct amdgpu_b= o **bo_gart); @@ -446,7 +448,6 @@ bool kgd2kfd_vmfault_fast_path(struct amdgpu_device *ad= ev, struct amdgpu_iv_entr bool retry_fault); void kgd2kfd_lock_kfd(void); void kgd2kfd_teardown_processes(struct amdgpu_device *adev); - #else static inline int kgd2kfd_init(void) { @@ -576,5 +577,15 @@ static inline void kgd2kfd_teardown_processes(struct a= mdgpu_device *adev) { } =20 +#endif + +#if defined(CONFIG_HSA_AMD_KNOD) +int knod_init(struct amdgpu_device *adev); +void knod_fini(struct amdgpu_device *adev); +void knod_exit(void); +#else +static inline int knod_init(struct amdgpu_device *adev) { return 0; } +static inline void knod_fini(struct amdgpu_device *adev) { } +static inline void knod_exit(void) { } #endif #endif /* AMDGPU_AMDKFD_H_INCLUDED */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu= /drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 35fe2c974699..79fd0ce7fc24 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -2336,6 +2336,71 @@ int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct = kgd_mem *mem, return ret; } =20 +/** amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel() - Map a VRAM BO for kernel= CPU access + * + * @mem: Buffer object to be mapped for CPU access + * @kptr[out]: pointer in kernel CPU address space + * @size[out]: size of the buffer + * + * Pins the BO and maps it for kernel CPU access. The eviction fence is re= moved + * from the BO, since pinned BOs cannot be evicted. The bo must remain on = the + * validate_list, so the GPU mapping can be restored after a page table was + * evicted. + * + * Return: 0 on success, error code on failure + */ +int amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel(struct kgd_mem *mem, + void **kptr, uint64_t *size) +{ + int ret; + struct amdgpu_bo *bo =3D mem->bo; + + if (amdgpu_ttm_tt_get_usermm(bo->tbo.ttm)) { + pr_err("userptr can't be mapped to kernel\n"); + return -EINVAL; + } + + mutex_lock(&mem->process_info->lock); + + ret =3D amdgpu_bo_reserve(bo, true); + if (ret) { + pr_err("Failed to reserve bo. ret %d\n", ret); + goto bo_reserve_failed; + } + + ret =3D amdgpu_bo_pin(bo, AMDGPU_GEM_DOMAIN_VRAM); + if (ret) { + pr_err("Failed to pin bo. ret %d\n", ret); + goto pin_failed; + } + + ret =3D amdgpu_bo_kmap(bo, kptr); + if (ret) { + pr_err("Failed to map bo to kernel. ret %d\n", ret); + goto kmap_failed; + } + + amdgpu_amdkfd_remove_eviction_fence( + bo, mem->process_info->eviction_fence); + + if (size) + *size =3D amdgpu_bo_size(bo); + + amdgpu_bo_unreserve(bo); + + mutex_unlock(&mem->process_info->lock); + return 0; + +kmap_failed: + amdgpu_bo_unpin(bo); +pin_failed: + amdgpu_bo_unreserve(bo); +bo_reserve_failed: + mutex_unlock(&mem->process_info->lock); + + return ret; +} + /** amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel() - Unmap a GTT BO for kernel= CPU access * * @mem: Buffer object to be unmapped for CPU access diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/= amdgpu/amdgpu_drv.c index 4c0c77eafbd1..aa974929d22a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -2480,6 +2480,8 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, drm_client_setup(adev_to_drm(adev), format); } =20 + knod_init(adev); + ret =3D amdgpu_debugfs_init(adev); if (ret) DRM_ERROR("Creating debugfs files failed (%d).\n", ret); @@ -2540,6 +2542,7 @@ amdgpu_pci_remove(struct pci_dev *pdev) struct drm_device *dev =3D pci_get_drvdata(pdev); struct amdgpu_device *adev =3D drm_to_adev(dev); =20 + knod_fini(adev); amdgpu_ras_eeprom_check_and_recover(adev); amdgpu_xcp_dev_unplug(adev); amdgpu_gmc_prepare_nps_mode_change(adev); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd= /amdkfd/kfd_chardev.c index c7edebd2fd8a..7879a6ffaa76 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -335,32 +335,27 @@ static int set_queue_properties_from_user(struct queu= e_properties *q_properties, return 0; } =20 -static int kfd_ioctl_create_queue(struct file *filep, struct kfd_process *= p, - void *data) +/* + * Create a queue from a kernel-filled queue_properties. @q_properties is + * owned by the caller and must already be populated (the ioctl handler do= es + * this from user args via set_queue_properties_from_user(); in-kernel cal= lers + * fill it directly). On success *@queue_id_out and *@doorbell_offset_out= are + * set. + */ +int kfd_create_queue(struct kfd_process *p, struct queue_properties *q_pro= perties, + u32 gpu_id, u32 *queue_id_out, u64 *doorbell_offset_out) { - struct kfd_ioctl_create_queue_args *args =3D data; struct kfd_node *dev; int err =3D 0; unsigned int queue_id; struct kfd_process_device *pdd; - struct queue_properties q_properties; uint32_t doorbell_offset_in_process =3D 0; =20 - memset(&q_properties, 0, sizeof(struct queue_properties)); - - pr_debug("Creating queue ioctl\n"); - - err =3D set_queue_properties_from_user(&q_properties, args); - if (err) - return err; - - pr_debug("Looking for gpu id 0x%x\n", args->gpu_id); - mutex_lock(&p->mutex); =20 - pdd =3D kfd_process_device_data_by_id(p, args->gpu_id); + pdd =3D kfd_process_device_data_by_id(p, gpu_id); if (!pdd) { - pr_debug("Could not find gpu id 0x%x\n", args->gpu_id); + pr_debug("Could not find gpu id 0x%x\n", gpu_id); err =3D -EINVAL; goto err_pdd; } @@ -372,14 +367,14 @@ static int kfd_ioctl_create_queue(struct file *filep,= struct kfd_process *p, goto err_bind_process; } =20 - if (q_properties.type =3D=3D KFD_QUEUE_TYPE_SDMA_BY_ENG_ID) { + if (q_properties->type =3D=3D KFD_QUEUE_TYPE_SDMA_BY_ENG_ID) { int max_sdma_eng_id =3D kfd_get_num_sdma_engines(dev) + kfd_get_num_xgmi_sdma_engines(dev) - 1; =20 - if (q_properties.sdma_engine_id > max_sdma_eng_id) { + if (q_properties->sdma_engine_id > max_sdma_eng_id) { err =3D -EINVAL; pr_err("sdma_engine_id %i exceeds maximum id of %i\n", - q_properties.sdma_engine_id, max_sdma_eng_id); + q_properties->sdma_engine_id, max_sdma_eng_id); goto err_sdma_engine_id; } } @@ -392,7 +387,7 @@ static int kfd_ioctl_create_queue(struct file *filep, s= truct kfd_process *p, } } =20 - err =3D kfd_queue_acquire_buffers(pdd, &q_properties); + err =3D kfd_queue_acquire_buffers(pdd, q_properties); if (err) { pr_debug("failed to acquire user queue buffers\n"); goto err_acquire_queue_buf; @@ -402,42 +397,32 @@ static int kfd_ioctl_create_queue(struct file *filep,= struct kfd_process *p, p->lead_thread->pid, dev->id); =20 - err =3D pqm_create_queue(&p->pqm, dev, &q_properties, &queue_id, + err =3D pqm_create_queue(&p->pqm, dev, q_properties, &queue_id, NULL, NULL, NULL, &doorbell_offset_in_process); if (err !=3D 0) goto err_create_queue; =20 - args->queue_id =3D queue_id; - + *queue_id_out =3D queue_id; =20 /* Return gpu_id as doorbell offset for mmap usage */ - args->doorbell_offset =3D KFD_MMAP_TYPE_DOORBELL; - args->doorbell_offset |=3D KFD_MMAP_GPU_ID(args->gpu_id); + *doorbell_offset_out =3D KFD_MMAP_TYPE_DOORBELL; + *doorbell_offset_out |=3D KFD_MMAP_GPU_ID(gpu_id); if (KFD_IS_SOC15(dev)) /* On SOC15 ASICs, include the doorbell offset within the * process doorbell frame, which is 2 pages. */ - args->doorbell_offset |=3D doorbell_offset_in_process; + *doorbell_offset_out |=3D doorbell_offset_in_process; =20 mutex_unlock(&p->mutex); =20 - pr_debug("Queue id %d was created successfully\n", args->queue_id); - - pr_debug("Ring buffer address =3D=3D 0x%016llX\n", - args->ring_base_address); - - pr_debug("Read ptr address =3D=3D 0x%016llX\n", - args->read_pointer_address); - - pr_debug("Write ptr address =3D=3D 0x%016llX\n", - args->write_pointer_address); + pr_debug("Queue id %d was created successfully\n", queue_id); =20 kfd_dbg_ev_raise(KFD_EC_MASK(EC_QUEUE_NEW), p, dev, queue_id, false, NULL= , 0); return 0; =20 err_create_queue: - kfd_queue_unref_bo_vas(pdd, &q_properties); - kfd_queue_release_buffers(pdd, &q_properties); + kfd_queue_unref_bo_vas(pdd, q_properties); + kfd_queue_release_buffers(pdd, q_properties); err_acquire_queue_buf: err_sdma_engine_id: err_bind_process: @@ -446,6 +431,23 @@ static int kfd_ioctl_create_queue(struct file *filep, = struct kfd_process *p, return err; } =20 +static int kfd_ioctl_create_queue(struct file *filep, struct kfd_process *= p, + void *data) +{ + struct kfd_ioctl_create_queue_args *args =3D data; + struct queue_properties q_properties; + int err; + + memset(&q_properties, 0, sizeof(q_properties)); + + err =3D set_queue_properties_from_user(&q_properties, args); + if (err) + return err; + + return kfd_create_queue(p, &q_properties, args->gpu_id, + &args->queue_id, &args->doorbell_offset); +} + static int kfd_ioctl_destroy_queue(struct file *filp, struct kfd_process *= p, void *data) { @@ -643,7 +645,7 @@ static int kfd_ioctl_set_memory_policy(struct file *fil= ep, return err; } =20 -static int kfd_ioctl_set_trap_handler(struct file *filep, +int kfd_ioctl_set_trap_handler(struct file *filep, struct kfd_process *p, void *data) { struct kfd_ioctl_set_trap_handler_args *args =3D data; @@ -726,7 +728,7 @@ static int kfd_ioctl_get_clock_counters(struct file *fi= lep, =20 =20 static int kfd_ioctl_get_process_apertures(struct file *filp, - struct kfd_process *p, void *data) + struct kfd_process *p, void *data) { struct kfd_ioctl_get_process_apertures_args *args =3D data; struct kfd_process_device_apertures *pAperture; @@ -859,7 +861,7 @@ static int kfd_ioctl_get_process_apertures_new(struct f= ile *filp, } =20 static int kfd_ioctl_create_event(struct file *filp, struct kfd_process *p, - void *data) + void *data) { struct kfd_ioctl_create_event_args *args =3D data; int err; @@ -894,24 +896,24 @@ static int kfd_ioctl_destroy_event(struct file *filp,= struct kfd_process *p, return kfd_event_destroy(p, args->event_id); } =20 -static int kfd_ioctl_set_event(struct file *filp, struct kfd_process *p, - void *data) +int kfd_ioctl_set_event(struct file *filp, struct kfd_process *p, + void *data) { struct kfd_ioctl_set_event_args *args =3D data; =20 return kfd_set_event(p, args->event_id); } =20 -static int kfd_ioctl_reset_event(struct file *filp, struct kfd_process *p, - void *data) +int kfd_ioctl_reset_event(struct file *filp, struct kfd_process *p, + void *data) { struct kfd_ioctl_reset_event_args *args =3D data; =20 return kfd_reset_event(p, args->event_id); } =20 -static int kfd_ioctl_wait_events(struct file *filp, struct kfd_process *p, - void *data) +int kfd_ioctl_wait_events(struct file *filp, struct kfd_process *p, + void *data) { struct kfd_ioctl_wait_events_args *args =3D data; =20 @@ -920,8 +922,9 @@ static int kfd_ioctl_wait_events(struct file *filp, str= uct kfd_process *p, (args->wait_for_all !=3D 0), &args->timeout, &args->wait_result); } -static int kfd_ioctl_set_scratch_backing_va(struct file *filep, - struct kfd_process *p, void *data) + +int kfd_ioctl_set_scratch_backing_va(struct file *filep, + struct kfd_process *p, void *data) { struct kfd_ioctl_set_scratch_backing_va_args *args =3D data; struct kfd_process_device *pdd; @@ -1003,8 +1006,8 @@ static int kfd_ioctl_get_tile_config(struct file *fil= ep, return 0; } =20 -static int kfd_ioctl_acquire_vm(struct file *filep, struct kfd_process *p, - void *data) +int kfd_ioctl_acquire_vm(struct file *filep, struct kfd_process *p, + void *data) { struct kfd_ioctl_acquire_vm_args *args =3D data; struct kfd_process_device *pdd; @@ -1078,8 +1081,8 @@ static int kfd_ioctl_get_available_memory(struct file= *filep, return 0; } =20 -static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep, - struct kfd_process *p, void *data) +int kfd_ioctl_alloc_memory_of_gpu(struct file *filep, + struct kfd_process *p, void *data) { struct kfd_ioctl_alloc_memory_of_gpu_args *args =3D data; struct kfd_process_device *pdd; @@ -1279,8 +1282,8 @@ static int kfd_ioctl_free_memory_of_gpu(struct file *= filep, return ret; } =20 -static int kfd_ioctl_map_memory_to_gpu(struct file *filep, - struct kfd_process *p, void *data) +int kfd_ioctl_map_memory_to_gpu(struct file *filep, + struct kfd_process *p, void *data) { struct kfd_ioctl_map_memory_to_gpu_args *args =3D data; struct kfd_process_device *pdd, *peer_pdd; @@ -1624,8 +1627,8 @@ static int kfd_ioctl_import_dmabuf(struct file *filep, return r; } =20 -static int kfd_ioctl_export_dmabuf(struct file *filep, - struct kfd_process *p, void *data) +int kfd_ioctl_export_dmabuf(struct file *filep, + struct kfd_process *p, void *data) { struct kfd_ioctl_export_dmabuf_args *args =3D data; struct kfd_process_device *pdd; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c b/drivers/gpu/drm/am= d/amdkfd/kfd_doorbell.c index fdcf7f2d1b5b..78b7e956f590 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c @@ -145,10 +145,28 @@ int kfd_doorbell_mmap(struct kfd_node *dev, struct kf= d_process *process, vma->vm_page_prot); } =20 +void __iomem *kfd_kernel_doorbell_mmap(struct kfd_node *dev, + struct kfd_process *process) +{ + phys_addr_t address; + struct kfd_process_device *pdd; + + pdd =3D kfd_get_process_device_data(dev, process); + if (!pdd) + return NULL; + + /* Calculate physical address of doorbell */ + address =3D kfd_get_process_doorbells(pdd); + if (!address) + return NULL; + + return ioremap(address, kfd_doorbell_process_slice(dev->kfd)); +} + =20 /* get kernel iomem pointer for a doorbell */ void __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, - unsigned int *doorbell_off) + unsigned int *doorbell_off) { u32 inx; =20 diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/= amdkfd/kfd_events.c index 81900b49d9d5..91ef4c61a8bc 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c @@ -326,7 +326,7 @@ static bool event_can_be_cpu_signaled(const struct kfd_= event *ev) return ev->type =3D=3D KFD_EVENT_TYPE_SIGNAL; } =20 -static int kfd_event_page_set(struct kfd_process *p, void *kernel_address, +int kfd_event_page_set(struct kfd_process *p, void *kernel_address, uint64_t size, uint64_t user_handle) { struct kfd_signal_page *page; @@ -1068,6 +1068,116 @@ int kfd_wait_on_events(struct kfd_process *p, return ret; } =20 +int kfd_wait_on_events_kernel(struct kfd_process *p, + uint32_t num_events, void __user *data, + bool all, uint32_t *user_timeout_ms, + uint32_t *wait_result) +{ + struct kfd_event_data *events =3D (struct kfd_event_data *) data; + uint32_t i; + int ret =3D 0; + + struct kfd_event_waiter *event_waiters =3D NULL; + long timeout =3D user_timeout_to_jiffies(*user_timeout_ms); + + event_waiters =3D alloc_event_waiters(num_events); + if (!event_waiters) { + ret =3D -ENOMEM; + goto out; + } + + /* Use p->event_mutex here to protect against concurrent creation and + * destruction of events while we initialize event_waiters. + */ + mutex_lock(&p->event_mutex); + + for (i =3D 0; i < num_events; i++) { + struct kfd_event_data event_data; + + memcpy(&event_data, &events[i], sizeof(struct kfd_event_data)); + ret =3D init_event_waiter(p, &event_waiters[i], &event_data); + if (ret) + goto out_unlock; + } + + /* Check condition once. */ + *wait_result =3D test_event_condition(all, num_events, event_waiters); + if (*wait_result =3D=3D KFD_IOC_WAIT_RESULT_COMPLETE) { + ret =3D copy_signaled_event_data(num_events, + event_waiters, events); + goto out_unlock; + } else if (WARN_ON(*wait_result =3D=3D KFD_IOC_WAIT_RESULT_FAIL)) { + /* This should not happen. Events shouldn't be + * destroyed while we're holding the event_mutex + */ + goto out_unlock; + } + + mutex_unlock(&p->event_mutex); + + while (true) { + if (fatal_signal_pending(current)) { + ret =3D -EINTR; + break; + } + + if (signal_pending(current)) { + ret =3D -ERESTARTSYS; + if (*user_timeout_ms !=3D KFD_EVENT_TIMEOUT_IMMEDIATE && + *user_timeout_ms !=3D KFD_EVENT_TIMEOUT_INFINITE) + *user_timeout_ms =3D jiffies_to_msecs( + max(0l, timeout-1)); + break; + } + + /* Set task state to interruptible sleep before + * checking wake-up conditions. A concurrent wake-up + * will put the task back into runnable state. In that + * case schedule_timeout will not put the task to + * sleep and we'll get a chance to re-check the + * updated conditions almost immediately. Otherwise, + * this race condition would lead to a soft hang or a + * very long sleep. + */ + set_current_state(TASK_INTERRUPTIBLE); + + *wait_result =3D test_event_condition(all, num_events, + event_waiters); + if (*wait_result !=3D KFD_IOC_WAIT_RESULT_TIMEOUT) + break; + + if (timeout <=3D 0) + break; + + timeout =3D schedule_timeout(timeout); + } + __set_current_state(TASK_RUNNING); + + mutex_lock(&p->event_mutex); + /* copy_signaled_event_data may sleep. So this has to happen + * after the task state is set back to RUNNING. + * + * The event may also have been destroyed after signaling. So + * copy_signaled_event_data also must confirm that the event + * still exists. Therefore this must be under the p->event_mutex + * which is also held when events are destroyed. + */ + if (!ret && *wait_result =3D=3D KFD_IOC_WAIT_RESULT_COMPLETE) + ret =3D copy_signaled_event_data(num_events, + event_waiters, events); + +out_unlock: + free_waiters(num_events, event_waiters, ret =3D=3D -ERESTARTSYS); + mutex_unlock(&p->event_mutex); +out: + if (ret) + *wait_result =3D KFD_IOC_WAIT_RESULT_FAIL; + else if (*wait_result =3D=3D KFD_IOC_WAIT_RESULT_FAIL) + ret =3D -EIO; + + return ret; +} + int kfd_event_mmap(struct kfd_process *p, struct vm_area_struct *vma) { unsigned long pfn; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.h b/drivers/gpu/drm/amd/= amdkfd/kfd_events.h index 1dc21c13833b..aa6ead122084 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.h @@ -87,5 +87,9 @@ struct kfd_event { extern void kfd_signal_event_interrupt(u32 pasid, uint32_t partial_id, uint32_t valid_id_bits, bool signal_mailbox_updated); +int kfd_wait_on_events_kernel(struct kfd_process *p, + uint32_t num_events, void __user *data, + bool all, uint32_t *user_timeout_ms, + uint32_t *wait_result); =20 #endif diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_hsa.h b/drivers/gpu/drm/amd/amd= kfd/kfd_hsa.h new file mode 100644 index 000000000000..794688c615f0 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_hsa.h @@ -0,0 +1,451 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef KFD_HSA_CODE_H_ +#define KFD_HSA_CODE_H_ + +struct amd_queue_properties { + u32 enable_trap_handler:1, + is_ptr64:1, + enable_trap_handler_debug_sgprs:1, + enable_profiling:1, + use_scratch_once:1, + reserved1:27; +}; + +enum hsa_packet_type { + /* + * Vendor-specific packet. + */ + HSA_PACKET_TYPE_VENDOR_SPECIFIC =3D 0, + /* + * The packet has been processed in the past, but has not been reassigned= to + * the packet processor. A packet processor must not process a packet of = this + * type. All queues support this packet type. + */ + HSA_PACKET_TYPE_INVALID =3D 1, + /* + * Packet used by agents for dispatching jobs to kernel agents. Not all + * queues support packets of this type (see ::hsa_queue_feature_t). + */ + HSA_PACKET_TYPE_KERNEL_DISPATCH =3D 2, + /* + * Packet used by agents to delay processing of subsequent packets, and to + * express complex dependencies between multiple packets. All queues supp= ort + * this packet type. + */ + HSA_PACKET_TYPE_BARRIER_AND =3D 3, + /* + * Packet used by agents for dispatching jobs to agents. Not all + * queues support packets of this type (see ::hsa_queue_feature_t). + */ + HSA_PACKET_TYPE_AGENT_DISPATCH =3D 4, + /* + * Packet used by agents to delay processing of subsequent packets, and to + * express complex dependencies between multiple packets. All queues supp= ort + * this packet type. + */ + HSA_PACKET_TYPE_BARRIER_OR =3D 5 +}; + +enum hsa_packet_header { + /* + * Packet type. The value of this sub-field must be one of + * ::hsa_packet_type_t. If the type is ::HSA_PACKET_TYPE_VENDOR_SPECIFIC,= the + * packet layout is vendor-specific. + */ + HSA_PACKET_HEADER_TYPE =3D 0, + /* + * Barrier bit. If the barrier bit is set, the processing of the current + * packet only launches when all preceding packets (within the same queue= ) are + * complete. + */ + HSA_PACKET_HEADER_BARRIER =3D 8, + /* + * Acquire fence scope. The value of this sub-field determines the scope = and + * type of the memory fence operation applied before the packet enters the + * active phase. An acquire fence ensures that any subsequent global segm= ent + * or image loads by any unit of execution that belongs to a dispatch tha= t has + * not yet entered the active phase on any queue of the same kernel agent, + * sees any data previously released at the scopes specified by the acqui= re + * fence. The value of this sub-field must be one of ::hsa_fence_scope_t. + */ + HSA_PACKET_HEADER_ACQUIRE_FENCE_SCOPE =3D 9, + /* + * Release fence scope, The value of this sub-field determines the scope = and + * type of the memory fence operation applied after kernel completion but + * before the packet is completed. A release fence makes any global segme= nt or + * image data that was stored by any unit of execution that belonged to a + * dispatch that has completed the active phase on any queue of the same + * kernel agent visible in all the scopes specified by the release fence.= The + * value of this sub-field must be one of ::hsa_fence_scope_t. + */ + HSA_PACKET_HEADER_RELEASE_FENCE_SCOPE =3D 11 +}; + +struct code_properties { + /* 4-sgprs */ + u16 enable_sgpr_private_segment_buffer:1, + /* 2-sgprs */ + enable_sgpr_dispatch_ptr:1, + /* 2-sgprs */ + enable_sgpr_queue_ptr:1, + /* 2-sgprs */ + enable_sgpr_kernarg_segment_ptr:1, + /* 2-sgprs */ + enable_sgpr_dispatch_id:1, + /* 2-sgprs */ + enable_sgpr_flat_scratch_init:1, + /* 2-sgprs */ + enable_sgpr_private_segment_size:1, + reserved0:3, + enable_wavefront_size32:1, // gfx10+ + uses_dynamic_stack:1, + reserved1:4; +}; + +struct compute_pgm_rsrc1 { + u32 granulated_workitem_vgpr_count:6, + granulated_wavefront_sgpr_count:4, + priority:2, + float_round_mode_32:2, + float_round_mode_16_64:2, + float_denorm_mode_32:2, + float_denorm_mode_16_64:2, + priv:1, + enable_dx10_clamp:1, + debug_mode:1, + enable_ieee_mode:1, + bulky:1, + cdbg_user:1, + fp16_ovfl:1, /* gfx9+ */ + reserved0:2, + wgp_mode:1, /* gfx10+ */ + mem_ordered:1, /* gfx10+ */ + fwd_progress:1; /* gfx10+ */ +}; + +struct compute_pgm_rsrc2 { + /* 1-sgprs + * enable_sgpr_private_segment_wavefront_offset + */ + u32 enable_private_segment:1, + /* More or equal to enable_sgpr_*. + * Exceed 16 will be ignored. + */ + user_sgpr_count:5, + enable_trap_handler:1, + /* 1-sgpr */ + enable_sgpr_workgroup_id_x:1, + /* 1-sgpr */ + enable_sgpr_workgroup_id_y:1, + /* 1-sgpr */ + enable_sgpr_workgroup_id_z:1, + /* 1-sgpr */ + enable_sgpr_workgroup_info:1, + enable_vgpr_workitem_id:2, + enable_exception_address_watch:1, + enable_exception_memory:1, + granulated_lds_size:9, + enable_exception_ieee_754_fp_invalid_operation:1, + enable_exception_fp_denormal_source:1, + enable_exception_ieee_754_fp_division_by_zero:1, + enable_exception_ieee_754_fp_overflow:1, + enable_exception_ieee_754_fp_underflow:1, + enable_exception_ieee_754_fp_inexact:1, + enable_exception_int_divide_by_zero:1, + reserved0:1; +}; + +struct compute_pgm_rsrc3 { + u32 accum_offset:6, + reserved0:10, + tg_split:1, + reserved1:15; +}; + +struct kernel_descriptor { + u32 group_segment_fixed_size; + u32 private_segment_fixed_size; + u32 kernarg_size; + u8 reserved0[4]; + s64 kernel_code_entry_byte_offset; + u8 reserved1[20]; + struct compute_pgm_rsrc3 compute_pgm_rsrc3; /* GFX10+ and GFX90A+ */ + struct compute_pgm_rsrc1 compute_pgm_rsrc1; + struct compute_pgm_rsrc2 compute_pgm_rsrc2; + struct code_properties code_properties; + u8 reserved2[6]; +}; + +enum hsa_queue_type { + HSA_QUEUE_TYPE_MULTI =3D 0, + HSA_QUEUE_TYPE_SINGLE =3D 1 +}; + +enum hsa_queue_feature { + HSA_QUEUE_FEATURE_KERNEL_DISPATCH =3D 1, + HSA_QUEUE_FEATURE_AGENT_DISPATCH =3D 2 +}; + +struct hsa_kernel_dispatch_packet { + u16 header; + u16 setup; + u16 workgroup_size_x; + u16 workgroup_size_y; + u16 workgroup_size_z; + u16 reserved0; + u32 grid_size_x; + u32 grid_size_y; + u32 grid_size_z; + u32 private_segment_size; + u32 group_segment_size; + u64 kernel_object; + void *kernarg_address; + u64 reserved2; + u64 completion_signal; +}; + +enum amd_signal_kind { + AMD_SIGNAL_KIND_INVALID =3D 0, + AMD_SIGNAL_KIND_USER =3D 1, + AMD_SIGNAL_KIND_DOORBELL =3D -1, + AMD_SIGNAL_KIND_LEGACY_DOORBELL =3D -2 +}; + +/* An AMD Signal object must always be 64 byte aligned to ensure it cannot + * span a page boundary. This is required by CP microcode which optimizes + * access to the structure by only doing a single SUA (System Uniform Addr= ess) + * translation when accessing signal fields. This optimization is used in = GFX8. + */ +struct amd_signal { + u64 kind; + union { + volatile s64 value; + volatile u32 *legacy_hardware_doorbell_ptr; + volatile u64 *hardware_doorbell_ptr; + }; + /* For AMD_SIGNAL_KIND_USER: mailbox address for event notification + * in Signal operations. + */ + u64 event_mailbox_ptr; + /* For AMD_SIGNAL_KIND_USER: event id for event notification in Signal + * operations. + */ + u32 event_id; + u32 reserved1; + /* Start of the AQL packet timestamp, when profiled. */ + u64 start_ts; + /* End of the AQL packet timestamp, when profiled. */ + u64 end_ts; + union { + /* For AMD_SIGNAL_KIND_*DOORBELL: the address of the associated + * amd_queue, otherwise reserved and must be 0. + */ + void *queue_ptr; + u64 reserved2; + }; + u32 reserved3[2]; +} __aligned(64); + +struct hsa_queue { + u32 type; + + u32 features; + + void *base_address; + /* + * Signal object used by the application to indicate the ID of a packet t= hat + * is ready to be processed. The HSA runtime manages the doorbell signal.= If + * the application tries to replace or destroy this signal, the behavior = is + * undefined. + * + * If @a type is ::HSA_QUEUE_TYPE_SINGLE, the doorbell signal value must = be + * updated in a monotonically increasing fashion. If @a type is + * ::HSA_QUEUE_TYPE_MULTI, the doorbell signal value can be updated with = any + * value. + */ + u64 doorbell_signal; + + /* + * Maximum number of packets the queue can hold. Must be a power of 2. + */ + u32 size; + /* Reserved. Must be 0. */ + u32 reserved1; + /* + * Queue identifier, which is unique over the lifetime of the application. + */ + u64 id; + +}; + +enum hsa_fence_scope { + HSA_FENCE_SCOPE_NONE =3D 0, + HSA_FENCE_SCOPE_AGENT =3D 1, + HSA_FENCE_SCOPE_SYSTEM =3D 2 +}; + +struct amd_queue { + struct hsa_queue hsa_queue; + u32 reserved1[4]; + volatile u64 write_dispatch_id; + u32 group_segment_aperture_base_hi; + u32 private_segment_aperture_base_hi; + u32 max_cu_id; + u32 max_wave_id; + volatile u64 max_legacy_doorbell_dispatch_id_plus_1; + volatile u32 legacy_doorbell_lock; + u32 reserved2[9]; + volatile u64 read_dispatch_id; + u32 read_dispatch_id_field_base_byte_offset; + u32 compute_tmpring_size; + u32 scratch_resource_descriptor[4]; + u64 scratch_backing_memory_location; + u64 scratch_backing_memory_byte_size; + u32 scratch_wave64_lane_byte_size; + struct amd_queue_properties queue_properties; + u32 reserved3[2]; + u64 queue_inactive_signal; + u32 reserved4[14]; +} __aligned(64); + +struct hsa_sync_var { + union { + /* pointer to user mode data */ + void *user_data; + /* 64bit compatibility of value */ + u64 user_data_ptr_value; + }; + u64 SyncVarSize; +}; + +enum hsa_event_type { + /* user-mode generated GPU signal */ + HSA_EVENTTYPE_SIGNAL =3D 0, + /* HSA node change (attach/detach) */ + HSA_EVENTTYPE_NODECHANGE =3D 1, + /* HSA device state change( start/stop ) */ + HSA_EVENTTYPE_DEVICESTATECHANGE =3D 2, + /* GPU shader exception event */ + HSA_EVENTTYPE_HW_EXCEPTION =3D 3, + /* GPU SYSCALL with parameter info */ + HSA_EVENTTYPE_SYSTEM_EVENT =3D 4, + /* GPU signal for debugging */ + HSA_EVENTTYPE_DEBUG_EVENT =3D 5, + /* GPU signal for profiling */ + HSA_EVENTTYPE_PROFILE_EVENT =3D 6, + /* GPU signal queue idle state (EOP pm4) */ + HSA_EVENTTYPE_QUEUE_EVENT =3D 7, + /* GPU signal for signaling memory access faults and memory subsystem iss= ues */ + HSA_EVENTTYPE_MEMORY =3D 8, + /* ... */ + HSA_EVENTTYPE_MAXID, + HSA_EVENTTYPE_TYPE_SIZE =3D 0xffffffff +}; + +enum hsa_eventtype_nodechange_flags { + HSA_EVENTTYPE_NODECHANGE_ADD =3D 0, + HSA_EVENTTYPE_NODECHANGE_REMOVE =3D 1, + HSA_EVENTTYPE_NODECHANGE_SIZE =3D 0xffffffff +}; + +struct hsa_node_change { + /* HSA node added/removed on the platform */ + enum hsa_eventtype_nodechange_flags flags; +}; + +enum hsa_device { + HSA_DEVICE_CPU =3D 0, + HSA_DEVICE_GPU =3D 1, + MAX_HSA_DEVICE =3D 2 +}; + +enum hsa_eventtype_devicestatechange_flags { + /* device started (and available) */ + HSA_EVENTTYPE_DEVICESTATUSCHANGE_START =3D 0, + /* device stopped (i.e. unavailable) */ + HSA_EVENTTYPE_DEVICESTATUSCHANGE_STOP =3D 1, + HSA_EVENTTYPE_DEVICESTATUSCHANGE_SIZE =3D 0xffffffff +}; + +struct hsa_device_state_change { + /* F-NUMA node that contains the device */ + u32 node_id; + /* device type: GPU or CPU */ + enum hsa_device device; + /* event flags */ + enum hsa_eventtype_devicestatechange_flags flags; +}; + +struct hsa_access_attribute_failure { + /* Page not present or supervisor privilege */ + unsigned int not_present:1; + /* Write access to a read-only page */ + unsigned int readonly:1; + /* Execute access to a page marked NX */ + unsigned int no_execute:1; + /* Host access only */ + unsigned int gpu_access:1; + /* RAS ECC failure (notification of DRAM ECC - non-recoverable - + * error, if supported by HW) + */ + unsigned int ecc:1; + /* Can't determine the exact fault address */ + unsigned int imprecise:1; + /* Indicates RAS errors or other errors causing the access to GPU to fail + * 0 =3D no RAS error, + * 1 =3D ECC_SRAM, + * 2 =3D Link_SYNFLOOD (poison), + * 3 =3D GPU hang (not attributable to a specific cause), other values re= served + */ + unsigned int error_type:3; + /* must be 0 */ + unsigned int Reserved:23; +}; + +enum hsa_eventid_memory_flags { + /* access fault, recoverable after page adjustment */ + HSA_EVENTID_MEMORY_RECOVERABLE =3D 0, + /* memory access requires process context destruction, unrecoverable */ + HSA_EVENTID_MEMORY_FATAL_PROCESS =3D 1, + /* memory access requires all GPU VA context destruction, unrecoverable */ + HSA_EVENTID_MEMORY_FATAL_VM =3D 2, +}; + +struct hsa_memory_access_fault { + /* H-NUMA node that contains the device where the memory access occurred = */ + u32 node_id; + /* virtual address this occurred on */ + u64 virtual_address; + /* failure attribute */ + struct hsa_access_attribute_failure failure; + /* event flags */ + enum hsa_eventid_memory_flags flags; +}; + +struct hsa_event_data { + enum hsa_event_type event_type; + + union { + /* return data associated with HSA_EVENTTYPE_SIGNAL and other events */ + struct hsa_sync_var sync_var; + /* data associated with HSA_EVENTTYPE_NODE_CHANGE */ + struct hsa_node_change node_change_state; + /* data associated with HSA_EVENTTYPE_DEVICE_STATE_CHANGE */ + struct hsa_device_state_change device_state; + /* data associated with HSA_EVENTTYPE_MEMORY */ + struct hsa_memory_access_fault memory_access_fault; + }; + + // the following data entries are internal to the KFD & thunk itself. + + u64 hw_data1; // internal thunk store for Event data (OsEventHandle) + u64 hw_data2; // internal thunk store for Event data (HWAddress) + u32 hw_data3; // internal thunk store for Event data (HWData) +}; + +struct hsa_event { + u32 event_id; + struct hsa_event_data event_data; +}; + +#endif /* KFD_HSA_CODE_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_module.c b/drivers/gpu/drm/amd/= amdkfd/kfd_module.c index 33aa23450b3f..2a2405db5b9c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_module.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_module.c @@ -77,6 +77,7 @@ static int kfd_init(void) =20 static void kfd_exit(void) { + knod_exit(); kfd_cleanup_processes(); kfd_process_destroy_wq(); kfd_debugfs_fini(); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/am= dkfd/kfd_priv.h index acd0e41e744c..4f5d4b0bfa89 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -1069,6 +1069,7 @@ bool kfd_dev_is_large_bar(struct kfd_node *dev); struct kfd_process *create_process(const struct task_struct *thread, bool = primary); int kfd_process_create_wq(void); void kfd_process_destroy_wq(void); +void kfd_process_flush_wq(void); void kfd_cleanup_processes(void); struct kfd_process *kfd_create_process(struct task_struct *thread); int kfd_create_process_sysfs(struct kfd_process *process); @@ -1536,6 +1537,9 @@ void kfd_signal_hw_exception_event(u32 pasid); int kfd_set_event(struct kfd_process *p, uint32_t event_id); int kfd_reset_event(struct kfd_process *p, uint32_t event_id); int kfd_kmap_event_page(struct kfd_process *p, uint64_t event_page_offset); +int kfd_event_page_set(struct kfd_process *p, void *kernel_address, + uint64_t size, uint64_t user_handle); + =20 int kfd_event_create(struct file *devkfd, struct kfd_process *p, uint32_t event_type, bool auto_reset, uint32_t node_id, @@ -1641,3 +1645,33 @@ static inline void kfd_debugfs_remove_process(struct= kfd_process *p) {} #endif =20 #endif + +int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd, + uint64_t gpu_va, uint32_t size, + uint32_t flags, struct kgd_mem **mem, void **kptr); +void kfd_process_free_gpuvm(struct kgd_mem *mem, + struct kfd_process_device *pdd, void **kptr); +int kfd_create_queue(struct kfd_process *p, struct queue_properties *q_pro= perties, + u32 gpu_id, u32 *queue_id_out, u64 *doorbell_offset_out); +int kfd_ioctl_set_event(struct file *filp, struct kfd_process *p, + void *data); +int kfd_ioctl_reset_event(struct file *filp, struct kfd_process *p, + void *data); +int kfd_ioctl_wait_events(struct file *filp, struct kfd_process *p, void *= data); + +int kfd_ioctl_set_trap_handler(struct file *filep, + struct kfd_process *p, void *data); +int kfd_ioctl_set_scratch_backing_va(struct file *filep, + struct kfd_process *p, void *data); + + +int kfd_ioctl_acquire_vm(struct file *filep, struct kfd_process *p, void *= data); +int kfd_ioctl_map_memory_to_gpu(struct file *filep, struct kfd_process *p, + void *data); +int kfd_ioctl_alloc_memory_of_gpu(struct file *filep, struct kfd_process *= p, + void *data); +int kfd_ioctl_export_dmabuf(struct file *filep, + struct kfd_process *p, void *data); + +void __iomem *kfd_kernel_doorbell_mmap(struct kfd_node *dev, + struct kfd_process *process); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd= /amdkfd/kfd_process.c index ca71fa726e32..3c9268241800 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -715,8 +715,14 @@ void kfd_process_destroy_wq(void) } } =20 -static void kfd_process_free_gpuvm(struct kgd_mem *mem, - struct kfd_process_device *pdd, void **kptr) +void kfd_process_flush_wq(void) +{ + if (kfd_process_wq) + flush_workqueue(kfd_process_wq); +} + +void kfd_process_free_gpuvm(struct kgd_mem *mem, + struct kfd_process_device *pdd, void **kptr) { struct kfd_node *dev =3D pdd->dev; =20 @@ -736,9 +742,9 @@ static void kfd_process_free_gpuvm(struct kgd_mem *mem, * to avoid concurrency. Because of that exclusiveness, we do * not need to take p->mutex. */ -static int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd, - uint64_t gpu_va, uint32_t size, - uint32_t flags, struct kgd_mem **mem, void **kptr) +int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd, + uint64_t gpu_va, uint32_t size, + uint32_t flags, struct kgd_mem **mem, void **kptr) { struct kfd_node *kdev =3D pdd->dev; int err; @@ -761,10 +767,14 @@ static int kfd_process_alloc_gpuvm(struct kfd_process= _device *pdd, } =20 if (kptr) { - err =3D amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel( - (struct kgd_mem *)*mem, kptr, NULL); + if (flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM) + err =3D amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel( + (struct kgd_mem *)*mem, kptr, NULL); + else + err =3D amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel( + (struct kgd_mem *)*mem, kptr, NULL); if (err) { - pr_debug("Map GTT BO to kernel failed\n"); + pr_debug("Map BO to kernel failed\n"); goto sync_memory_failed; } } --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 803113AEF57 for ; Sun, 19 Jul 2026 18:01:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484080; cv=none; b=UHXKQrdN8qOTTghcoV+VTPuz4mnBbAf9s94PbpaW5JXlfrp86POCxkvi5Eg6VCxr1xglaU3yvE+HxPdMm40L/+i/OdhrOo7ZhWVhUeY2TfeQyJuJPeTbY3P9CSWCI3nj86DNrp/XMfFx62Skm2TExQFAo3zpvOEYlE77wTTg0gs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484080; c=relaxed/simple; bh=ah0JiDYJJ/aQNUbXioQROndNCndE69hAQp4JNbJsErY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mzesTvA2fPM+ok4RBvlIsOfijRbeVcur5nOMirIkvmLBjU5YauUjmQWAPIQyBES9OXBnJR72xAtGCegC8kGjmgOnFX+l2a+IJMACgAbUPzA2y0nQerKDFhFhaHtL9AzQpxha0OhcFHVziVGpWYYhaY+XY8NC4lLsIevvf9lxaCU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CWejZZBD; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CWejZZBD" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2ce87c7e3bbso104952515ad.1 for ; Sun, 19 Jul 2026 11:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484063; x=1785088863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=W1yw2ovWVlZBikeiXxsUocrBF67xVKX5IYM7DkLC69s=; b=CWejZZBDzWqfcg1Tjm8SC2G/zTbBZOVx/TUodYPba+ciiWG7qvPzdAo72hDvqsKJvq eJmY9s/d96Ul8vm/xKGPnqqpo/Ijp9lLSrdPyEqpymleAl7GykmoyCLyX+C5eC7MCMOK gHkO651WxkPw7iFteiKt+jmHJeCO6tARsorbF7EaV+9k1ojQhDXs/0nZG22sIxa1Ny2+ F9roAbXxWTFKC+yLLqmLL9Du6tVj+vzW/hPGWQEw1fvX1hrYbCeALsxHbBEXAHyCTK7L bTgNN3LHv5RqrcL2hZfD+q6cEzaMGMMDruPAtUDVJZuKrTnlURSoy+j6ot09pVEuqMUt UjkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484063; x=1785088863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=W1yw2ovWVlZBikeiXxsUocrBF67xVKX5IYM7DkLC69s=; b=VDBNO3Oq4EqJRSDLa8Oqj+tM2VWZfUl6epwbLsIUqewv5H1gLRWVYWDQI993wr2PsN Hu0gnNxdengFGx5o2I7L3lT19S3fZIfa8vyFHqozLA1xLjMwdFziL/Pl0eMI2D6cpm9J nTFyZQM82Wz/LE6PgOhLQ0sjg/O7phwOt5PaI/OTsLoSFNc4Vb9qcwIjTt+zT7BvjxCI 9esexOCrGYbgVb/5RKBg8Lpt6S4gEJO5yWgi+A/msCBbnJDMaxgO+vMpeY6R42n2UPdC FPRwqVyj7mh8gAHKLwUFKkTQgDT8fmOp0gOS4G+9uGnTlIxWY6qU7ZrGRudR9PYRHF94 8rvw== X-Forwarded-Encrypted: i=1; AHgh+RoHtPqKWO7jZtXzyuWwLX3VlHyN8fBqelS8RjcW/tc6suPHZnMmuRKaQoBg5AZpc65iIrCXOdGiR5EA13c=@vger.kernel.org X-Gm-Message-State: AOJu0Yw1eABsS4zOeiC3hAUkw5GA1/ajj1wSwOQtpuzQzPl/XbCahWq5 MVpkJ2rSleFfbamb9zcrnJ4gE97dSbUn+wwZTbZm0hlsU4tn5ZlBQXmw X-Gm-Gg: AfdE7cllF7zdvEZ9B78/ZtZF3wQJo6lw6u8QDdS0geY3OfWwxmDSLfbrX/1tpBcMxhR ICczDriFy5J4s/9TEZBTqA4F+ssPSzulBjVmjko4YHMIsCabOBHqgCd50S+JztBhJo/XRUJa5Vl pcIFiYOpJnPyjpg1wLy4QFOwgUp9QArJWOvqpI/SfA0zOocnZWp2jc/pR+qJekIEwv/+2w7kErw PWyDIFmmGS2OJd8GhsZ/9ow5VIXm5FeA+nM5CaA3wW/HGCtXtE9xK3KhyCyUBvotAJZjUsW7CvV 5CUsUUhTujtz2eS4S7c2pQkKkF0nmb3zB7sMgMwmudvn9zqEAHtlGEYfr1rCht0gAwtD+6jN/nN sBttDzJIhf5JHQUxpAa42hmeOt1sp6f+js14qV7w32+r/BOhmf/Rbp7eqvHXxIKYBlg== X-Received: by 2002:a17:902:e80e:b0:2ca:d666:df72 with SMTP id d9443c01a7336-2cf3489ae91mr117260965ad.21.1784484061981; Sun, 19 Jul 2026 11:01:01 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.00.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:01:00 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 07/13] drm/amdkfd: add knod provider core Date: Sun, 19 Jul 2026 17:58:51 +0000 Message-ID: <20260719175857.4071636-8-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the knod accelerator provider, built into amdgpu. It allocates and drives GPU AQL/SDMA queues, manages GPU memory for the RX data path, and registers a knod accelerator that binds to a NIC via the knod core. Feature workers (BPF, IPsec) plug in on top through the accel ops. Signed-off-by: Taehee Yoo (cherry picked from commit acaff825a0e87188906fd59f8a377c04d0b0ed2e) --- drivers/gpu/drm/amd/amdgpu/Makefile | 1 + drivers/gpu/drm/amd/amdkfd/Kconfig | 11 + drivers/gpu/drm/amd/amdkfd/Makefile | 4 + drivers/gpu/drm/amd/amdkfd/kfd_knod.c | 2202 ++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h | 270 +++ 5 files changed, 2488 insertions(+) create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_knod.c create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdg= pu/Makefile index ba80542ead9d..7a9826df9f37 100644 --- a/drivers/gpu/drm/amd/amdgpu/Makefile +++ b/drivers/gpu/drm/amd/amdgpu/Makefile @@ -42,6 +42,7 @@ ccflags-y :=3D -I$(FULL_AMD_PATH)/include/asic_reg \ -I$(FULL_AMD_DISPLAY_PATH)/dc \ -I$(FULL_AMD_DISPLAY_PATH)/amdgpu_dm \ -I$(FULL_AMD_PATH)/amdkfd \ + -I$(FULL_AMD_PATH)/amdkfd/knod \ -I$(FULL_AMD_PATH)/ras/ras_mgr =20 # Locally disable W=3D1 warnings enabled in drm subsystem Makefile diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkf= d/Kconfig index a5d7467c2f34..d93f1af749ff 100644 --- a/drivers/gpu/drm/amd/amdkfd/Kconfig +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig @@ -38,3 +38,14 @@ config HSA_AMD_P2P GPUs with large memory BARs that expose the entire VRAM in PCIe bus address space within the physical address limits of the GPUs. =20 +config HSA_AMD_KNOD + bool "KNOD GPU network offload core" + depends on HSA_AMD + default y + help + Core framework for KNOD, GPU-accelerated zero-copy network packet + processing on AMD GPUs. It is built into amdgpu and cannot be a + module because it relies on non-exported amdgpu/amdkfd internals. + + Say N to drop the KNOD core along with the BPF and IPsec offloads + layered on top of it. If unsure, say Y. diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdk= fd/Makefile index 85fc67d521e5..1834faa54863 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -71,3 +71,7 @@ ifneq ($(CONFIG_HSA_AMD_SVM),) AMDKFD_FILES +=3D $(AMDKFD_PATH)/kfd_svm.o \ $(AMDKFD_PATH)/kfd_migrate.o endif + +ifneq ($(CONFIG_HSA_AMD_KNOD),) +AMDKFD_FILES +=3D $(AMDKFD_PATH)/kfd_knod.o +endif diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_knod.c b/drivers/gpu/drm/amd/am= dkfd/kfd_knod.c new file mode 100644 index 000000000000..03fff054dcb9 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_knod.c @@ -0,0 +1,2202 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* + * Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software= "), + * to deal in the Software without restriction, including without limitati= on + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "kfd_priv.h" +#include "kfd_hsa.h" +#include "kfd_knod.h" +#include "kfd_topology.h" +#include "kfd_device_queue_manager.h" +#include "kfd_events.h" +#include +#include +#include +#include +#include +#include "../amdgpu/amdgpu_amdkfd.h" +#include "../amdgpu/amdgpu_gfx.h" +#include "../amdgpu/./navi10_sdma_pkt_open.h" +#include "amdgpu_dpm.h" +#include "kgd_pp_interface.h" +#include +#include +#include +#include +#include "knod_bpf.h" +#include +#include + +struct umh_data { + pid_t pid; +}; + +LIST_HEAD(ctx_list); + +/* + * AQL/SDMA queue pair count requested by the highest-demand accel + * consumer (currently knod_ipsec's parallel-dispatcher machinery). + * Each accel module sets this via knod_request_queue_cnt() during its + * module_init BEFORE the NOD attach happens, so knod_attach() creates + * a context with enough kaql[]/sdma[] pairs for the worst-case + * consumer. + * + * The value is a high-water mark across all accel types - whichever + * consumer asks for the most queue pairs wins; a consumer that needs + * fewer just leaves the extra pairs unused (a minor, harmless GPU + * resource waste). + * + * Default is 1 when nothing has called the setter - mirrors the + * historical single-queue behaviour. + */ +static int knod_requested_queue_cnt =3D 1; + +static struct knod_accel *accels[KNOD_MAX_AQL]; +static int nr_accels; +static struct knod_accel_ops accel_ops; + +/* + * Thin wrappers around the KFD core entry points. knod drives KFD from + * kernel context using kernel-internal types, so it calls these directly + * instead of going through the kfd_ioctl_* handlers (which marshal user + * data and stay static to KFD). + */ +static int knod_create_queue(struct kfd_process *p, + struct queue_properties *qp, u32 gpu_id, + u32 *queue_id, u64 *doorbell_offset) +{ + return kfd_create_queue(p, qp, gpu_id, queue_id, doorbell_offset); +} + +static int knod_destroy_queue(struct kfd_process *p, u32 queue_id) +{ + int ret; + + mutex_lock(&p->mutex); + ret =3D pqm_destroy_queue(&p->pqm, queue_id); + mutex_unlock(&p->mutex); + return ret; +} + +static int knod_create_event(struct kfd_process *p, u32 event_type, + bool auto_reset, u32 node_id, + struct knod_event *event) +{ + u32 event_trigger_data; + u64 event_page_offset =3D 0; + + return kfd_event_create(NULL, p, event_type, auto_reset, node_id, + &event->id, &event_trigger_data, + &event_page_offset, &event->slot); +} + +static int knod_destroy_event(struct kfd_process *p, u32 event_id) +{ + return kfd_event_destroy(p, event_id); +} + +struct knod_mem *__knod_alloc_mem(struct knod *knod, size_t size, + int flags) +{ + struct kfd_process_device *pdd =3D knod->process->pdds[0]; + struct kfd_node *kdev =3D pdd->dev; + struct knod_mem *mem; + int err; + + mem =3D kzalloc_obj(struct knod_mem, GFP_KERNEL); + if (!mem) + return ERR_PTR(-ENOMEM); + + size =3D ALIGN(size, PAGE_SIZE); + mem->flags =3D flags; + mem->size =3D size; + mem->gaddr =3D gen_pool_alloc(knod->pool, size); + if (!mem->gaddr) { + kfree(mem); + return ERR_PTR(-ENOMEM); + } + + err =3D amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(kdev->adev, mem->gaddr, + size, pdd->drm_priv, + &mem->mem, NULL, flags, + false); + if (err) { + knod_err(" failed to alloc mem\n"); + gen_pool_free(knod->pool, mem->gaddr, mem->size); + kfree(mem); + return ERR_PTR(-ENOMEM); + } + + err =3D amdgpu_amdkfd_gpuvm_sync_memory(kdev->adev, mem->mem, true); + if (err) { + pr_debug("Sync memory failed, wait interrupted by user signal\n"); + amdgpu_amdkfd_gpuvm_free_memory_of_gpu(kdev->adev, mem->mem, + pdd->drm_priv, NULL); + gen_pool_free(knod->pool, mem->gaddr, mem->size); + kfree(mem); + return ERR_PTR(-ENOMEM); + } + + list_add_tail(&mem->list, &knod->active_list); + + return mem; +} + +int __knod_export_dma_buf(struct knod *knod, struct knod_mem *mem) +{ + struct dma_buf *dmabuf; + int err; + + err =3D amdgpu_amdkfd_gpuvm_export_dmabuf(mem->mem, &dmabuf); + if (err) { + pr_debug("export dmabuf failed\n"); + return -ENOMEM; + } + + dma_buf_put(dmabuf); + + return 0; +} + +struct knod_mem *knod_alloc_mem(struct knod *knod, size_t size, int flags) +{ + struct knod_mem *mem; + int err; + + mem =3D kzalloc_obj(struct knod_mem, GFP_KERNEL); + if (!mem) + return ERR_PTR(-ENOMEM); + + size =3D ALIGN(size, PAGE_SIZE); + mem->flags =3D flags; + mem->size =3D size; + mem->gaddr =3D gen_pool_alloc(knod->pool, size); + if (!mem->gaddr) { + kfree(mem); + return ERR_PTR(-ENOMEM); + } + + err =3D kfd_process_alloc_gpuvm(knod->process->pdds[0], + mem->gaddr, + mem->size, + flags, + &mem->mem, + &mem->kaddr); + + if (err) { + knod_err(" err =3D %d\n", err); + gen_pool_free(knod->pool, mem->gaddr, mem->size); + kfree(mem); + return ERR_PTR(-ENOMEM); + } + + list_add_tail(&mem->list, &knod->active_list); + + return mem; +} +EXPORT_SYMBOL(knod_alloc_mem); + +int __knod_map_kaddr(struct knod *knod, struct knod_mem *mem) +{ + int err =3D 0; + + if (mem->flags & KFD_IOC_ALLOC_MEM_FLAGS_GTT) { + err =3D amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(mem->mem, + &mem->kaddr, + NULL); + if (err) { + pr_debug("Map GTT BO to kernel failed\n"); + err =3D -ENOMEM; + } + } else if (mem->flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM) { + err =3D amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel(mem->mem, + &mem->kaddr, + NULL); + if (err) { + pr_debug("Map VRAM BO to kernel failed\n"); + err =3D -ENOMEM; + } + } + + return err; +} + +int __knod_map_mem(struct knod *knod, struct knod_mem *mem) +{ + struct kfd_process_device *pdd =3D knod->process->pdds[0]; + struct kfd_node *kdev =3D pdd->dev; + int err; + + err =3D amdgpu_amdkfd_gpuvm_map_memory_to_gpu(kdev->adev, mem->mem, + pdd->drm_priv); + if (err) { + knod_err(" failed to map gpu\n"); + return 1; + } + + err =3D amdgpu_amdkfd_gpuvm_sync_memory(kdev->adev, mem->mem, true); + if (err) { + pr_debug("Sync memory failed, wait interrupted by user signal\n"); + return 1; + } + + return 0; +} +EXPORT_SYMBOL(__knod_map_mem); + +void knod_free_mem(struct knod *knod, struct knod_mem *mem) +{ + if (mem) { + list_del_init(&mem->list); + kfd_process_free_gpuvm(mem->mem, knod->process->pdds[0], + &mem->kaddr); + gen_pool_free(knod->pool, mem->gaddr, mem->size); + kfree(mem); + } +} +EXPORT_SYMBOL(knod_free_mem); + +void knod_sdma_copy(struct knod *knod, u64 dst_offset, + u64 src_offset, int idx, int size) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u32 ring_mask =3D (sdma->sdma->size / 4) - 1; + u64 *wptr =3D (u64 *)sdma->queue->kaddr + 1; + u32 *ptr =3D sdma->sdma->kaddr; + + ptr[sdma->idx++ & ring_mask] =3D SDMA_PKT_HEADER_OP(SDMA_OP_COPY) | + SDMA_PKT_HEADER_SUB_OP(SDMA_SUBOP_COPY_LINEAR) | + SDMA_PKT_COPY_LINEAR_HEADER_TMZ((0)); + ptr[sdma->idx++ & ring_mask] =3D size - 1; + ptr[sdma->idx++ & ring_mask] =3D 0; /* src/dst endian swap */ + ptr[sdma->idx++ & ring_mask] =3D lower_32_bits(src_offset); + ptr[sdma->idx++ & ring_mask] =3D upper_32_bits(src_offset); + ptr[sdma->idx++ & ring_mask] =3D lower_32_bits(dst_offset); + ptr[sdma->idx++ & ring_mask] =3D upper_32_bits(dst_offset); + + *wptr +=3D 7 * 4; +} + +void knod_sdma_fence(struct knod *knod, u64 fence_addr, u32 fence_val, + int idx) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u32 ring_mask =3D (sdma->sdma->size / 4) - 1; + u64 *wptr =3D (u64 *)sdma->queue->kaddr + 1; + u32 *ptr =3D sdma->sdma->kaddr; + u32 hdr =3D SDMA_PKT_HEADER_OP(SDMA_OP_FENCE); + + if (knod->isa_version >=3D 10) + hdr |=3D SDMA_PKT_FENCE_HEADER_MTYPE(3); + + ptr[sdma->idx++ & ring_mask] =3D hdr; + ptr[sdma->idx++ & ring_mask] =3D lower_32_bits(fence_addr); + ptr[sdma->idx++ & ring_mask] =3D upper_32_bits(fence_addr); + ptr[sdma->idx++ & ring_mask] =3D fence_val; + + *wptr +=3D 4 * 4; +} + +void knod_sdma_trap(struct knod *knod, int idx) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u32 ring_mask =3D (sdma->sdma->size / 4) - 1; + u64 *wptr =3D (u64 *)sdma->queue->kaddr + 1; + u32 *ptr =3D sdma->sdma->kaddr; + u32 ctx =3D knod->sdma_event[idx].id & 0x0fffffff; + u64 slot_addr =3D knod->mailbox->gaddr + + knod->sdma_event[idx].slot * 8; + u32 hdr; + + /* Write 0 to the signal page slot so lookup_signaled_event + * sees it as non-UNSIGNALED (0xFFFFFFFFFFFFFFFF). + */ + hdr =3D SDMA_PKT_HEADER_OP(SDMA_OP_FENCE); + if (knod->isa_version >=3D 10) + hdr |=3D SDMA_PKT_FENCE_HEADER_MTYPE(3); + + ptr[sdma->idx++ & ring_mask] =3D hdr; + ptr[sdma->idx++ & ring_mask] =3D lower_32_bits(slot_addr); + ptr[sdma->idx++ & ring_mask] =3D upper_32_bits(slot_addr); + ptr[sdma->idx++ & ring_mask] =3D 0; + + ptr[sdma->idx++ & ring_mask] =3D SDMA_PKT_HEADER_OP(SDMA_OP_TRAP); + ptr[sdma->idx++ & ring_mask] =3D ctx; + + *wptr +=3D 6 * 4; +} + +void knod_sdma_doorbell(struct knod *knod, int idx) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u64 *wptr =3D (u64 *)sdma->queue->kaddr + 1; + + /* ensure the SDMA ring writes are visible before ringing doorbell */ + wmb(); + writeq(*wptr, sdma->doorbell); +} + +u32 knod_sdma_submit(struct knod *knod, int idx, + const struct knod_sdma_copy_desc *copies, int n) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u32 capacity =3D sdma->sdma->size / 4; + u32 completed, inflight; + int i; + + /* + * knod_sdma_copy has no overflow guard (sdma->idx is the dword write + * cursor that just wraps), so drop the whole batch once the ring is + * near full. @completed is the last fenced cursor (knod_sdma_kick + * writes sdma->idx into the signal), so @inflight is the unprocessed + * span; reserve room for these @n copies (7 dwords each) plus a fence. + * Signed compare so a stale signal ahead of the cursor (e.g. at + * startup) reads as "negative" inflight rather than a false full. + */ + completed =3D (u32)READ_ONCE(((struct amd_signal *) + sdma->queue_signal->kaddr)->value); + inflight =3D (u32)sdma->idx - completed; + if ((s32)(inflight + n * 7) >=3D (s32)(capacity - 64)) + return 0; + + for (i =3D 0; i < n; i++) + knod_sdma_copy(knod, copies[i].dst, copies[i].src, idx, + copies[i].len); + + return (u32)sdma->idx; +} +EXPORT_SYMBOL(knod_sdma_submit); + +void knod_sdma_kick(struct knod *knod, int idx) +{ + struct knod_sdma *sdma =3D &knod->sdma[idx]; + u64 fence_addr; + + fence_addr =3D sdma->queue_signal->gaddr + + offsetof(struct amd_signal, value); + knod_sdma_fence(knod, fence_addr, (u32)sdma->idx, idx); + knod_sdma_doorbell(knod, idx); +} +EXPORT_SYMBOL(knod_sdma_kick); + +int knod_gart_map(struct amdgpu_device *adev, u64 npages, + dma_addr_t *addr, u64 *gart_addr, u64 flags) +{ + struct amdgpu_ring *ring =3D + to_amdgpu_ring(adev->mman.buffer_funcs_scheds[0]); + struct amdgpu_job *job; + unsigned int num_dw, num_bytes; + struct dma_fence *fence; + u64 src_addr, dst_addr; + u64 pte_flags; + void *cpu_addr; + int r; + + /* use gart window 0 */ + *gart_addr =3D adev->gmc.gart_start; + + num_dw =3D ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8); + num_bytes =3D npages * 8; + + r =3D amdgpu_job_alloc_with_ib(adev, &adev->mman.default_entity.base, + AMDGPU_FENCE_OWNER_UNDEFINED, + num_dw * 4 + num_bytes, + AMDGPU_IB_POOL_DELAYED, + &job, + AMDGPU_KERNEL_JOB_ID_TTM_MAP_BUFFER); + if (r) + return r; + + src_addr =3D num_dw * 4; + src_addr +=3D job->ibs[0].gpu_addr; + + dst_addr =3D amdgpu_bo_gpu_offset(adev->gart.bo); + amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr, + dst_addr, num_bytes, 0); + + amdgpu_ring_pad_ib(ring, &job->ibs[0]); + WARN_ON(job->ibs[0].length_dw > num_dw); + + pte_flags =3D AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE; + pte_flags |=3D AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; + if (!(flags & KFD_IOCTL_SVM_FLAG_GPU_RO)) + pte_flags |=3D AMDGPU_PTE_WRITEABLE; + pte_flags |=3D adev->gart.gart_pte_flags; + + cpu_addr =3D &job->ibs[0].ptr[num_dw]; + + amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr); + fence =3D amdgpu_job_submit(job); + dma_fence_put(fence); + + return r; +} + +static void knod_init_aql_queue(struct knod *knod, int qid, int idx) +{ + struct amd_queue *amd_queue =3D + (struct amd_queue *)knod->kaql[idx].queue->kaddr; + union knod_aql_rsrc1 rsrc1; + u32 scratch_gaddr; + int i; + + for (i =3D 0; i < knod->nr_aql_ring; i++) + knod_setup_invalidate(knod, i, idx); + + amd_queue->hsa_queue.type =3D HSA_QUEUE_TYPE_MULTI; + amd_queue->hsa_queue.features =3D HSA_QUEUE_FEATURE_KERNEL_DISPATCH; + amd_queue->hsa_queue.base_address =3D (void *)knod->kaql[idx].aql->gaddr; + amd_queue->hsa_queue.doorbell_signal =3D + knod->kaql[idx].queue_signal->gaddr; + amd_queue->hsa_queue.size =3D knod->nr_aql_ring; + amd_queue->hsa_queue.reserved1 =3D 0; + amd_queue->hsa_queue.id =3D qid; + + amd_queue->write_dispatch_id =3D 0; /* id is index */ + amd_queue->group_segment_aperture_base_hi =3D 0; + amd_queue->private_segment_aperture_base_hi =3D 0; + amd_queue->max_cu_id =3D -1; + amd_queue->max_wave_id =3D -1; + amd_queue->max_legacy_doorbell_dispatch_id_plus_1 =3D 0; + amd_queue->legacy_doorbell_lock =3D 0; + amd_queue->read_dispatch_id =3D 0; /* id is index */ + amd_queue->read_dispatch_id_field_base_byte_offset =3D 0x80; + /* scratch resource descriptor - use allocated scratch BO address */ + scratch_gaddr =3D (u32)(knod->kaql[idx].scratch->gaddr & 0xffffffff); + amd_queue->scratch_resource_descriptor[0] =3D scratch_gaddr; + scratch_gaddr =3D (u32)((knod->kaql[idx].scratch->gaddr >> 32) & 0xffff); + rsrc1.base_address_hi =3D scratch_gaddr; + rsrc1.stride =3D 0; + rsrc1.cache_swizzle =3D 0; + rsrc1.swizzle_enable =3D 1; + memcpy(&amd_queue->scratch_resource_descriptor[1], &rsrc1, sizeof(u32)); + amd_queue->scratch_resource_descriptor[2] =3D + knod->kaql[idx].scratch->size; + amd_queue->scratch_resource_descriptor[3] =3D 0x00ffffff; + amd_queue->scratch_backing_memory_location =3D + knod->kaql[idx].scratch->gaddr; + amd_queue->scratch_backing_memory_byte_size =3D + knod->kaql[idx].scratch->size; + amd_queue->scratch_wave64_lane_byte_size =3D 64; + + amd_queue->queue_properties.is_ptr64 =3D 1; + amd_queue->queue_properties.enable_trap_handler_debug_sgprs =3D 0; +} + +static void knod_init_sdma_queue(struct knod *knod, int qid, int idx) +{ +} + +static void knod_init_queue(struct knod *knod, int qid, int idx, int type) +{ + if (type =3D=3D KFD_IOC_QUEUE_TYPE_COMPUTE_AQL) + knod_init_aql_queue(knod, qid, idx); + else if (type =3D=3D KFD_IOC_QUEUE_TYPE_SDMA) + knod_init_sdma_queue(knod, qid, idx); +} + +static void stop_umh(pid_t umh_pid) +{ + struct pid *p =3D find_get_pid(umh_pid); + + if (!p) + return; + + kill_pid(p, SIGKILL, 1); + put_pid(p); +} + +static int umh_init(struct subprocess_info *info, struct cred *new) +{ + struct umh_data *d =3D info->data; + + d->pid =3D current->pid; + + return 0; +} + +static int launch_and_get_pid(void) +{ + static const char * const argv[] =3D { "/bin/sleep", "2147483647", NULL }; + static const char * const envp[] =3D { "HOME=3D/", "PATH=3D/sbin:/bin", N= ULL }; + struct umh_data data =3D { .pid =3D -1 }; + struct subprocess_info *info; + int ret; + + info =3D call_usermodehelper_setup(argv[0], (char **)argv, + (char **)envp, GFP_KERNEL, + umh_init, NULL, &data); + if (!info) { + pr_err("UMH setup failed\n"); + return 0; + } + + ret =3D call_usermodehelper_exec(info, UMH_WAIT_EXEC); + if (ret < 0) { + pr_err("UMH exec failed: %d\n", ret); + return 0; + } + + return data.pid; +} + +static int knod_set_isa(struct knod *knod) +{ + enum amd_asic_type asic_type; + + asic_type =3D knod->process->pdds[0]->dev->adev->asic_type; + if (knod->isa_version !=3D 9 && knod->isa_version !=3D 10) + knod->isa_version =3D 0; + if (knod->isa_version =3D=3D 0) { + if (asic_type <=3D CHIP_VEGAM) { + pr_err("Not supported chip"); + return -EOPNOTSUPP; + } else if (asic_type =3D=3D CHIP_VEGA10 || + asic_type =3D=3D CHIP_VEGA12 || + asic_type =3D=3D CHIP_VEGA20 || + asic_type =3D=3D CHIP_RAVEN || + asic_type =3D=3D CHIP_RENOIR) { + pr_debug("GCN5 is detected"); + knod->isa_version =3D 9; + } else if (asic_type =3D=3D CHIP_NAVI10 || + asic_type =3D=3D CHIP_NAVI12 || + asic_type =3D=3D CHIP_NAVI14 || + asic_type =3D=3D CHIP_CYAN_SKILLFISH) { + pr_err("RDNA1 is not supported yet"); + return -EOPNOTSUPP; + } else if (asic_type =3D=3D CHIP_SIENNA_CICHLID || + asic_type =3D=3D CHIP_NAVY_FLOUNDER || + asic_type =3D=3D CHIP_DIMGREY_CAVEFISH || + asic_type =3D=3D CHIP_BEIGE_GOBY || + asic_type =3D=3D CHIP_YELLOW_CARP) { + pr_debug("RDNA2 is detected"); + knod->isa_version =3D 10; + } else if (asic_type =3D=3D CHIP_ARCTURUS || + asic_type =3D=3D CHIP_ALDEBARAN) { + pr_err("CDNA is not supported yet"); + return -EOPNOTSUPP; + } else if (asic_type =3D=3D CHIP_IP_DISCOVERY) { + pr_err("Can't detect chip version, firmware update may be needed"); + return -EOPNOTSUPP; + } else { + pr_err("Not supported chip"); + return -EOPNOTSUPP; + } + } + + knod->igpu =3D false; + if (asic_type =3D=3D CHIP_RENOIR || asic_type =3D=3D CHIP_RAVEN || + asic_type =3D=3D CHIP_CYAN_SKILLFISH || asic_type =3D=3D CHIP_BEIGE_G= OBY || + asic_type =3D=3D CHIP_YELLOW_CARP) { + pr_debug("iGPU is detected"); + knod->igpu =3D true; + } + + return 0; +} + +static void knod_destroy_one_queue(struct knod *knod, int idx) +{ + /* Destroy queue and event BEFORE freeing BOs. + * Queue holds references to BO VAs; freeing BOs first causes + * NULL deref in kfd_queue_unref_bo_vas when UMH exits. + * Use the created flag (not queue_id value) since KFD IDR can + * assign queue_id=3D0 to the first queue. + */ + if (knod->aql_queue_created[idx]) { + int ret; + + ret =3D knod_destroy_queue(knod->process, + knod->aql_queue_id[idx]); + if (ret) + pr_err("knod: destroy_queue failed: %d (queue_id=3D%u)\n", + ret, knod->aql_queue_id[idx]); + knod->aql_queue_created[idx] =3D false; + knod->aql_queue_id[idx] =3D 0; + } + if (knod->aql_event[idx].id) { + knod_destroy_event(knod->process, knod->aql_event[idx].id); + knod->aql_event[idx].id =3D 0; + } + + knod_free_mem(knod, knod->kaql[idx].aql); + knod_free_mem(knod, knod->kaql[idx].queue); + knod_free_mem(knod, knod->kaql[idx].eop); + knod_free_mem(knod, knod->kaql[idx].ctx); + knod_free_mem(knod, knod->kaql[idx].queue_signal); + knod_free_mem(knod, knod->kaql[idx].amd_queue); + knod_free_mem(knod, knod->kaql[idx].scratch); +} + +static int knod_alloc_one_queue(struct knod *knod, int idx, + struct kfd_topology_device *topo_dev, + struct kfd_process_device *pdd, void *ptr) +{ + int buf_flags =3D KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_VRAM; + int flags =3D KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_GTT; + struct amd_signal *queue_signal; + struct amd_queue *amd_queue; + struct queue_properties qp; + u32 total_cwsr_size; + int err; + + knod->kaql[idx].aql =3D knod_alloc_mem(knod, + (NR_AQL_RING * + sizeof(struct hsa_kernel_dispatch_packet) * 2), + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED | + KFD_IOC_ALLOC_MEM_FLAGS_AQL_QUEUE_MEM | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE); + if (IS_ERR(knod->kaql[idx].aql)) { + err =3D PTR_ERR(knod->kaql[idx].aql); + knod->kaql[idx].aql =3D NULL; + goto err_mem; + } + + knod->kaql[idx].queue =3D knod_alloc_mem(knod, PAGE_SIZE, flags); + if (IS_ERR(knod->kaql[idx].queue)) { + err =3D PTR_ERR(knod->kaql[idx].queue); + knod->kaql[idx].queue =3D NULL; + goto err_mem; + } + + knod->kaql[idx].eop =3D knod_alloc_mem(knod, PAGE_SIZE, flags); + if (IS_ERR(knod->kaql[idx].eop)) { + err =3D PTR_ERR(knod->kaql[idx].eop); + knod->kaql[idx].eop =3D NULL; + goto err_mem; + } + + total_cwsr_size =3D (topo_dev->node_props.cwsr_size + + topo_dev->node_props.debug_memory_size) + * NUM_XCC(pdd->dev->xcc_mask); + total_cwsr_size =3D ALIGN(total_cwsr_size, PAGE_SIZE); + + knod->kaql[idx].ctx =3D knod_alloc_mem(knod, total_cwsr_size, buf_flags); + if (IS_ERR(knod->kaql[idx].ctx)) { + err =3D PTR_ERR(knod->kaql[idx].ctx); + knod->kaql[idx].ctx =3D NULL; + goto err_mem; + } + + knod->kaql[idx].queue_signal =3D knod_alloc_mem(knod, + PAGE_SIZE << 5, + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE); + if (IS_ERR(knod->kaql[idx].queue_signal)) { + err =3D PTR_ERR(knod->kaql[idx].queue_signal); + knod->kaql[idx].queue_signal =3D NULL; + goto err_mem; + } + + knod->kaql[idx].amd_queue =3D knod_alloc_mem(knod, + PAGE_SIZE << 5, + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(knod->kaql[idx].amd_queue)) { + err =3D PTR_ERR(knod->kaql[idx].amd_queue); + knod->kaql[idx].amd_queue =3D NULL; + goto err_mem; + } + + knod->kaql[idx].scratch =3D knod_alloc_mem(knod, + PAGE_SIZE << 5, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(knod->kaql[idx].scratch)) { + err =3D PTR_ERR(knod->kaql[idx].scratch); + knod->kaql[idx].scratch =3D NULL; + goto err_mem; + } + + knod->kaql[idx].idx =3D 0; + + /* + * KNOD dispatchers poll queue_signal->value directly. Do not attach a + * KFD signal event to every AQL completion; at high packet rates the + * unused event notifications overflow the KFD interrupt/event path and + * add latency without contributing to completion detection. + */ + knod->aql_event[idx].id =3D 0; + knod->aql_event[idx].slot =3D 0; + knod->signal_eid =3D 0; + + /* Zero ALL queue-related GPU memory BEFORE creating the HW queue. + * kfd_ioctl_create_queue loads HQD immediately - if the memory + * contains stale data from a previous session (write/read pointers, + * AQL dispatch packets, doorbell values), the GPU starts processing + * garbage packets and goes to 100%. + */ + memset(knod->kaql[idx].queue->kaddr, 0, knod->kaql[idx].queue->size); + /* AQL ring is allocated 2x with AQL_QUEUE_MEM flag; the second + * half is a GPU-side mirror for wrap-around. kaddr only covers + * the first half. + */ + memset(knod->kaql[idx].aql->kaddr, 0, knod->kaql[idx].aql->size / 2); + memset(knod->kaql[idx].eop->kaddr, 0, knod->kaql[idx].eop->size); + memset(knod->kaql[idx].amd_queue->kaddr, 0, + knod->kaql[idx].amd_queue->size); + knod_init_queue(knod, idx, idx, KFD_IOC_QUEUE_TYPE_COMPUTE_AQL); + + memset(&qp, 0, sizeof(qp)); + qp.type =3D KFD_QUEUE_TYPE_COMPUTE; + qp.format =3D KFD_QUEUE_FORMAT_AQL; + qp.queue_percent =3D 100; + qp.priority =3D 15; + qp.ctl_stack_size =3D topo_dev->node_props.ctl_stack_size; + qp.ctx_save_restore_area_size =3D topo_dev->node_props.cwsr_size; + qp.ctx_save_restore_area_address =3D (u64)knod->kaql[idx].ctx->gaddr; + qp.queue_address =3D (u64)knod->kaql[idx].aql->gaddr; + qp.queue_size =3D knod->kaql[idx].aql->size / 2; + qp.write_ptr =3D + (void __user *)((u64)knod->kaql[idx].queue->gaddr + 0x38); + qp.read_ptr =3D (void __user *)((u64)knod->kaql[idx].queue->gaddr + 0x80); + qp.eop_ring_buffer_address =3D (u64)knod->kaql[idx].eop->gaddr; + qp.eop_ring_buffer_size =3D knod->kaql[idx].eop->size; + + err =3D knod_create_event(knod->process, KFD_IOC_EVENT_SIGNAL, true, 1, + &knod->aql_event[idx]); + if (err) { + knod_err(" failed to create AQL event[%d] err=3D%d\n", idx, err); + goto err_mem; + } + + queue_signal =3D (struct amd_signal *)knod->kaql[idx].queue_signal->kaddr; + queue_signal->kind =3D AMD_SIGNAL_KIND_USER; + if (queue_signal->kind =3D=3D AMD_SIGNAL_KIND_DOORBELL) { + queue_signal->value =3D 0; + queue_signal->event_id =3D 0; + queue_signal->queue_ptr =3D (void *)knod->kaql[idx].queue->gaddr; + queue_signal->hardware_doorbell_ptr =3D ptr; + queue_signal->event_mailbox_ptr =3D 0; + } else if (queue_signal->kind =3D=3D AMD_SIGNAL_KIND_USER) { + queue_signal->event_id =3D knod->aql_event[idx].id; + queue_signal->queue_ptr =3D 0; + queue_signal->event_mailbox_ptr =3D knod->mailbox->gaddr + + knod->aql_event[idx].slot * 8; + /* value is in union with hardware_doorbell_ptr - set last */ + queue_signal->value =3D 0xffffffffffffffff; + } + knod_dbg(" pre-create-queue signal=3D%lld kaddr=3D%p gaddr=3D0x%llx\n", + READ_ONCE(queue_signal->value), queue_signal, + knod->kaql[idx].queue_signal->gaddr); + + err =3D knod_create_queue(knod->process, &qp, pdd->user_gpu_id, + &knod->aql_queue_id[idx], + &knod->aql_doorbell_offset[idx]); + if (err) { + knod_err(" failed to create queue, err =3D %d\n", err); + goto err_mem; + } + knod->aql_queue_created[idx] =3D true; + + knod_dbg(" post-create-queue signal=3D%lld queue_id=3D%d\n", + READ_ONCE(queue_signal->value), knod->aql_queue_id[idx]); + + /* Update queue id with the actual KFD-assigned value */ + amd_queue =3D (struct amd_queue *)knod->kaql[idx].queue->kaddr; + amd_queue->hsa_queue.id =3D knod->aql_queue_id[idx]; + return 0; + +err_mem: + if (knod->aql_event[idx].id) { + knod_destroy_event(knod->process, knod->aql_event[idx].id); + knod->aql_event[idx].id =3D 0; + } + knod_free_mem(knod, knod->kaql[idx].aql); + knod_free_mem(knod, knod->kaql[idx].queue); + knod_free_mem(knod, knod->kaql[idx].eop); + knod_free_mem(knod, knod->kaql[idx].ctx); + knod_free_mem(knod, knod->kaql[idx].queue_signal); + knod_free_mem(knod, knod->kaql[idx].amd_queue); + knod_free_mem(knod, knod->kaql[idx].scratch); + memset(&knod->kaql[idx], 0, sizeof(knod->kaql[idx])); + return err; +} + +/* + * Write a minimal no-op kernel into the kernel BO so that any dispatch + * before a real shader (BPF/IPsec/MACsec/WG) is loaded executes a + * harmless s_endpgm instead of faulting on uninitialised VRAM. + * + * Layout: + * [0..63] kernel_descriptor (kernel_code_entry_byte_offset =3D 256) + * [256..259] s_endpgm (0xBF810000) + * [260..1023] s_code_end padding (GFX10 SQC prefetch safety) + */ +#define KNOD_DEFAULT_KD_ENTRY_OFFSET 256 +#define KNOD_S_ENDPGM 0xBF810000u +#define KNOD_S_CODE_END 0xBF9F0000u + +static void knod_init_default_kernel(struct knod *knod) +{ + struct kernel_descriptor *kd =3D knod->kernels[0]->kaddr; + u32 *code =3D (u32 *)((u8 *)knod->kernels[0]->kaddr + + KNOD_DEFAULT_KD_ENTRY_OFFSET); + int i; + + memset(kd, 0, sizeof(*kd)); + kd->kernel_code_entry_byte_offset =3D KNOD_DEFAULT_KD_ENTRY_OFFSET; + kd->compute_pgm_rsrc1.granulated_workitem_vgpr_count =3D 0; + kd->compute_pgm_rsrc1.granulated_wavefront_sgpr_count =3D 0; + kd->compute_pgm_rsrc1.float_denorm_mode_32 =3D 3; + kd->compute_pgm_rsrc1.float_denorm_mode_16_64 =3D 3; + kd->compute_pgm_rsrc1.enable_dx10_clamp =3D 1; + kd->compute_pgm_rsrc1.enable_ieee_mode =3D 1; + if (knod->isa_version >=3D 10) + kd->compute_pgm_rsrc1.mem_ordered =3D 1; + kd->compute_pgm_rsrc2.enable_sgpr_workgroup_id_x =3D 1; + + code[0] =3D KNOD_S_ENDPGM; + for (i =3D 1; i < (1024 - KNOD_DEFAULT_KD_ENTRY_OFFSET) / 4; i++) + code[i] =3D KNOD_S_CODE_END; +} + +#define KNOD_DEFAULT_BATCH 64 + +static struct knod_accel_xdp_ops *registered_xdp_ops; + +/* + * feature=3Dnone has no per-feature ops: the default worker stamps XDP_PA= SS and + * the NIC act handler delivers via knod_d2h_copy / knod_d2h_drain. + */ +static struct knod_accel_xdp_ops default_xdp_ops =3D { +}; + +static int knod_default_worker(void *arg) +{ + struct knod *knod =3D arg; + struct spsc_bd *bds[KNOD_DEFAULT_BATCH]; + struct knod_dev *knodev; + unsigned int cnt; + int qi, i; + + while (!kthread_should_stop()) { + knodev =3D READ_ONCE(knod->accel->knodev); + if (!knodev || !knodev->started) { + usleep_range(1000, 2000); + continue; + } + + for (qi =3D 0; qi < knod->channels; qi++) { + struct knod_work_priv *wpriv =3D &knodev->wpriv[qi]; + + if (!wpriv->napi) + continue; + + if (spsc_peek(&wpriv->spsc_bds, (void **)bds, + KNOD_DEFAULT_BATCH, &cnt)) + continue; + + /* + * feature=3Dnone has no program, so every packet passes. + * Stamp the verdict before advancing the consumer + * cursor: spsc_acquire() publishes the window with a + * release barrier the act handler pairs with, so the + * verdict has to be written first or the act handler + * races a POISON read. The NIC act handler does the + * device->host delivery via knod_d2h_copy. + */ + for (i =3D 0; i < cnt; i++) + WRITE_ONCE(bds[i]->act, XDP_PASS); + + spsc_acquire(&wpriv->spsc_bds, NULL, cnt, NULL); + knod_napi_kick(wpriv); + } + + usleep_range(100, 200); + } + return 0; +} + +static int knod_start_default_worker(struct knod *knod) +{ + struct task_struct *p; + + knod->worker_fn =3D knod_default_worker; + knod->flush_fn =3D NULL; + knod->worker_ctx =3D knod; + p =3D kthread_run(knod_default_worker, knod, "knod_dflt_%d", + knod->accel->id); + if (IS_ERR(p)) + return PTR_ERR(p); + + get_task_struct(p); + knod->worker =3D p; + return 0; +} + +static void knod_stop_worker(struct knod *knod) +{ + if (!knod->worker) + return; + + if (knod->flush_fn) + knod->flush_fn(knod->worker_ctx); + + kthread_stop(knod->worker); + put_task_struct(knod->worker); + knod->worker =3D NULL; + knod->worker_fn =3D NULL; + knod->flush_fn =3D NULL; + knod->worker_ctx =3D NULL; +} + +int knod_register_worker(struct knod *knod, knod_worker_fn_t fn, + knod_flush_fn_t flush, void *ctx) +{ + struct task_struct *p; + + knod_stop_worker(knod); + + knod->worker_fn =3D fn; + knod->flush_fn =3D flush; + knod->worker_ctx =3D ctx; + p =3D kthread_run(fn, ctx, "knod_%d", knod->accel->id); + if (IS_ERR(p)) { + knod->worker_fn =3D NULL; + knod->flush_fn =3D NULL; + knod->worker_ctx =3D NULL; + return PTR_ERR(p); + } + + get_task_struct(p); + knod->worker =3D p; + return 0; +} + +void knod_unregister_worker(struct knod *knod) +{ + knod_stop_worker(knod); + knod_start_default_worker(knod); +} + +int knod_wait_on_events(struct kfd_process *p, u32 num_events, + void __user *data, bool all, u32 *user_timeout_ms, + u32 *wait_result) +{ + return kfd_wait_on_events_kernel(p, num_events, data, all, + user_timeout_ms, wait_result); +} +EXPORT_SYMBOL(knod_wait_on_events); + +static int knod_alloc_ctx_init(struct knod *knod, int id, void **doorbell, + struct kfd_topology_device **out_topo_dev, + struct kfd_process_device **out_pdd) +{ + struct kfd_topology_device *topo_dev; + struct kfd_process_device *pdd; + struct file *drm_file; + size_t mem_size; + char path[64]; + int err; + + sprintf(path, "/dev/dri/renderD%d", id); + + drm_file =3D filp_open(path, O_RDWR, 0); + if (IS_ERR(drm_file)) + return PTR_ERR(drm_file); + + knod->drm_file =3D drm_file; + + knod->process =3D kfd_create_process(knod->umh_task); + if (IS_ERR(knod->process)) { + err =3D PTR_ERR(knod->process); + goto err_filp_close; + } + kref_get(&knod->process->ref); + + err =3D knod_set_isa(knod); + if (err < 0) + goto err_unref_process; + + pdd =3D knod->process->pdds[0]; + + topo_dev =3D kfd_topology_device_by_id(pdd->dev->id); + if (!topo_dev) { + pr_err("knod: can't find topo_dev for dev id %u\n", + pdd->dev->id); + err =3D -ENODEV; + goto err_unref_process; + } + + /* Acquire VM directly without installing an fd into any process's + * fdtable. get_file() provides the ref that pdd->drm_file will own. + * kfd_process_destroy_pdds will fput it during process cleanup. + */ + get_file(drm_file); + mutex_lock(&knod->process->mutex); + err =3D kfd_process_device_init_vm(pdd, drm_file); + mutex_unlock(&knod->process->mutex); + if (err) { + fput(drm_file); + goto err_unref_process; + } + + *doorbell =3D kfd_kernel_doorbell_mmap(pdd->dev, knod->process); + if (!*doorbell) { + err =3D -ENOMEM; + goto err_unref_process; + } + + knod->pool =3D gen_pool_create(PAGE_SHIFT, + dev_to_node(pdd->dev->adev->dev)); + if (!knod->pool) { + err =3D -ENOMEM; + goto err_release_doorbell; + } + + knod->dev =3D pdd->dev; + knod->reserved_addr =3D pdd->gpuvm_base << 2; + knod->limit_addr =3D pdd->gpuvm_limit; + mem_size =3D knod->limit_addr - knod->reserved_addr; + + err =3D gen_pool_add(knod->pool, knod->reserved_addr, mem_size, + dev_to_node(pdd->dev->adev->dev)); + if (err) + goto err_gen_pool_destroy; + + knod->kernels[0] =3D knod_alloc_mem(knod, PAGE_SIZE << 10, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE); + if (IS_ERR(knod->kernels[0])) { + err =3D PTR_ERR(knod->kernels[0]); + knod->kernels[0] =3D NULL; + goto err_gen_pool_destroy; + } + knod_init_default_kernel(knod); + + /* + * Second dispatch slot for the BPF ping-pong swap, allocated right + * after knod->kernels[0] so both kernel BOs sit in the same low VA + * region. + * Allocating it later (at feature activate, past the RX buffers) put it + * at a high VA, and switching the dispatch kernel_object to that BO + * mid-stream wedged the compute queue. + */ + knod->kernels[1] =3D knod_alloc_mem(knod, PAGE_SIZE << 10, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE); + if (IS_ERR(knod->kernels[1])) { + err =3D PTR_ERR(knod->kernels[1]); + knod->kernels[1] =3D NULL; + goto err_free_kernel; + } + + knod->mailbox =3D knod_alloc_mem(knod, PAGE_SIZE << 5, + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_VRAM); + if (IS_ERR(knod->mailbox)) { + err =3D PTR_ERR(knod->mailbox); + knod->mailbox =3D NULL; + goto err_free_kernel; + } + + *out_topo_dev =3D topo_dev; + *out_pdd =3D pdd; + return 0; + +err_free_kernel: + if (knod->kernels[1]) + knod_free_mem(knod, knod->kernels[1]); + knod->kernels[1] =3D NULL; + knod_free_mem(knod, knod->kernels[0]); + knod->kernels[0] =3D NULL; +err_gen_pool_destroy: + gen_pool_destroy(knod->pool); + knod->pool =3D NULL; +err_release_doorbell: + iounmap(*doorbell); + *doorbell =3D NULL; +err_unref_process: + kfd_unref_process(knod->process); +err_filp_close: + fput(knod->drm_file); + return err; +} + +struct knod *knod_alloc_ctx(struct knod_dev *knodev, int queue_cnt, int id, + int channels) +{ + int buf_flags =3D KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_VRAM; + struct kfd_topology_device *topo_dev; + struct kfd_process_device *pdd; + struct process_queue_node *pqn; + struct knod_work_priv *wpriv; + struct task_struct *task; + unsigned int sdma_cnt; + void *ptr =3D NULL; + struct knod *knod; + struct knod_mem *buf; + struct pid *spid; + struct queue *q; + int err, size; + pid_t pid; + int idx; + + pid =3D launch_and_get_pid(); + if (!pid) { + knod_err(" failed to create UMH\n"); + return ERR_PTR(-EINVAL); + } + + spid =3D find_get_pid(pid); + if (!spid) { + knod_err(" failed to find pid\n"); + return ERR_PTR(-ESRCH); + } + + task =3D get_pid_task(spid, PIDTYPE_PID); + put_pid(spid); + if (!task) { + knod_err(" no task found\n"); + return ERR_PTR(-ESRCH); + } + + knod =3D kzalloc(sizeof(struct knod), GFP_KERNEL); + if (!knod) { + put_task_struct(task); + return ERR_PTR(-ENOMEM); + } + + knod->queue_cnt =3D queue_cnt; + knod->sdma_cnt =3D queue_cnt; + knod->channels =3D channels; + knod->umh_pid =3D pid; + knod->umh_task =3D task; + knod->nr_aql_ring =3D NR_AQL_RING; + INIT_LIST_HEAD(&knod->list); + INIT_LIST_HEAD(&knod->active_list); + + err =3D knod_alloc_ctx_init(knod, id, &ptr, &topo_dev, &pdd); + if (err) + goto err_free_knod; + + sdma_cnt =3D topo_dev->node_props.num_sdma_engines * + topo_dev->node_props.num_sdma_queues_per_engine; + if (!sdma_cnt) + sdma_cnt =3D queue_cnt; + knod->sdma_cnt =3D min_t(int, queue_cnt, sdma_cnt); + if (!knod->sdma_cnt) + knod->sdma_cnt =3D 1; + pr_info("knod: AQL queues=3D%d SDMA queues=3D%d channels=3D%d\n", + queue_cnt, knod->sdma_cnt, channels); + + knod->doorbell_base =3D ptr; + + knod->buf =3D kmalloc_array(channels, sizeof(struct knod_mem *), + GFP_KERNEL | __GFP_ZERO); + if (!knod->buf) { + err =3D -ENOMEM; + goto err_free_mailbox; + } + + if (knod->igpu) + size =3D PAGE_SIZE << MAX_PAGE_ORDER; + else + size =3D PAGE_SIZE << 14; + + for (idx =3D 0; idx < channels; idx++) { + wpriv =3D &knodev->wpriv[idx]; + buf =3D __knod_alloc_mem(knod, size, buf_flags); + if (IS_ERR(buf)) { + err =3D PTR_ERR(buf); + goto err_free_bufs; + } + if (__knod_export_dma_buf(knod, buf)) { + knod_free_mem(knod, buf); + err =3D -ENOMEM; + goto err_free_bufs; + } + if (__knod_map_kaddr(knod, buf)) { + knod_free_mem(knod, buf); + err =3D -ENOMEM; + goto err_free_bufs; + } + if (__knod_map_mem(knod, buf)) { + knod_free_mem(knod, buf); + err =3D -ENOMEM; + goto err_free_bufs; + } + wpriv->dmabuf =3D buf->mem->dmabuf; + wpriv->index =3D idx; + knod->buf[idx] =3D buf; + } + + /* + * GPU->host delivery buffer + per-queue page_pools are owned by the + * NOD framework (knod_pass_attach), allocated via accel_ops->alloc_mem. + */ + + memset(knod->mailbox->kaddr, 0, knod->mailbox->size); + + err =3D kfd_event_page_set(knod->process, + knod->mailbox->kaddr, + KFD_SIGNAL_EVENT_LIMIT * 8, + knod->mailbox->gaddr); + if (err < 0) + goto err_free_bufs; + + for (idx =3D 0; idx < queue_cnt; idx++) { + err =3D knod_alloc_one_queue(knod, idx, topo_dev, pdd, ptr); + if (err) + goto err_free_queues; + } + + /* Allocate only the SDMA queues the device can actually provide. BPF + * XDP_TX does not use SDMA for the verdict path, while PASS/d2h maps RX + * queues onto the available SDMA queues modulo sdma_cnt. + */ + for (idx =3D 0; idx < knod->sdma_cnt; idx++) { + struct amd_signal *sdma_signal; + struct queue_properties qp; + long *sdma_lptr; + int sdma_flags =3D KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_GTT; + + knod->sdma[idx].sdma =3D knod_alloc_mem(knod, PAGE_SIZE << 4, + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE); + if (IS_ERR(knod->sdma[idx].sdma)) { + err =3D PTR_ERR(knod->sdma[idx].sdma); + knod->sdma[idx].sdma =3D NULL; + goto err_free_sdma; + } + knod->sdma[idx].queue =3D knod_alloc_mem(knod, PAGE_SIZE, + sdma_flags); + if (IS_ERR(knod->sdma[idx].queue)) { + err =3D PTR_ERR(knod->sdma[idx].queue); + knod->sdma[idx].queue =3D NULL; + goto err_free_sdma; + } + knod->sdma[idx].queue_signal =3D knod_alloc_mem(knod, + PAGE_SIZE << 5, + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE); + if (IS_ERR(knod->sdma[idx].queue_signal)) { + err =3D PTR_ERR(knod->sdma[idx].queue_signal); + knod->sdma[idx].queue_signal =3D NULL; + goto err_free_sdma; + } + knod->sdma[idx].idx =3D 0; + + err =3D knod_create_event(knod->process, KFD_IOC_EVENT_SIGNAL, + true, 1, &knod->sdma_event[idx]); + if (err) { + knod_err(" failed to create SDMA event[%d] err=3D%d\n", + idx, err); + goto err_free_sdma; + } + + memset(&qp, 0, sizeof(qp)); + qp.type =3D KFD_QUEUE_TYPE_SDMA; + qp.format =3D KFD_QUEUE_FORMAT_PM4; + qp.queue_percent =3D 100; + qp.priority =3D 15; + qp.queue_address =3D (u64)knod->sdma[idx].sdma->gaddr; + qp.queue_size =3D knod->sdma[idx].sdma->size; + qp.write_ptr =3D (void __user *) + ((u64)knod->sdma[idx].queue->gaddr + 0x08); + qp.read_ptr =3D (void __user *) + ((u64)knod->sdma[idx].queue->gaddr + 0x10); + + sdma_signal =3D (struct amd_signal *) + knod->sdma[idx].queue_signal->kaddr; + sdma_signal->kind =3D AMD_SIGNAL_KIND_USER; + sdma_signal->event_id =3D knod->sdma_event[idx].id; + sdma_signal->queue_ptr =3D 0; + sdma_signal->event_mailbox_ptr =3D knod->mailbox->gaddr + + (knod->sdma_event[idx].slot * 8); + /* SDMA fence writes u32 to low 32 bits of value. + * Init to 0 (not -1 like AQL) so comparison works. + * value is in union with hardware_doorbell_ptr - set last. + */ + sdma_signal->value =3D 0; + sdma_lptr =3D knod->mailbox->kaddr + (idx * 8); + sdma_lptr[knod->sdma_event[idx].slot] =3D 0; + + err =3D knod_create_queue(knod->process, &qp, pdd->user_gpu_id, + &knod->sdma_queue_id[idx], + &knod->sdma_doorbell_offset[idx]); + if (err) { + knod_err(" failed to create SDMA queue[%d] err=3D%d\n", + idx, err); + goto err_free_sdma; + } + knod->sdma_queue_created[idx] =3D true; + + knod_init_queue(knod, knod->sdma_queue_id[idx], idx, + KFD_IOC_QUEUE_TYPE_SDMA); + knod->sdma[idx].doorbell =3D + (u64 *)((u8 *)ptr + + (u32)knod->sdma_doorbell_offset[idx]); + knod_dbg(" SDMA queue[%d] created: id=3D%d doorbell_offset=3D0x%x doorbe= ll=3D%p\n", + idx, knod->sdma_queue_id[idx], + (u32)knod->sdma_doorbell_offset[idx], + knod->sdma[idx].doorbell); + } + + list_for_each_entry(pqn, &knod->process->pqm.queues, + process_queue_list) { + if (!pqn->q) + continue; + q =3D pqn->q; + for (idx =3D 0; idx < queue_cnt; idx++) { + if (q->properties.queue_id =3D=3D idx) { + knod->kaql[idx].doorbell =3D + (u64 *)((u8 *)ptr + + (u32)knod->aql_doorbell_offset[idx]); + q->properties.doorbell_ptr =3D + knod->kaql[idx].doorbell; + knod_dbg(" doorbell: idx=3D%d qid=3D%d door_off=3D0x%x doorbell=3D%p\n= ", + idx, q->properties.queue_id, + (u32)knod->aql_doorbell_offset[idx], + knod->kaql[idx].doorbell); + } + } + } + + list_add_tail(&knod->list, &ctx_list); + + /* Create shared debugfs directory under dri//knod/ */ + { + struct drm_minor *minor =3D adev_to_drm(pdd->dev->adev)->render; + + if (minor && minor->debugfs_root) { + struct dentry *dir; + + dir =3D debugfs_lookup("knod", minor->debugfs_root); + if (!dir) { + dir =3D debugfs_create_dir("knod", + minor->debugfs_root); + if (IS_ERR(dir)) + dir =3D NULL; + } + knod->debug_dir =3D dir; + } + } + + return knod; + +err_free_sdma: + { + int j; + + for (j =3D idx; j >=3D 0; j--) { + if (knod->sdma_queue_created[j]) { + knod_destroy_queue(knod->process, + knod->sdma_queue_id[j]); + knod->sdma_queue_created[j] =3D false; + } + if (knod->sdma_event[j].id) { + knod_destroy_event(knod->process, + knod->sdma_event[j].id); + knod->sdma_event[j].id =3D 0; + } + knod_free_mem(knod, knod->sdma[j].sdma); + knod_free_mem(knod, knod->sdma[j].queue); + knod_free_mem(knod, knod->sdma[j].queue_signal); + } + } + idx =3D queue_cnt; +err_free_queues: + while (idx-- > 0) + knod_destroy_one_queue(knod, idx); +err_free_bufs: + for (idx =3D 0; idx < channels; idx++) + knod_free_mem(knod, knod->buf[idx]); + kfree(knod->buf); +err_free_mailbox: + knod_free_mem(knod, knod->mailbox); + knod_free_mem(knod, knod->kernels[1]); + knod_free_mem(knod, knod->kernels[0]); + gen_pool_destroy(knod->pool); + iounmap(ptr); + kfd_unref_process(knod->process); + fput(knod->drm_file); +err_free_knod: + put_task_struct(knod->umh_task); + stop_umh(knod->umh_pid); + kfree(knod); + return ERR_PTR(err); +} +EXPORT_SYMBOL(knod_alloc_ctx); + +void knod_release_ctx(struct knod *knod) +{ + int idx; + + list_del(&knod->list); + + debugfs_remove_recursive(knod->debug_dir); + + /* Release SDMA queues - destroy queue/event BEFORE freeing BOs. + * Same ordering as knod_destroy_one_queue() for AQL: the queue + * holds references to BO VAs, so freeing BOs first causes the + * queue destroy to fail and leaves internal BOs in kfd_bo_list. + */ + for (idx =3D 0; idx < knod->sdma_cnt; idx++) { + knod_destroy_queue(knod->process, knod->sdma_queue_id[idx]); + knod_destroy_event(knod->process, knod->sdma_event[idx].id); + knod_free_mem(knod, knod->sdma[idx].sdma); + knod_free_mem(knod, knod->sdma[idx].queue); + knod_free_mem(knod, knod->sdma[idx].queue_signal); + } + + for (idx =3D 0; idx < knod->queue_cnt; idx++) + knod_destroy_one_queue(knod, idx); + + for (idx =3D 0; idx < knod->channels; idx++) + knod_free_mem(knod, knod->buf[idx]); + kfree(knod->buf); + + knod_free_mem(knod, knod->mailbox); + knod_free_mem(knod, knod->kernels[1]); + knod_free_mem(knod, knod->kernels[0]); + gen_pool_destroy(knod->pool); + iounmap(knod->doorbell_base); + + /* Tear down the KFD process. It holds 3 refs: + * ref 1: "open" ref from kfd_create_process (normally dropped + * by kfd_release when /dev/kfd is closed - KNOD never + * opens /dev/kfd so we must drop this ourselves) + * ref 2: KNOD's explicit kref_get in knod_alloc_ctx + * ref 3: mmu_notifier alloc_notifier ref (dropped by + * free_notifier callback via SRCU after mmu_notifier_put) + * + * kfd_process_notifier_release_internal removes from hash, + * destroys queues, and calls mmu_notifier_put which schedules + * the SRCU callback to drop ref 3. We drop refs 1 and 2 here. + * After SRCU fires, ref reaches 0 -> kfd_process_wq_release + * runs (sysfs removal, BO/PDD/doorbell/event cleanup, kfree). + * + * kfd_process_destroy_pdds fput's pdd->drm_file (the get_file + * ref from init_vm), bringing the DRM file ref from 2->1. + * The file/VM stays alive until we fput the filp_open ref below. + */ + kfd_process_notifier_release_internal(knod->process); + kfd_unref_process(knod->process); + kfd_unref_process(knod->process); + mmu_notifier_synchronize(); + kfd_process_flush_wq(); + + /* Drop the filp_open ref (file ref 1->0). kfd_process_destroy_pdds + * already fput'd the get_file ref during wq_release above. + */ + fput(knod->drm_file); + flush_delayed_fput(); + + stop_umh(knod->umh_pid); + put_task_struct(knod->umh_task); + kfree(knod); +} +EXPORT_SYMBOL(knod_release_ctx); + +/* ---- feature control (knod genetlink) ---- */ + +static void *knod_feature_ops(enum knod_feature feat) +{ + switch (feat) { + case KNOD_FEATURE_BPF: + return registered_xdp_ops; + case KNOD_FEATURE_IPSEC: + return accel_ops.ipsec_ops; + default: + return NULL; + } +} + +/* + * Feature lifecycle across three independent axes: + * init/exit attach/detach permanent per-attach state + * activate/deactivate feature select feature GPU resources + * start/stop interface up/down worker + GPU in-flight drain + * + * The worker only does useful work while (interface up AND a feature is + * selected), so a feature switch is bracketed stop -> change -> start: the + * worker is stopped and its GPU dispatch drained before deactivate() frees + * the resources it used, then restarted afterwards. start/stop never tou= ch + * the framework/NIC-owned RX SPSC ring (mlx5 co-owns it via rq->knodev), = so + * they are safe to run while the interface is up and traffic is flowing. + */ +static void knod_feature_stop(struct knod *knod) +{ + struct knod_dev *knodev =3D knod->accel->knodev; + + knod_stop_worker(knod); + + switch (knod->active_feature) { + case KNOD_FEATURE_BPF: + if (registered_xdp_ops && registered_xdp_ops->stop) + registered_xdp_ops->stop(knodev); + break; + case KNOD_FEATURE_IPSEC: + if (accel_ops.ipsec_ops && accel_ops.ipsec_ops->stop) + accel_ops.ipsec_ops->stop(knodev); + break; + default: + break; + } +} + +static void knod_feature_start(struct knod *knod) +{ + struct knod_dev *knodev =3D knod->accel->knodev; + + switch (knod->active_feature) { + case KNOD_FEATURE_BPF: + if (registered_xdp_ops && registered_xdp_ops->start) + registered_xdp_ops->start(knodev); + break; + case KNOD_FEATURE_IPSEC: + if (accel_ops.ipsec_ops && accel_ops.ipsec_ops->start) + accel_ops.ipsec_ops->start(knodev); + break; + default: + knod_start_default_worker(knod); + break; + } +} + +static void knod_feature_deactivate(struct knod *knod) +{ + struct knod_dev *knodev =3D knod->accel->knodev; + + switch (knod->active_feature) { + case KNOD_FEATURE_BPF: + /* + * Phase 1: unregister the offload dev - this force-frees any + * user XDP progs/maps still bound. The map-free ndo routes + * back through accel_ops.xdp_ops->xdp_install, so xdp_ops must + * still point at the BPF ops (and priv must be alive) here. + */ + if (registered_xdp_ops && + registered_xdp_ops->xdp_offload_uninit) + registered_xdp_ops->xdp_offload_uninit(knodev); + /* + * Phase 2: repoint RX delivery at the default drain_pass and + * wait out the in-flight NAPI readers. + */ + WRITE_ONCE(accel_ops.xdp_ops, &default_xdp_ops); + synchronize_net(); + knod_stop_worker(knod); + if (registered_xdp_ops && registered_xdp_ops->deactivate) + registered_xdp_ops->deactivate(knodev); + break; + case KNOD_FEATURE_IPSEC: + /* + * knod_ipsec_detach() clears the netdev xfrm flags and runs + * ->deactivate(), which NULLs ipsec_priv and does its own + * synchronize_net() before freeing. + */ + knod_ipsec_detach(knodev); + break; + default: + break; + } +} + +static int knod_feature_activate(struct knod *knod) +{ + struct knod_dev *knodev =3D knod->accel->knodev; + int err; + + switch (knod->active_feature) { + case KNOD_FEATURE_BPF: + if (!registered_xdp_ops) + return -ENODEV; + /* Phase A: GPU compute buffers. */ + if (registered_xdp_ops->activate) { + err =3D registered_xdp_ops->activate(knodev); + if (err) + return err; + } + /* + * Publish xdp_ops, then phase B: register the offload dev so + * user XDP progs/maps can bind (their install/free route + * through accel_ops.xdp_ops->xdp_install). + */ + WRITE_ONCE(accel_ops.xdp_ops, registered_xdp_ops); + if (registered_xdp_ops->xdp_offload_init) { + err =3D registered_xdp_ops->xdp_offload_init(knodev); + if (err) { + WRITE_ONCE(accel_ops.xdp_ops, &default_xdp_ops); + synchronize_net(); + knod_stop_worker(knod); + if (registered_xdp_ops->deactivate) + registered_xdp_ops->deactivate(knodev); + return err; + } + } + return 0; + case KNOD_FEATURE_IPSEC: + if (!accel_ops.ipsec_ops) + return -ENODEV; + /* knod_ipsec_attach() runs ->activate() + sets netdev flags. */ + return knod_ipsec_attach(knodev); + case KNOD_FEATURE_NONE: + return 0; + default: + return -EINVAL; + } +} + +static int knod_accel_feature_get(struct knod_accel *accel, + u32 *ena, u32 *cap) +{ + struct knod *knod =3D READ_ONCE(accel->priv); + u32 mask =3D 0; + int i; + + /* A registered-but-not-attached accel has no context yet. */ + *ena =3D knod ? knod->active_feature : KNOD_FEATURE_NONE; + for (i =3D 0; i < KNOD_FEATURE_MAX; i++) + if (i =3D=3D KNOD_FEATURE_NONE || knod_feature_ops(i)) + mask |=3D BIT(i); + *cap =3D mask; + return 0; +} + +static int knod_accel_feature_set(struct knod_accel *accel, u32 feature, + struct netlink_ext_ack *extack) +{ + struct knod *knod =3D READ_ONCE(accel->priv); + struct knod_dev *knodev; + bool started; + int err =3D 0; + + if (feature >=3D KNOD_FEATURE_MAX) { + NL_SET_ERR_MSG(extack, "unknown feature"); + return -EINVAL; + } + if (feature !=3D KNOD_FEATURE_NONE && !knod_feature_ops(feature)) { + NL_SET_ERR_MSG(extack, "feature not available on this accelerator"); + return -ENOENT; + } + knodev =3D knod ? READ_ONCE(knod->accel->knodev) : NULL; + if (!knod || !knodev) { + NL_SET_ERR_MSG(extack, "accelerator not attached to a NIC"); + return -ENODEV; + } + if (feature =3D=3D knod->active_feature) + return 0; + + /* + * Refuse to leave BPF while a user XDP prog or offloaded map is still + * bound - tearing the offload down underneath them is unsafe (the GPU + * dispatch keeps running against freed state). The user must detach + * first, e.g. "ip link set dev xdp off". + */ + if (knod->active_feature =3D=3D KNOD_FEATURE_BPF && registered_xdp_ops && + registered_xdp_ops->busy && registered_xdp_ops->busy(knodev)) { + NL_SET_ERR_MSG(extack, "detach the XDP program/maps first"); + return -EBUSY; + } + if (knod->active_feature =3D=3D KNOD_FEATURE_IPSEC && accel_ops.ipsec_ops= && + accel_ops.ipsec_ops->busy && accel_ops.ipsec_ops->busy(knodev)) { + NL_SET_ERR_MSG(extack, "remove the offloaded xfrm SAs first"); + return -EBUSY; + } + + /* + * stop -> change -> start. The worker only runs while the interface + * is up; bracket the resource change with worker stop/start in that + * case. stop() drains the GPU in-flight so deactivate() frees safely. + */ + started =3D READ_ONCE(knodev->started); + if (started) + knod_feature_stop(knod); + knod_feature_deactivate(knod); + knod->active_feature =3D KNOD_FEATURE_NONE; + + if (feature !=3D KNOD_FEATURE_NONE) { + knod->active_feature =3D feature; + err =3D knod_feature_activate(knod); + if (err) { + knod->active_feature =3D KNOD_FEATURE_NONE; + NL_SET_ERR_MSG(extack, "failed to activate feature"); + } + } + if (started) + knod_feature_start(knod); + return err; +} + +static int knod_attach(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + int render_idx =3D accel->id / KNOD_MAX_AQL; + struct net_device *netdev =3D knodev->netdev; + struct amdgpu_device *adev; + struct knod *knod; + + { + int q_cnt =3D clamp(READ_ONCE(knod_requested_queue_cnt), 1, + KNOD_MAX_QUEUE_CNT); + + knod =3D knod_alloc_ctx(knodev, q_cnt, render_idx, + min(netdev->num_rx_queues, + KNOD_SPSC_MAX)); + } + if (IS_ERR(knod)) { + pr_err("knod: Failed to allocate context\n"); + return -EINVAL; + } + + accel->priv =3D knod; + knod->accel =3D accel; + + /* + * Keep GFX engine out of GFXOFF while a KNOD accel is attached. + * On RDNA2 (tested RX6600 / gfx1032) the very first AQL dispatch + * after boot hangs with signal=3D-1 when GFXOFF is active - SMU exit + * from GFXOFF races with the doorbell ring and the completion + * signal is never decremented. Forcing GFXOFF off for the attached + * lifetime avoids the race entirely. + * + * Pin MCLK soft-min to HW max. KNOD's RX path is bandwidth-bound + * on VRAM (NIC->GPU p2pdma delivers frames directly to VRAM, then + * the shader streams them back out) but the individual dispatches + * are too short for SMU's activity monitor to react - MCLK sticks + * at the lowest DPM (~96 MHz on RX6600) in AUTO mode and caps + * throughput far below what the compute path can sustain. Switching + * to the COMPUTE power profile did not help on RDNA2: the COMPUTE + * DpmActivityMonitor coefficients tune GFX upclock aggressively + * but leave memory activity detection conservative. + * + * Setting soft_min via SetSoftMinByFreq is independent of + * pp_power_profile_mode and of power_dpm_force_performance_level, + * so AUTO governance stays in effect for SCLK/voltage - we get the + * same 30W full-throughput state the user reaches via "force + * performance =3D manual + echo 3 > pp_dpm_mclk", without the 75W + * voltage pin that PROFILE_PEAK imposes. Passing 0xFFFF MHz lets + * SMU firmware clamp to the actual hardware max. + */ + adev =3D knod->process->pdds[0]->dev->adev; + amdgpu_gfx_off_ctrl_immediate(adev, false); + amdgpu_dpm_set_soft_freq_range(adev, PP_MCLK, 0xFFFF, 0xFFFF); + + /* + * Attach settles in KNOD_FEATURE_NONE with no worker running. The + * worker is started by the NIC driver bringing the interface up + * (knod_dev_start -> ->dev_start), and each feature's GPU resources + * are allocated when the feature is selected (->activate). + */ + knod->active_feature =3D KNOD_FEATURE_NONE; + + /* + * Permanent per-attach feature state (e.g. the BPF bpf_offload_dev, + * which must outlive feature switches so user XDP progs/maps survive). + */ + if (registered_xdp_ops && registered_xdp_ops->init) + registered_xdp_ops->init(knodev); + return 0; +} + +static void knod_pre_detach(struct knod_dev *knodev) +{ + struct knod *knod =3D knodev->accel->priv; + + /* + * Detach requires the interface down, so the worker is already + * stopped; stop again defensively, free the active feature's + * resources, then tear down the permanent per-attach state. + */ + knod_feature_stop(knod); + knod_feature_deactivate(knod); + knod->active_feature =3D KNOD_FEATURE_NONE; + + if (registered_xdp_ops && registered_xdp_ops->exit) + registered_xdp_ops->exit(knodev); +} + +static void knod_dev_start_worker(struct knod_dev *knodev) +{ + struct knod *knod =3D knodev->accel->priv; + + /* Interface up: start the current feature's worker. */ + if (knod) + knod_feature_start(knod); +} + +static void knod_dev_stop_worker(struct knod_dev *knodev) +{ + struct knod *knod =3D knodev->accel->priv; + + /* Interface down: stop the worker + drain the GPU in-flight. */ + if (knod) + knod_feature_stop(knod); +} + +static void knod_detach(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod *knod =3D accel->priv; + struct amdgpu_device *adev =3D knod->process->pdds[0]->dev->adev; + + knod_stop_worker(knod); + knod_release_ctx(knod); + WRITE_ONCE(accel->priv, NULL); + + /* + * Restore default MCLK range. min=3D1 triggers SetSoftMinByFreq (the + * API skips the call when min=3D=3D0) and SMU clamps to HW min; max + * 0xFFFF clamps to HW max. Together this matches the pre-attach + * "no soft constraint" state so DPM can idle MCLK back down. + */ + amdgpu_dpm_set_soft_freq_range(adev, PP_MCLK, 1, 0xFFFF); + amdgpu_gfx_off_ctrl(adev, true); +} + +static void *knod_accel_alloc_mem(struct knod_dev *knodev, size_t size, + u64 *gaddr, struct page ***pages, void **priv) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod *knod =3D accel->priv; + struct knod_mem *mem; + struct ttm_tt *tt; + u32 flags; + + /* SPSC rings are hot producer/consumer control data. Keep them plain + * GTT; only host-read delivery buffers need coherent CPU visibility. + */ + flags =3D KFD_IOC_ALLOC_MEM_FLAGS_GTT | KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE; + if (pages) + flags |=3D KFD_IOC_ALLOC_MEM_FLAGS_COHERENT; + + mem =3D knod_alloc_mem(knod, size, flags); + if (IS_ERR(mem)) + return NULL; + + if (pages) { + tt =3D mem->mem->bo ? mem->mem->bo->tbo.ttm : NULL; + if (!tt || !tt->pages) { + knod_free_mem(knod, mem); + return NULL; + } + *pages =3D tt->pages; + } + + *gaddr =3D mem->gaddr; + *priv =3D mem; + return mem->kaddr; +} + +static void knod_accel_free_mem(struct knod_dev *knodev, void *priv) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod *knod =3D accel->priv; + struct knod_mem *mem =3D priv; + + knod_free_mem(knod, mem); +} + +/* + * Device->host copy primitives for the common knod_d2h_copy/knod_d2h_drain + * path. Thin wrappers over the SDMA engine (sdma[0]); the framework owns= the + * pending ring, fence counter and dst pool. + */ +static u32 knod_accel_d2h_submit(struct knod_dev *knodev, u64 dst, int que= ue, + u32 page_idx, u16 off, u32 len) +{ + struct knod *knod =3D knodev->accel->priv; + struct knod_sdma_copy_desc c; + + /* The netmem page_pool dma_addr is the NIC address; SDMA needs + * the GPU VM address, so derive the source from buf[queue]. + */ + c.src =3D knod->buf[queue]->gaddr + ((u64)page_idx << PAGE_SHIFT) + off; + c.dst =3D dst; + c.len =3D len; + return knod_sdma_submit(knod, 0, &c, 1); +} + +static void knod_accel_d2h_kick(struct knod_dev *knodev) +{ + knod_sdma_kick(knodev->accel->priv, 0); +} + +static u32 knod_accel_d2h_fence(struct knod_dev *knodev, int sdma_idx) +{ + struct knod *knod =3D READ_ONCE(knodev->accel->priv); + struct knod_mem *signal; + + if (!knod || sdma_idx < 0 || sdma_idx >=3D READ_ONCE(knod->sdma_cnt)) + return 0; + signal =3D READ_ONCE(knod->sdma[sdma_idx].queue_signal); + if (!signal || !signal->kaddr) + return 0; + + return (u32)READ_ONCE(((struct amd_signal *)signal->kaddr)->value); +} + +static int knod_accel_mp_map(struct knod_dev *knodev) +{ + struct knod *knod =3D knodev->accel->priv; + int i; + + if (!knod) + return -ENODEV; + + for (i =3D 0; i < knod->channels; i++) + if (__knod_map_mem(knod, knod->buf[i])) + knod_err(" mp_map failed ch %d\n", i); + + return 0; +} + +static struct knod_accel_ops accel_ops =3D { + .attach =3D knod_attach, + .pre_detach =3D knod_pre_detach, + .detach =3D knod_detach, + .dev_start =3D knod_dev_start_worker, + .dev_stop =3D knod_dev_stop_worker, + .alloc_mem =3D knod_accel_alloc_mem, + .free_mem =3D knod_accel_free_mem, + .mp_map =3D knod_accel_mp_map, + .d2h_submit =3D knod_accel_d2h_submit, + .d2h_kick =3D knod_accel_d2h_kick, + .d2h_fence =3D knod_accel_d2h_fence, + .xdp_ops =3D &default_xdp_ops, + .feature_get =3D knod_accel_feature_get, + .feature_set =3D knod_accel_feature_set, +}; + +/* + * Tear a feature down on every accel still using it before its ops pointer + * is cleared (module unload). Mirrors the feature_set transition to NONE. + */ +static void knod_feature_force_none(enum knod_feature feat) +{ + int i; + + for (i =3D 0; i < nr_accels; i++) { + struct knod_dev *knodev; + struct knod *knod; + bool started; + + if (!accels[i]) + continue; + knod =3D READ_ONCE(accels[i]->priv); + knodev =3D knod ? READ_ONCE(accels[i]->knodev) : NULL; + if (!knod || !knodev || knod->active_feature !=3D feat) + continue; + started =3D READ_ONCE(knodev->started); + if (started) + knod_feature_stop(knod); + knod_feature_deactivate(knod); + knod->active_feature =3D KNOD_FEATURE_NONE; + if (started) + knod_feature_start(knod); + } +} + +void knod_accel_xdp_register(struct knod_accel_xdp_ops *xdp_ops) +{ + int i; + + /* + * Module load advertises the feature and sets up the permanent + * per-attach state (bpf_offload_dev) on already-attached accels; + * GPU compute resources wait for ->activate() on feature select. + */ + WRITE_ONCE(registered_xdp_ops, xdp_ops); + for (i =3D 0; i < nr_accels; i++) { + struct knod_dev *knodev; + + if (!accels[i]) + continue; + knodev =3D READ_ONCE(accels[i]->knodev); + if (knodev && xdp_ops->init) + xdp_ops->init(knodev); + } +} +EXPORT_SYMBOL(knod_accel_xdp_register); + +void knod_accel_xdp_unregister(void) +{ + int i; + + /* Force any active BPF feature off, then drop the permanent state. */ + knod_feature_force_none(KNOD_FEATURE_BPF); + for (i =3D 0; i < nr_accels; i++) { + struct knod_dev *knodev; + + if (!accels[i]) + continue; + knodev =3D READ_ONCE(accels[i]->knodev); + if (knodev && registered_xdp_ops && registered_xdp_ops->exit) + registered_xdp_ops->exit(knodev); + } + WRITE_ONCE(registered_xdp_ops, NULL); +} +EXPORT_SYMBOL(knod_accel_xdp_unregister); + +void knod_accel_ipsec_register(struct knod_accel_ipsec_ops *ipsec_ops) +{ + /* Advertise only; resources are allocated by ->activate() on select. */ + WRITE_ONCE(accel_ops.ipsec_ops, ipsec_ops); +} +EXPORT_SYMBOL(knod_accel_ipsec_register); + +void knod_accel_ipsec_unregister(void) +{ + knod_feature_force_none(KNOD_FEATURE_IPSEC); + WRITE_ONCE(accel_ops.ipsec_ops, NULL); +} +EXPORT_SYMBOL(knod_accel_ipsec_unregister); + +/* + * Accel modules call this from their module_init before NOD attach to + * request the minimum AQL/SDMA queue pair count their dispatcher + * infrastructure needs. The value is a high-water mark: multiple + * callers raise it but never lower it, so whichever module needs the + * most queues wins. + * + * Must be called before knod_attach() runs - i.e. before any + * `echo X,0 > /sys/kernel/debug/knod_dev/attach` - otherwise + * the already-created knod context keeps its old queue_cnt and the + * caller must unbind/rebind to pick up the new value. + * + * Clamped to [1, KNOD_MAX_QUEUE_CNT]. Values outside that range are ignor= ed + * (the internal cap can grow later without ABI break). + */ +void knod_request_queue_cnt(int n) +{ + int cur; + + if (n < 1 || n > KNOD_MAX_QUEUE_CNT) + return; + + do { + cur =3D READ_ONCE(knod_requested_queue_cnt); + if (n <=3D cur) + return; + } while (cmpxchg(&knod_requested_queue_cnt, cur, n) !=3D cur); +} +EXPORT_SYMBOL(knod_request_queue_cnt); + +int knod_init(struct amdgpu_device *adev) +{ + int base_id =3D adev_to_drm(adev)->render->index * KNOD_MAX_AQL; + struct knod_accel *accel; + int i, n; + + n =3D KNOD_MAX_AQL; + + for (i =3D 0; i < n; i++) { + accel =3D kzalloc_obj(struct knod_accel, GFP_KERNEL); + if (!accel) { + pr_err("knod: Failed to allocate accel node %d\n", i); + goto err; + } + + INIT_LIST_HEAD(&accel->list); + accel->type =3D KNOD_TYPE_GPU; + accel->accel_ops =3D &accel_ops; + accel->owner =3D THIS_MODULE; + accel->id =3D base_id + i; + snprintf(accel->name, sizeof(accel->name), "amdgpu-%d", i); + knod_accel_register(accel); + accels[i] =3D accel; + } + nr_accels =3D n; + + return 0; + +err: + while (i-- > 0) { + knod_accel_unregister(accels[i]); + kfree(accels[i]); + accels[i] =3D NULL; + } + return 0; +} + +void knod_fini(struct amdgpu_device *adev) +{ + int base_id =3D adev_to_drm(adev)->render->index * KNOD_MAX_AQL; + int i; + + for (i =3D 0; i < nr_accels; i++) { + if (!accels[i] || + accels[i]->id / KNOD_MAX_AQL !=3D base_id / KNOD_MAX_AQL) + continue; + knod_accel_unregister(accels[i]); + kfree(accels[i]); + accels[i] =3D NULL; + } +} + +void knod_exit(void) +{ + struct knod *knod, *tmp; + int i; + + for (i =3D 0; i < nr_accels; i++) { + if (!accels[i]) + continue; + knod_accel_unregister(accels[i]); + kfree(accels[i]); + accels[i] =3D NULL; + } + nr_accels =3D 0; + + list_for_each_entry_safe(knod, tmp, &ctx_list, list) { + knod_release_ctx(knod); + } +} diff --git a/drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h b/drivers/gpu/drm/a= md/amdkfd/knod/kfd_knod.h new file mode 100644 index 000000000000..a1eafb13a410 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h @@ -0,0 +1,270 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_KNOD_H_ +#define KFD_KNOD_H_ +#include +#include +#include +#include "kfd_hsa.h" +#include + +/* + * knod_dbg() is a pr_debug(), so it is off by default and toggled with + * dynamic debug; knod_err() always fires. + */ +#define knod_dbg(fmt, ...) \ + pr_debug("knod %s:%d" fmt, __func__, __LINE__, ##__VA_ARGS__) +#define knod_err(fmt, ...) \ + pr_err("knod %s:%d" fmt, __func__, __LINE__, ##__VA_ARGS__) + +struct page_pool; + +struct knod_mem { + struct list_head list; + struct kgd_mem *mem; + void *kaddr; + u32 flags; + u32 size; + u32 order; + u64 gaddr; +}; + +union knod_aql_rsrc1 { + struct { +#if defined(__LITTLE_ENDIAN) + unsigned int base_address_hi : 16; + unsigned int stride : 14; + unsigned int cache_swizzle : 1; + unsigned int swizzle_enable : 1; +#elif defined(__LITTLE_ENDIAN) + unsigned int swizzle_enable : 1; + unsigned int cache_swizzle : 1; + unsigned int stride : 14; + unsigned int base_address_hi : 16; +#endif + }; +}; + +struct knod_aql { + struct knod_mem *aql; + struct knod_mem *ctx; + struct knod_mem *queue; + struct knod_mem *scratch; + struct knod_mem *eop; + struct knod_mem *queue_signal; + struct knod_mem *tba; + struct knod_mem *tma; + struct knod_mem *amd_queue; + u64 *doorbell; + int idx; +}; + +struct knod_sdma { + struct knod_mem *sdma; + struct knod_mem *queue; + struct knod_mem *queue_signal; + u64 *doorbell; + int idx; +}; + +/* KFD event handle as knod tracks it: signal event id + its slot index. */ +struct knod_event { + u32 id; + u32 slot; +}; + +typedef int (*knod_worker_fn_t)(void *ctx); +typedef void (*knod_flush_fn_t)(void *ctx); + +enum knod_feature { + KNOD_FEATURE_NONE =3D 0, + KNOD_FEATURE_BPF, + KNOD_FEATURE_IPSEC, + KNOD_FEATURE_MAX, +}; + +/* + * One accel per GPU: NIC:GPU is fixed 1:1 so a single NIC owns the whole + * device. The accel id is just the DRM render index (stride 1). Pipeline + * depth within the one accel is provided by KNOD_MAX_QUEUE_CNT HW queues, + * unrelated to this stride. + */ +#define KNOD_MAX_AQL 1 + +/* Internal AQL/SDMA queue pairs per attached knod context. BPF can keep t= hese + * queues in flight independently while the public accel-id ABI remains st= able. + */ +#define KNOD_MAX_QUEUE_CNT 32 + +#define NR_AQL_RING 16384 +#define AQL_STRUCT_SIZE 128 +struct knod { + struct list_head list; + struct list_head active_list; + + struct gen_pool *pool; + + struct hsa_kernel_dispatch_packet *dp; + pid_t umh_pid; + struct task_struct *umh_task; + u32 nr_aql_ring; + /* AQLs */ + int queue_cnt; + int sdma_cnt; + int igpu; + int isa_version; + /* NAPIs */ + int channels; + struct kfd_process *process; + struct mm_struct *mm; + struct kfd_node *dev; + struct file *drm_file; + void __iomem *doorbell_base; + u64 reserved_addr; + u64 limit_addr; + struct mutex lock; + struct hsa_event *event; + struct knod_mem *kernels[2]; /* dispatch slots: [0] default/pass, [1] BPF= alt */ + struct knod_mem *mailbox; + /* packet data path buf */ + struct knod_mem **buf; + + u32 signal_eid; + u32 completion_eid; + struct knod_aql kaql[NR_CPUS]; + struct knod_sdma sdma[NR_CPUS]; + struct knod_event aql_event[NR_CPUS]; + struct knod_event sdma_event[NR_CPUS]; + u32 aql_queue_id[NR_CPUS]; + u32 sdma_queue_id[NR_CPUS]; + u64 aql_doorbell_offset[NR_CPUS]; + u64 sdma_doorbell_offset[NR_CPUS]; + bool aql_queue_created[NR_CPUS]; + bool sdma_queue_created[NR_CPUS]; + struct kfd_event_data *event_data; + struct knod_accel *accel; + struct dentry *debug_dir; + /* Worker callback - one active worker at a time */ + enum knod_feature active_feature; + knod_worker_fn_t worker_fn; + knod_flush_fn_t flush_fn; + void *worker_ctx; + struct task_struct *worker; +}; + +struct knod_dispatch_params { + u16 workgroup_size_x; + u32 grid_size_x; + u32 grid_size_y; + u32 private_segment_size; + u32 group_segment_size; + u64 kernel_object; + u64 kernarg_address; +}; + +static inline void +knod_setup_invalidate(struct knod *knod, int idx, int q_idx) +{ + struct hsa_kernel_dispatch_packet *dp =3D knod->kaql[q_idx].aql->kaddr; + + dp +=3D idx; + dp->header =3D HSA_PACKET_TYPE_INVALID << HSA_PACKET_HEADER_TYPE; +} + +static inline void +knod_setup_dispatch(struct knod *knod, int idx, + const struct knod_dispatch_params *p, int q_idx) +{ + struct hsa_kernel_dispatch_packet *dp =3D knod->kaql[q_idx].aql->kaddr; + + dp +=3D idx; + dp->setup =3D 2; + dp->workgroup_size_x =3D p->workgroup_size_x; + dp->workgroup_size_y =3D 1; + dp->workgroup_size_z =3D 1; + dp->grid_size_x =3D p->grid_size_x; + dp->grid_size_y =3D p->grid_size_y; + dp->grid_size_z =3D 1; + dp->private_segment_size =3D p->private_segment_size; + dp->group_segment_size =3D p->group_segment_size; + dp->kernel_object =3D p->kernel_object; + dp->kernarg_address =3D (void *)p->kernarg_address; + dp->completion_signal =3D knod->kaql[q_idx].queue_signal->gaddr; + /* publish the packet body before the valid header (WRITE_ONCE below) */ + wmb(); + WRITE_ONCE(dp->header, + (HSA_PACKET_TYPE_KERNEL_DISPATCH << + HSA_PACKET_HEADER_TYPE) | + (HSA_FENCE_SCOPE_SYSTEM << + HSA_PACKET_HEADER_ACQUIRE_FENCE_SCOPE) | + (HSA_FENCE_SCOPE_SYSTEM << + HSA_PACKET_HEADER_RELEASE_FENCE_SCOPE)); +} + +static inline void +knod_setup_header(struct knod *knod, + const struct knod_dispatch_params *p, int q_idx) +{ + struct amd_queue *amd_queue =3D (struct amd_queue *)knod->kaql[q_idx].amd= _queue->kaddr; + int curr_idx =3D knod->kaql[q_idx].idx; + int next_idx =3D curr_idx + 1; + u64 *ptr =3D knod->kaql[q_idx].doorbell; + + knod_setup_invalidate(knod, next_idx % knod->nr_aql_ring, q_idx); + knod_setup_dispatch(knod, curr_idx % knod->nr_aql_ring, p, q_idx); + WRITE_ONCE(amd_queue->write_dispatch_id, curr_idx); + writeq(curr_idx, ptr); + knod->kaql[q_idx].idx =3D next_idx; +} + +#define KNOD_NR_AQL_DEFAULT 1 +struct knod *knod_alloc_ctx(struct knod_dev *knodev, int queue_cnt, int id, + int channels); +void knod_release_ctx(struct knod *knod); +void knod_accel_xdp_register(struct knod_accel_xdp_ops *xdp_ops); +void knod_accel_xdp_unregister(void); +void knod_accel_ipsec_register(struct knod_accel_ipsec_ops *ipsec_ops); +void knod_accel_ipsec_unregister(void); +void knod_request_queue_cnt(int n); +struct knod_mem *knod_alloc_mem(struct knod *knod, size_t size, int flags); +struct knod_mem *__knod_alloc_mem(struct knod *knod, size_t size, int flag= s); +int __knod_map_mem(struct knod *knod, struct knod_mem *mem); +int __knod_export_dma_buf(struct knod *knod, struct knod_mem *mem); +int __knod_map_kaddr(struct knod *knod, struct knod_mem *mem); +void knod_free_mem(struct knod *pknod, struct knod_mem *mem); +void knod_sdma_copy(struct knod *knod, u64 dst_gart_addr, u64 src_gart_add= r, + int idx, int size); +void knod_sdma_fence(struct knod *knod, u64 fence_addr, u32 fence_val, + int idx); +void knod_sdma_trap(struct knod *knod, int idx); +void knod_sdma_doorbell(struct knod *knod, int idx); + +/* One linear GPU->host SDMA copy (GPU VM addresses). */ +struct knod_sdma_copy_desc { + u64 dst; + u64 src; + u32 len; +}; + +/* + * Emit @n copies on sdma[@idx] as one batch with ring backpressure. + * Returns the post-batch ring position to fence/await, or 0 if the ring + * is too full (caller drops the whole batch). Pair with knod_sdma_kick(). + */ +u32 knod_sdma_submit(struct knod *knod, int idx, + const struct knod_sdma_copy_desc *copies, int n); +void knod_sdma_kick(struct knod *knod, int idx); + +int knod_gart_map(struct amdgpu_device *adev, u64 npages, + dma_addr_t *addr, u64 *gart_addr, u64 flags); +int knod_register_worker(struct knod *knod, knod_worker_fn_t fn, + knod_flush_fn_t flush, void *ctx); +void knod_unregister_worker(struct knod *knod); +int knod_wait_on_events(struct kfd_process *p, u32 num_events, + void __user *data, bool all, u32 *user_timeout_ms, + u32 *wait_result); + +#endif /* KFD_KNOD_H_ */ --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E92AB3B7746 for ; Sun, 19 Jul 2026 18:01:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484148; cv=none; b=KLtJl7X1V4CueTso2Tz5SklSl8yTQliqTnp11w1lwjL8lSoQsTTxIxfkoNW6lozU1XmCd22jTdSLBY/vuiEpS3WOh9K/jWGeV6zpRKf/d4e+hR2R/l6h3mLNeSVmRNA+LqmThspu14eJhG2VFCpnKhxZ6eTB9CClPQgOGhvloDQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484148; c=relaxed/simple; bh=zSTt7WOeYt4J4cXT77K8J8ALOPiZfSdpI7rMkfg6IHE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jc5ur11yVqmeIJBBcLVnWUAH9e2Ip4JohCGw4NahDvCK9bdSZOwi1uP9L0/j6ZV1UFWP446OnxvR1aC86wRoxiOVszkzC1K91FvSdiBPyEZglUmrox1y4+9lGCbM8JPNcLkOxm+O7XsvvTCttvlt3MsRg7gUFRYdSJwD4KbmL9I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OMu9beJI; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OMu9beJI" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2caed617615so104306035ad.3 for ; Sun, 19 Jul 2026 11:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484077; x=1785088877; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=OFMO/6wAurP0TcklziZGT9lYnZVfFUdBzO/mH5p+qV0=; b=OMu9beJI4fwf4T8QGkIxf/+YofVii/mOTeme8+y5D0DJ4M1Rglh+1fL3/dvTcEKor7 Ykjzhe2JOL4yVTrkMX8sMx04iROD1SG4HJu0cbF2ml2laybQ2NaX9JN6XWNwEZzYGSnZ 1IXZtsSM4gUZaExwXGiJVwerVQdvvnhQ2sv7vKSms7hIauLuqSdm14sS51eEH1ddlyD5 6gsA0D4eFjo9Nz0QetISkcJzpTCu7kBdSycAjkUEub3EzOe8zZE7wREVbSWTBb4pAXe0 tTszUMKARAnQ60j0fNGqr98zqHk2qYOb78Iwd2D9SCbmSgRFrkMFzH0qPSHR5GrbrDEP uw6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484077; x=1785088877; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=OFMO/6wAurP0TcklziZGT9lYnZVfFUdBzO/mH5p+qV0=; b=EJy/LJXgeawX0yb6simbIrsr18HxGIbWFve3xF0livqiMm8ai0hh4J4+3BUrfKxaMy JK2x/L5cWZS/YlgweSeX2Y32CncMfo+d8GO3Fb14b2SBmx30igvD9T+Hj65rkEhS1km2 8m2ELqj9X8ydjp6nTs8GUz297qa/dJ0W1pSzz1crSLOvukalW8FXKLhyUwkcnEMn+QZs pjR7PsMTFjBlWPbT9B8M7KxtLhrJTdS/hiM0I6OZt3g9vo5Uzh7T5J+uGoHbb/t1GpaO sooFoA2Hv+vzOPTPBqklV1VyVcHku4czrZ8EUmo3ZDaer8Dny2HrB0k0qo2Pisf416hk sc/g== X-Forwarded-Encrypted: i=1; AHgh+RpBtz6YvkmSMzPct7GArpz+RA49gimwL+km/9bfJRr3FsI5210eXpA9jjZxmHNo/uABeZASFMHrCaA686M=@vger.kernel.org X-Gm-Message-State: AOJu0YyLEDfY370kX1CeS0pPMMPwXyY2DOaDMUS2QjI0nTtyeFs5SoOo 376Vz/gov+hoeTJ1gLWO7ZngTqAEjlCl0vADHxyh5SYgS+d2BKsOJVGR X-Gm-Gg: AfdE7ckIj1ZZX3vYDz81GW1vFJUlKMSUUPfp4inSM9MM50w/dGy+ncKJA3+kJ2hzT4d lPu1jV/2FZ+i1ULH90gDhrWRfBUzVigPLp2d1YJSeyZiGIPjaSaiYKaGsAV0ENlPPtoLYc4S7ZF Oz3Dtqf//sQdFRU+0b+Y2RbLHzj1pBp7qWj2Kg4q4Vh7blvBEPten3CKkkftWVnvoVl17CnGShE 6szEkSmjza/ldmLGM33/MPi6SXzK0wsiaxk8GYmXCvwzrLtltuxqfaWlE4Krba/dc0OIu9zmkUs Ti+/wSgFizYFAo2JcJsMkgR7lrYSMw1BfZGdH49b+MRXiOyfaSMEvubDz6fxZxcc+57aExthFxr DqfooJqD7wopLMj8ZSj+E2/KZMoExzLTgNRUXR8CUVkFvNG6FT5Q/qoLqG0/nY6qG+g== X-Received: by 2002:a17:902:e5cb:b0:2cc:9763:e607 with SMTP id d9443c01a7336-2cf349bc5c8mr112975505ad.27.1784484075635; Sun, 19 Jul 2026 11:01:15 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.01.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:01:14 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 08/13] drm/amdkfd: add GPU instruction emitter and disassembler Date: Sun, 19 Jul 2026 17:58:52 +0000 Message-ID: <20260719175857.4071636-9-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the AMD GCN (gfx9/gfx10) instruction encoder used to build the GPU shaders that knod dispatches, plus a matching disassembler used for debugging the generated code. Signed-off-by: Taehee Yoo (cherry picked from commit bbbe8531de11017f565a922b072d4aa5f99674fc) --- drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h | 173 + .../drm/amd/amdkfd/knod/knod_amdgpu_insn.h | 6362 +++++++++++++++++ .../gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h | 3978 +++++++++++ .../gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h | 4068 +++++++++++ 4 files changed, 14581 insertions(+) create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h b/drivers/gpu/dr= m/amd/amdkfd/knod/knod_amdgpu.h new file mode 100644 index 000000000000..77d1c6afdd0a --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h @@ -0,0 +1,173 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_AMDGPU_H_INCLUDED +#define KFD_AMDGPU_H_INCLUDED + +#define KNOD_AMDGPU_REG_PAIR(x) ((x) / 2) + +enum amdgcn_param_type { + AMDGCN_PARAM_TYPE_SGPR, + AMDGCN_PARAM_TYPE_VCC_LO, + AMDGCN_PARAM_TYPE_VCC_HI, + AMDGCN_PARAM_TYPE_TTPM_BASE, + AMDGCN_PARAM_TYPE_M0, + AMDGCN_PARAM_TYPE_NULL, + AMDGCN_PARAM_TYPE_EXEC_LO, + AMDGCN_PARAM_TYPE_EXEC_HI, + AMDGCN_PARAM_TYPE_INTEGER_0, + AMDGCN_PARAM_TYPE_INTEGER_MINUS_1, + AMDGCN_PARAM_TYPE_SHARED_BASE, + AMDGCN_PARAM_TYPE_SHARED_LIMIT, + AMDGCN_PARAM_TYPE_PRIVATE_BASE, + AMDGCN_PARAM_TYPE_PRIVATE_LIMIT, + AMDGCN_PARAM_TYPE_POPS_EXITING_WAVE_ID, + AMDGCN_PARAM_TYPE_SDWA, + AMDGCN_PARAM_TYPE_DPP16, + AMDGCN_PARAM_TYPE_VCCZ, + AMDGCN_PARAM_TYPE_EXECZ, + AMDGCN_PARAM_TYPE_SCC, + AMDGCN_PARAM_TYPE_LITERAL_CONST, + AMDGCN_PARAM_TYPE_VGPR, + __AMDGCN_PARAM_TYPE_MAX, +}; + +struct amdgcn_param32 { + enum amdgcn_param_type type; + int v; +}; + +struct amdgcn_param64 { + struct amdgcn_param32 lo; + struct amdgcn_param32 hi; + u64 imm; +}; + +inline void knod_set(struct amdgcn_param32 *param, + enum amdgcn_param_type type, int v) +{ + param->type =3D type; + param->v =3D v; +} + +inline void knod_vset32(struct amdgcn_param32 *param, int v) +{ + param->type =3D AMDGCN_PARAM_TYPE_VGPR; + param->v =3D v; +} + +inline void knod_vset64(struct amdgcn_param64 *param, int v) +{ + param->lo.type =3D AMDGCN_PARAM_TYPE_VGPR; + param->lo.v =3D v; + param->hi.type =3D AMDGCN_PARAM_TYPE_VGPR; + param->hi.v =3D v + 1; +} + +inline void knod_sset32(struct amdgcn_param32 *param, int v) +{ + param->type =3D AMDGCN_PARAM_TYPE_SGPR; + param->v =3D v; +} + +inline void knod_sset64(struct amdgcn_param64 *param, int v) +{ + param->lo.type =3D AMDGCN_PARAM_TYPE_SGPR; + param->lo.v =3D v; + param->hi.type =3D AMDGCN_PARAM_TYPE_SGPR; + param->hi.v =3D v + 1; +} + +inline void knod_iset32(struct amdgcn_param32 *param, int v) +{ + if (v > 64 || v < -16) { + param->type =3D AMDGCN_PARAM_TYPE_LITERAL_CONST; + param->v =3D v; + } else if (v >=3D 0) { + param->type =3D AMDGCN_PARAM_TYPE_INTEGER_0; + param->v =3D v; + } else { + param->type =3D AMDGCN_PARAM_TYPE_INTEGER_MINUS_1; + param->v =3D ~v; + } +} + +inline void knod_iset64(struct amdgcn_param64 *param, + u64 v) +{ + int imm0 =3D v & ~0U; + int imm1 =3D v >> 32; + + param->imm =3D v; + + if (imm0 > 64 || imm0 < -16) { + param->lo.type =3D AMDGCN_PARAM_TYPE_LITERAL_CONST; + param->lo.v =3D imm0; + } else if (imm0 >=3D 0) { + param->lo.type =3D AMDGCN_PARAM_TYPE_INTEGER_0; + param->lo.v =3D imm0; + } else { + param->lo.type =3D AMDGCN_PARAM_TYPE_INTEGER_MINUS_1; + param->lo.v =3D ~imm0; + } + + if (imm1 > 64 || imm1 < -16) { + param->hi.type =3D AMDGCN_PARAM_TYPE_LITERAL_CONST; + WARN_ON_ONCE(1); + param->hi.v =3D imm1; + } else if (imm1 >=3D 0) { + param->hi.type =3D AMDGCN_PARAM_TYPE_INTEGER_0; + param->hi.v =3D imm1; + } else { + param->hi.type =3D AMDGCN_PARAM_TYPE_INTEGER_MINUS_1; + param->hi.v =3D ~imm1; + } +} + +inline void knod_lset32(struct amdgcn_param32 *param, int v) +{ + param->type =3D AMDGCN_PARAM_TYPE_LITERAL_CONST; + param->v =3D v; +} + +inline void knod_vccset(struct amdgcn_param32 *param) +{ + param->type =3D AMDGCN_PARAM_TYPE_VCC_LO; + param->v =3D 0; +} + +inline bool knod_param_is_literal(struct amdgcn_param32 param) +{ + return param.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Param constructors - common to GFX9/GFX10 shader emitters. + * + * Usage: pass directly to emit_gfx{9,10}_* functions that take + * struct amdgcn_param32 operands. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +#define P_S(n) ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_SGPR, (n) }) +#define P_V(n) ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VGPR, (n) }) +#define P_I(n) ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_INTEGER_0, (n) = }) +#define P_L(v) ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_LITERAL_CONST, = (v) }) +#define P_VCC ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VCC_LO, 0 }) +#define P_EXEC ((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_EXEC_LO, 0 }) + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Branch patching - operates directly on u32 *buf + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static inline void patch_branch(u32 *buf, int patch_pos, int target_pos) +{ + int offset =3D target_pos - (patch_pos + 1); + + buf[patch_pos] =3D (buf[patch_pos] & 0xFFFF0000u) | (offset & 0xFFFF); +} + +#endif diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h b/drivers/g= pu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h new file mode 100644 index 000000000000..9703bf781ff5 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h @@ -0,0 +1,6362 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_AMDGPU_INSN_H_INCLUDED +#define KFD_AMDGPU_INSN_H_INCLUDED + +#include "knod_amdgpu.h" +#include "knod_gfx10_insn.h" +#include "knod_gfx9_insn.h" + +enum amdgcn_insn_type { + AMDGCN_INSN_TYPE_SOP2, + AMDGCN_INSN_TYPE_SOPK, + AMDGCN_INSN_TYPE_SOP1, + AMDGCN_INSN_TYPE_SOPC, + AMDGCN_INSN_TYPE_SOPP, + AMDGCN_INSN_TYPE_SMEM, + AMDGCN_INSN_TYPE_VOP2, + AMDGCN_INSN_TYPE_VOP1, + AMDGCN_INSN_TYPE_VOPC, + AMDGCN_INSN_TYPE_VOP3A, + AMDGCN_INSN_TYPE_VOP3B, + AMDGCN_INSN_TYPE_VOP3P, + AMDGCN_INSN_TYPE_SDWA, + AMDGCN_INSN_TYPE_SDWAB, + AMDGCN_INSN_TYPE_DPP16, + AMDGCN_INSN_TYPE_DPP8, + AMDGCN_INSN_TYPE_VINTRP, + AMDGCN_INSN_TYPE_DS, + AMDGCN_INSN_TYPE_MTBUF, + AMDGCN_INSN_TYPE_MUBUF, + AMDGCN_INSN_TYPE_MIMG, + AMDGCN_INSN_TYPE_FLAT, + AMDGCN_INSN_TYPE_EXP, + __AMDGCN_INSN_TYPE_MAX, +}; + +static const char opnames_gfx10[__AMDGCN_INSN_TYPE_MAX][900][30] =3D { + [AMDGCN_INSN_TYPE_VOP3A] =3D { + [GFX10_V_FMA_LEGACY_F32] =3D "v_fma_legacy_f32", + [GFX10_V_MAD_I32_I24] =3D "V_MAD_I32_I24", + [GFX10_V_MAD_U32_U24] =3D "v_mad_u32_u24", + [GFX10_V_CUBEID_F32] =3D "v_cubeid_f32", + [GFX10_V_CUBESC_F32] =3D "v_cubesc_f32", + [GFX10_V_CUBETC_F32] =3D "v_cubetc_f32", + [GFX10_V_CUBEMA_F32] =3D "v_cubema_f32", + [GFX10_V_BFE_U32] =3D "v_bfe_u32", + [GFX10_V_BFE_I32] =3D "v_bfe_i32", + [GFX10_V_BFI_B32] =3D "v_bfi_b32", + [GFX10_V_FMA_F32] =3D "v_fma_f32", + [GFX10_V_FMA_F64] =3D "v_fma_f64", + [GFX10_V_LERP_U8] =3D "v_lerp_u8", + [GFX10_V_ALIGNBIT_B32] =3D "v_alignbit_b32", + [GFX10_V_ALIGNBYTE_B32] =3D "v_alignbyte_b32", + [GFX10_V_MULLIT_F32] =3D "v_mullit_f32", + [GFX10_V_MIN3_F32] =3D "v_min3_f32", + [GFX10_V_MIN3_I32] =3D "v_min3_i32", + [GFX10_V_MIN3_U32] =3D "v_min3_u32", + [GFX10_V_MAX3_F32] =3D "v_max3_f32", + [GFX10_V_MAX3_I32] =3D "v_max3_i32", + [GFX10_V_MAX3_U32] =3D "v_max3_u32", + [GFX10_V_MED3_F32] =3D "v_med3_f32", + [GFX10_V_MED3_I32] =3D "v_med3_i32", + [GFX10_V_MED3_U32] =3D "v_med3_u32", + [GFX10_V_SAD_U8] =3D "v_sad_u8", + [GFX10_V_SAD_HI_U8] =3D "v_sad_hi_u8", + [GFX10_V_SAD_U16] =3D "v_sad_u16", + [GFX10_V_SAD_U32] =3D "v_sad_u32", + [GFX10_V_CVT_PK_U8_F32] =3D "v_cvt_pk_u8_f32", + [GFX10_V_DIV_FIXUP_F32] =3D "v_div_fixup_f32", + [GFX10_V_DIV_FIXUP_F64] =3D "v_div_fixup_f64", + [GFX10_V_ADD_F64] =3D "v_add_f64", + [GFX10_V_MUL_F64] =3D "v_mul_f64", + [GFX10_V_MIN_F64] =3D "v_min_f64", + [GFX10_V_MAX_F64] =3D "v_max_f64", + [GFX10_V_LDEXP_F64] =3D "v_ldexp_f64", + [GFX10_V_MUL_LO_U32] =3D "v_mul_lo_u32", + [GFX10_V_MUL_HI_U32] =3D "v_mul_hi_u32", + [GFX10_V_MUL_HI_I32] =3D "v_mul_hi_i32", + [GFX10_V_DIV_FMAS_F32] =3D "v_div_fmas_f32", + [GFX10_V_DIV_FMAS_F64] =3D "v_div_fmas_f64", + [GFX10_V_MSAD_U8] =3D "v_msad_u8", + [GFX10_V_QSAD_PK_U16_U8] =3D "v_qsad_pk_u16_u8", + [GFX10_V_MQSAD_PK_U16_U8] =3D "v_mqsad_pk_u16_u8", + [GFX10_V_TRIG_PREOP_F64] =3D "v_trig_preop_f64", + [GFX10_V_MQSAD_U32_U8] =3D "v_mqsad_u32_u8", + [GFX10_V_XOR3_B32] =3D "v_xor3_b32", + [GFX10_V_LSHLREV_B64] =3D "v_lshlrev_b64", + [GFX10_V_LSHRREV_B64] =3D "v_lshrrev_b64", + [GFX10_V_ASHRREV_I64] =3D "v_ashrrev_i64", + [GFX10_V_ADD_NC_U16] =3D "v_add_nc_u16", + [GFX10_V_SUB_NC_U16] =3D "v_sub_nc_u16", + [GFX10_V_MUL_LO_U16] =3D "v_mul_lo_u16", + [GFX10_V_LSHRREV_B16] =3D "v_lshrrev_b16", + [GFX10_V_ASHRREV_I16] =3D "v_ashrrev_i16", + [GFX10_V_MAX_U16] =3D "v_max_u16", + [GFX10_V_MAX_I16] =3D "v_max_i16", + [GFX10_V_MIN_U16] =3D "v_min_u16", + [GFX10_V_MIN_I16] =3D "v_min_i16", + [GFX10_V_ADD_NC_I16] =3D "v_add_nc_i16", + [GFX10_V_SUB_NC_I16] =3D "v_sub_nc_i16", + [GFX10_V_PACK_B32_F16] =3D "v_pack_b32_f16", + [GFX10_V_CVT_PKNORM_I16_F16] =3D "v_cvt_pknorm_i16_f16", + [GFX10_V_CVT_PKNORM_U16_F16] =3D "v_cvt_pknorm_u16_f16", + [GFX10_V_LSHLREV_B16] =3D "v_lshlrev_b16", + [GFX10_V_MAD_U16] =3D "v_mad_u16", + [GFX10_V_INTERP_P1LL_F16] =3D "v_interp_p1ll_f16", + [GFX10_V_INTERP_P1LV_F16] =3D "v_interp_p1lv_f16", + [GFX10_V_PERM_B32] =3D "v_perm_b32", + [GFX10_V_XAD_U32] =3D "v_xad_u32", + [GFX10_V_LSHL_ADD_U32] =3D "v_lshl_add_u32", + [GFX10_V_ADD_LSHL_U32] =3D "v_add_lshl_u32", + [GFX10_V_FMA_F16] =3D "v_fma_f16", + [GFX10_V_MIN3_F16] =3D "v_min3_f16", + [GFX10_V_MIN3_I16] =3D "v_min3_i16", + [GFX10_V_MIN3_U16] =3D "v_min3_u16", + [GFX10_V_MAX3_F16] =3D "v_max3_f16", + [GFX10_V_MAX3_I16] =3D "v_max3_i16", + [GFX10_V_MAX3_U16] =3D "v_max3_u16", + [GFX10_V_MED3_F16] =3D "v_med3_f16", + [GFX10_V_MED3_I16] =3D "v_med3_i16", + [GFX10_V_MED3_U16] =3D "v_med3_u16", + [GFX10_V_INTERP_P2_F16] =3D "v_interp_p2_f16", + [GFX10_V_MAD_I16] =3D "v_mad_i16", + [GFX10_V_DIV_FIXUP_F16] =3D "v_div_fixup_f16", + [GFX10_V_READLANE_B32] =3D "v_readlane_b32", + [GFX10_V_WRITELANE_B32] =3D "v_writelane_b32", + [GFX10_V_LDEXP_F32] =3D "v_ldexp_f32", + [GFX10_V_BFM_B32] =3D "v_bfm_b32", + [GFX10_V_BCNT_U32_B32] =3D "v_bcnt_u32_b32", + [GFX10_V_MBCNT_LO_U32_B32] =3D "v_mbcnt_lo_u32_b32", + [GFX10_V_MBCNT_HI_U32_B32] =3D "v_mbcnt_hi_u32_b32", + [GFX10_V_CVT_PKNORM_I16_F32] =3D "v_cvt_pknorm_i16_f32", + [GFX10_V_CVT_PKNORM_U16_F32] =3D "v_cvt_pknorm_u16_f32", + [GFX10_V_CVT_PK_U16_U32] =3D "v_cvt_pk_u16_u32", + [GFX10_V_CVT_PK_I16_I32] =3D "v_cvt_pk_i16_i32", + [GFX10_V_ADD3_U32] =3D "v_add3_u32", + [GFX10_V_LSHL_OR_B32] =3D "v_lshl_or_b32", + [GFX10_V_AND_OR_B32] =3D "v_and_or_b32", + [GFX10_V_OR3_B32] =3D "v_or3_b32", + [GFX10_V_MAD_U32_U16] =3D "v_mad_u32_u16", + [GFX10_V_MAD_I32_I16] =3D "v_mad_i32_i16", + [GFX10_V_SUB_NC_I32] =3D "v_sub_nc_i32", + [GFX10_V_PERMLANE16_B32] =3D "v_permlane16_b32", + [GFX10_V_PERMLANEX16_B32] =3D "v_permlanex16_b32", + [GFX10_V_ADD_NC_I32] =3D "v_add_nc_i32", + }, + [AMDGCN_INSN_TYPE_SOP2] =3D { + [GFX10_S_ADD_U32] =3D "s_add_u32", + [GFX10_S_SUB_U32] =3D "s_sub_u32", + [GFX10_S_ADD_I32] =3D "s_add_i32", + [GFX10_S_SUB_I32] =3D "s_sub_i32", + [GFX10_S_ADDC_U32] =3D "s_addc_u32", + [GFX10_S_SUBB_U32] =3D "s_subb_u32", + [GFX10_S_MIN_I32] =3D "s_min_i32", + [GFX10_S_MIN_U32] =3D "s_min_u32", + [GFX10_S_MAX_I32] =3D "s_max_i32", + [GFX10_S_MAX_U32] =3D "s_max_u32", + [GFX10_S_CSELECT_B32] =3D "s_cselect_b32", + [GFX10_S_CELECT_B64] =3D "s_celect_b64", + [GFX10_S_AND_B32] =3D "s_and_b32", + [GFX10_S_AND_B64] =3D "s_and_b64", + [GFX10_S_OR_B32] =3D "s_or_b32", + [GFX10_S_OR_B64] =3D "s_or_b64", + [GFX10_S_XOR_B32] =3D "s_xor_b32", + [GFX10_S_XOR_B64] =3D "s_xor_b64", + [GFX10_S_ANDN2_B32] =3D "s_andn2_b32", + [GFX10_S_ANDN2_B64] =3D "s_andn2_b64", + [GFX10_S_ORN2_B32] =3D "s_orn2_b32", + [GFX10_S_ORN2_B64] =3D "s_orn2_b64", + [GFX10_S_NAND_B32] =3D "s_nand_b32", + [GFX10_S_NAND_B64] =3D "s_nand_b64", + [GFX10_S_NOR_B32] =3D "s_nor_b32", + [GFX10_S_NOR_B64] =3D "s_nor_b64", + [GFX10_S_XNOR_B32] =3D "s_xnor_b32", + [GFX10_S_XNOR_B64] =3D "s_xnor_b64", + [GFX10_S_LSHL_B32] =3D "s_lshl_b32", + [GFX10_S_LSHL_B64] =3D "s_lshl_b64", + [GFX10_S_LSHR_B32] =3D "s_lshr_b32", + [GFX10_S_LSHR_B64] =3D "s_lshr_b64", + [GFX10_S_ASHR_I32] =3D "s_ashr_i32", + [GFX10_S_ASHR_I64] =3D "s_ashr_i64", + [GFX10_S_BFM_B32] =3D "s_bfm_b32", + [GFX10_S_BFM_B64] =3D "s_bfm_b64", + [GFX10_S_MUL_I32] =3D "s_mul_i32", + [GFX10_S_MUL_I64] =3D "s_mul_i64", + [GFX10_S_BFE_U32] =3D "s_bfe_u32", + [GFX10_S_BFE_I32] =3D "s_bfe_i32", + [GFX10_S_BFE_U64] =3D "s_bfe_u64", + [GFX10_S_ABSDIFF_I32] =3D "s_absdiff_i32", + [GFX10_S_LSHL1_ADD_U32] =3D "s_lshl1_add_u32", + [GFX10_S_LSHL2_ADD_U32] =3D "s_lshl2_add_u32", + [GFX10_S_LSHL3_ADD_U32] =3D "s_lshl3_add_u32", + [GFX10_S_LSHL4_ADD_U32] =3D "s_lshl4_add_u32", + [GFX10_S_PACK_LL_B32_B16] =3D "s_pack_ll_b32_b16", + [GFX10_S_PACK_LH_B32_B16] =3D "s_pack_lh_b32_b16", + [GFX10_S_MUL_HI_U32] =3D "s_mul_hi_u32", + [GFX10_S_MUL_HI_I32] =3D "s_mul_hi_i32", + }, + [AMDGCN_INSN_TYPE_SOPK] =3D { + [GFX10_S_MOVK_I32] =3D "s_movk_i32", + [GFX10_S_VERSION] =3D "s_version", + [GFX10_S_CMOVK_I32] =3D "s_cmovk_i32", + [GFX10_S_CMPK_EQ_I32] =3D "s_cmpk_eq_i32", + [GFX10_S_CMPK_LG_I32] =3D "s_cmpk_lg_i32", + [GFX10_S_CMPK_GT_I32] =3D "s_cmpk_gt_i32", + [GFX10_S_CMPK_GE_I32] =3D "s_cmpk_ge_i32", + [GFX10_S_CMPK_LT_I32] =3D "s_cmpk_lt_i32", + [GFX10_S_CMPK_LE_I32] =3D "s_cmpk_le_i32", + [GFX10_S_CMPK_EQ_U32] =3D "s_cmpk_eq_u32", + [GFX10_S_CMPK_LG_U32] =3D "s_cmpk_lg_u32", + [GFX10_S_CMPK_GT_U32] =3D "s_cmpk_gt_u32", + [GFX10_S_CMPK_GE_U32] =3D "s_cmpk_ge_u32", + [GFX10_S_CMPK_LT_U32] =3D "s_cmpk_lt_u32", + [GFX10_S_CMPK_LE_U32] =3D "s_cmpk_le_u32", + [GFX10_S_ADDK_I32] =3D "s_addk_i32", + [GFX10_S_MULK_I32] =3D "s_mulk_i32", + [GFX10_S_GETREG_B32] =3D "s_getreg_b32", + [GFX10_S_SETREG_B32] =3D "s_setreg_b32", + [GFX10_S_SETREG_IMM32_B32] =3D "s_setreg_imm32_b32", + [GFX10_S_CALL_B64] =3D "s_call_b64", + [GFX10_S_WAITCNT_VSCNT] =3D "s_waitcnt_vscnt", + [GFX10_S_WAITCNT_VMCNT] =3D "s_waitcnt_vmcnt", + [GFX10_S_WAITCNT_EXPCNT] =3D "s_waitcnt_expcnt", + [GFX10_S_WAITCNT_LGKMCNT] =3D "s_waitcnt_lgkmcnt", + [GFX10_S_SUBVECTOR_LOOP_BEGIN] =3D "s_subvector_loop_begin", + [GFX10_S_SUBVECTOR_LOOP_END] =3D "s_subvector_loop_end", + }, + [AMDGCN_INSN_TYPE_SOP1] =3D { + [GFX10_S_MOV_B32] =3D "s_mov_b32", + [GFX10_S_MOV_B64] =3D "s_mov_b64", + [GFX10_S_CMOV_B32] =3D "s_cmov_b32", + [GFX10_S_CMOV_B64] =3D "s_cmov_b64", + [GFX10_S_NOT_B32] =3D "s_not_b32", + [GFX10_S_NOT_B64] =3D "s_not_b64", + [GFX10_S_WQM_B32] =3D "s_wqm_b32", + [GFX10_S_WQM_B64] =3D "s_wqm_b64", + [GFX10_S_BREV_B32] =3D "s_brev_b32", + [GFX10_S_BREV_B64] =3D "s_brev_b64", + [GFX10_S_BCNT0_I32_B32] =3D "s_bcnt0_i32_b32", + [GFX10_S_BCNT0_I32_B64] =3D "s_bcnt0_i32_b64", + [GFX10_S_BCNT1_I32_B32] =3D "s_bcnt1_i32_b32", + [GFX10_S_BCNT1_I32_B64] =3D "s_bcnt1_i32_b64", + [GFX10_S_FF0_I32_B32] =3D "s_ff0_i32_b32", + [GFX10_S_FF0_I32_B64] =3D "s_ff0_i32_b64", + [GFX10_S_FF1_I32_B32] =3D "s_ff1_i32_b32", + [GFX10_S_FF1_I32_B64] =3D "s_ff1_i32_b64", + [GFX10_S_FLBIT_I32_B32] =3D "s_flbit_i32_b32", + [GFX10_S_FLBIT_I32_B64] =3D "s_flbit_i32_b64", + [GFX10_S_FLBIT_I32] =3D "s_flbit_i32", + [GFX10_S_FLBIT_I32_I64] =3D "s_flbit_i32_i64", + [GFX10_S_SEXT_I32_I8] =3D "s_sext_i32_i8", + [GFX10_S_SEXT_I32_I16] =3D "s_sext_i32_i16", + [GFX10_S_BITSET0_B32] =3D "s_bitset0_b32", + [GFX10_S_BITSET0_B64] =3D "s_bitset0_b64", + [GFX10_S_BITSET1_B32] =3D "s_bitset1_b32", + [GFX10_S_BITSET1_B64] =3D "s_bitset1_b64", + [GFX10_S_GETPC_B64] =3D "s_getpc_b64", + [GFX10_S_SETPC_B64] =3D "s_setpc_b64", + [GFX10_S_SWAPPC_B64] =3D "s_swappc_b64", + [GFX10_S_RFE_B64] =3D "s_rfe_b64", + [GFX10_S_AND_SAVEEXEC_B64] =3D "s_and_saveexec_b64", + [GFX10_S_XNOR_SAVEEXEC_B64] =3D "s_xnor_saveexec_b64", + [GFX10_S_QUADMASK_B32] =3D "s_quadmask_b32", + [GFX10_S_QUADMASK_B64] =3D "s_quadmask_b64", + [GFX10_S_MOVRELS_B32] =3D "s_movrels_b32", + [GFX10_S_MOVRELS_B64] =3D "s_movrels_b64", + [GFX10_S_MOVRELD_B32] =3D "s_movreld_b32", + [GFX10_S_MOVRELD_B64] =3D "s_movreld_b64", + [GFX10_S_ABS_I32] =3D "s_abs_i32", + [GFX10_S_ANDN1_SAVEEXEC_B64] =3D "s_andn1_saveexec_b64", + [GFX10_S_ORN1_SAVEEXEC_B64] =3D "s_orn1_saveexec_b64", + [GFX10_S_ANDN1_WREXEC_B64] =3D "s_andn1_wrexec_b64", + [GFX10_S_ANDN2_WREXEC_B64] =3D "s_andn2_wrexec_b64", + [GFX10_S_BITREPLICATE_B64_B32] =3D "s_bitreplicate_b64_b32", + [GFX10_S_AND_SAVEEXEC_B32] =3D "s_and_saveexec_b32", + [GFX10_S_OR_SAVEEXEC_B32] =3D "s_or_saveexec_b32", + [GFX10_S_XOR_SAVEEXEC_B32] =3D "s_xor_saveexec_b32", + [GFX10_S_ANDN2_SAVEEXEC_B32] =3D "s_andn2_saveexec_b32", + [GFX10_S_ORN2_SAVEEXEC_B32] =3D "s_orn2_saveexec_b32", + [GFX10_S_NAND_SAVEEXEC_B32] =3D "s_nand_saveexec_b32", + [GFX10_S_NOR_SAVEEXEC_B32] =3D "s_nor_saveexec_b32", + [GFX10_S_XNOR_SAVEEXEC_B32] =3D "s_xnor_saveexec_b32", + [GFX10_S_ANDN1_SAVEEXEC_B32] =3D "s_andn1_saveexec_b32", + [GFX10_S_ORN1_SAVEEXEC_B32] =3D "s_orn1_saveexec_b32", + [GFX10_S_ANDN1_WREXEC_B32] =3D "s_andn1_wrexec_b32", + [GFX10_S_ANDN2_WREXEC_B32] =3D "s_andn2_wrexec_b32", + [GFX10_S_MOVRELSD_2_B32] =3D "s_movrelsd_2_b32", + }, + [AMDGCN_INSN_TYPE_SOPC] =3D { + [GFX10_S_CMP_EQ_I32] =3D "s_cmp_eq_i32", + [GFX10_S_CMP_LG_I32] =3D "s_cmp_lg_i32", + [GFX10_S_CMP_GT_I32] =3D "s_cmp_gt_i32", + [GFX10_S_CMP_GE_I32] =3D "s_cmp_ge_i32", + [GFX10_S_CMP_LT_I32] =3D "s_cmp_lt_i32", + [GFX10_S_CMP_LE_I32] =3D "s_cmp_le_i32", + [GFX10_S_CMP_EQ_U32] =3D "s_cmp_eq_u32", + [GFX10_S_CMP_LG_U32] =3D "s_cmp_lg_u32", + [GFX10_S_CMP_GT_U32] =3D "s_cmp_gt_u32", + [GFX10_S_CMP_GE_U32] =3D "s_cmp_ge_u32", + [GFX10_S_CMP_LT_U32] =3D "s_cmp_lt_u32", + [GFX10_S_CMP_LE_U32] =3D "s_cmp_le_u32", + [GFX10_S_BITCMP0_B32] =3D "s_bitcmp0_b32", + [GFX10_S_BITCMP1_B32] =3D "s_bitcmp1_b32", + [GFX10_S_BITCMP0_B64] =3D "s_bitcmp0_b64", + [GFX10_S_BITCMP1_B64] =3D "s_bitcmp1_b64", + [GFX10_S_CMP_EQ_U64] =3D "s_cmp_eq_u64", + [GFX10_S_CMP_LG_U64] =3D "s_cmp_lg_u64", + }, + [AMDGCN_INSN_TYPE_SOPP] =3D { + [GFX10_S_NOP] =3D "s_nop", + [GFX10_S_ENDPGM] =3D "s_endpgm", + [GFX10_S_BRANCH] =3D "s_branch", + [GFX10_S_WAKEUP] =3D "s_wakeup", + [GFX10_S_CBRANCH_SCC0] =3D "s_cbranch_scc0", + [GFX10_S_CBRANCH_SCC1] =3D "s_cbranch_scc1", + [GFX10_S_CBRANCH_VCCZ] =3D "s_cbranch_vccz", + [GFX10_S_CBRANCH_VCCNZ] =3D "s_cbranch_vccnz", + [GFX10_S_CBRANCH_EXECZ] =3D "s_cbranch_execz", + [GFX10_S_CBRANCH_EXECNZ] =3D "s_cbranch_execnz", + [GFX10_S_BARRIER] =3D "s_barrier", + [GFX10_S_SETKILL] =3D "s_setkill", + [GFX10_S_WAITCNT] =3D "s_waitcnt", + [GFX10_S_SETHALT] =3D "s_sethalt", + [GFX10_S_SLEEP] =3D "s_sleep", + [GFX10_S_SETPRIO] =3D "s_setprio", + [GFX10_S_SENDMSG] =3D "s_sendmsg", + [GFX10_S_SENDMSGHALT] =3D "s_sendmsghalt", + [GFX10_S_TRAP] =3D "s_trap", + [GFX10_S_ICACHE_INV] =3D "s_icache_inv", + [GFX10_S_INCPERFLEVEL] =3D "s_incperflevel", + [GFX10_S_DECPERFLEVEL] =3D "s_decperflevel", + [GFX10_S_TTRACEDATA] =3D "s_ttracedata", + [GFX10_S_CBRANCH_CDBGSYS] =3D "s_cbranch_cdbgsys", + [GFX10_S_CBRANCH_CDBGUSER] =3D "s_cbranch_cdbguser", + [GFX10_S_CBRANCH_CDBGSYS_OR_USER] =3D "s_cbranch_cdbgsys_or_user", + [GFX10_S_CBRANCH_CDBGSYS_AND_USER] =3D "s_cbranch_cdbgsys_and_user", + [GFX10_S_ENDPGM_SAVED] =3D "s_endpgm_saved", + [GFX10_S_ENDPGM_ORDERED_PS_DONE] =3D "s_endpgm_ordered_ps_done", + [GFX10_S_CODE_END] =3D "s_code_end", + [GFX10_S_INST_PREFETCH] =3D "s_inst_prefetch", + [GFX10_S_CLAUSE] =3D "s_clause", + [GFX10_S_WAITCNT_DEPCTR] =3D "s_waitcnt_depctr", + [GFX10_S_ROUND_MODE] =3D "s_round_mode", + [GFX10_S_DENORM_MODE] =3D "s_denorm_mode", + [GFX10_S_TTRACEDATA_IMM] =3D "s_ttracedata_imm", + }, + [AMDGCN_INSN_TYPE_SMEM] =3D { + [GFX10_S_LOAD_DWORD] =3D "s_load_dword", + [GFX10_S_LOAD_DWORDX2] =3D "s_load_dwordx2", + [GFX10_S_LOAD_DWORDX4] =3D "s_load_dwordx4", + [GFX10_S_LOAD_DWORDX8] =3D "s_load_dwordx8", + [GFX10_S_LOAD_DWORDX16] =3D "s_load_dwordx16", + [GFX10_S_BUFFER_LOAD_DWORD] =3D "s_buffer_load_dword", + [GFX10_S_BUFFER_LOAD_DWORDX2] =3D "s_buffer_load_dwordx2", + [GFX10_S_BUFFER_LOAD_DWORDX4] =3D "s_buffer_load_dwordx4", + [GFX10_S_BUFFER_LOAD_DWORDX8] =3D "s_buffer_load_dwordx8", + [GFX10_S_BUFFER_LOAD_DWORDX16] =3D "s_buffer_load_dwordx16", + [GFX10_S_GL1_INV] =3D "s_gl1_inv", + [GFX10_S_DCACHE_INV] =3D "s_dcache_inv", + [GFX10_S_MEMTIME] =3D "s_memtime", + [GFX10_S_MEMREALTIME] =3D "s_memrealtime", + [GFX10_S_ATC_PROBE] =3D "s_atc_probe", + [GFX10_S_ATC_PROBE_BUFFER] =3D "s_atc_probe_buffer", + }, + [AMDGCN_INSN_TYPE_VOP2] =3D { + [GFX10_V_CNDMASK_B32] =3D "v_cndmask_b32", + [GFX10_V_DOT2C_F32_F16] =3D "v_dot2c_f32_f16", + [GFX10_V_ADD_F32] =3D "v_add_f32", + [GFX10_V_SUB_F32] =3D "v_sub_f32", + [GFX10_V_SUBREV_F32] =3D "v_subrev_f32", + [GFX10_V_FMAC_LEGACY_F32] =3D "v_fmac_legacy_f32", + [GFX10_V_MUL_LEGACY_F32] =3D "v_mul_legacy_f32", + [GFX10_V_MUL_F32] =3D "v_mul_f32", + [GFX10_V_MUL_I32_I24] =3D "v_mul_i32_i24", + [GFX10_V_MUL_HI_I32_I24] =3D "v_mul_hi_i32_i24", + [GFX10_V_MUL_U32_U24] =3D "v_mul_u32_u24", + [GFX10_V_MUL_HI_U32_U24] =3D "v_mul_hi_u32_u24", + [GFX10_V_DOT4C_I32_I8] =3D "v_dot4c_i32_i8", + [GFX10_V_MIN_F32] =3D "v_min_f32", + [GFX10_V_MAX_F32] =3D "v_max_f32", + [GFX10_V_MIN_I32] =3D "v_min_i32", + [GFX10_V_MAX_I32] =3D "v_max_i32", + [GFX10_V_MIN_U32] =3D "v_min_u32", + [GFX10_V_MAX_U32] =3D "v_max_u32", + [GFX10_V_LSHRREV_B32] =3D "v_lshrrev_b32", + [GFX10_V_ASHRREV_I32] =3D "v_ashrrev_i32", + [GFX10_V_LSHLREV_B32] =3D "v_lshlrev_b32", + [GFX10_V_AND_B32] =3D "v_and_b32", + [GFX10_V_OR_B32] =3D "v_or_b32", + [GFX10_V_XOR_B32] =3D "v_xor_b32", + [GFX10_V_XNOR_B32] =3D "v_xnor_b32", + [GFX10_V_ADD_NC_U32] =3D "v_add_nc_u32", + [GFX10_V_SUB_NC_U32] =3D "v_sub_nc_u32", + [GFX10_V_SUBREV_NC_U32] =3D "v_subrev_nc_u32", + [GFX10_V_ADD_CO_CI_U32] =3D "v_add_co_ci_u32", + [GFX10_V_SUB_CO_CI_U32] =3D "v_sub_co_ci_u32", + [GFX10_V_SUBREV_CO_CI_U32] =3D "v_subrev_co_ci_u32", + [GFX10_V_FMAC_F32] =3D "v_fmac_f32", + [GFX10_V_FMAMK_F32] =3D "v_fmamk_f32", + [GFX10_V_FMAAK_F32] =3D "v_fmaak_f32", + [GFX10_V_CVT_PKRTZ_F16_F32] =3D "v_cvt_pkrtz_f16_f32", + [GFX10_V_ADD_F16] =3D "v_add_f16", + [GFX10_V_SUB_F16] =3D "v_sub_f16", + [GFX10_V_SUBREV_F16] =3D "v_subrev_f16", + [GFX10_V_MUL_F16] =3D "v_mul_f16", + [GFX10_V_FMAC_F16] =3D "v_fmac_f16", + [GFX10_V_FMAMK_F16] =3D "v_fmamk_f16", + [GFX10_V_FMAAK_F16] =3D "v_fmaak_f16", + [GFX10_V_MAX_F16] =3D "v_max_f16", + [GFX10_V_MIN_F16] =3D "v_min_f16", + [GFX10_V_LDEXP_F16] =3D "v_ldexp_f16", + [GFX10_V_PK_FMAC_F16] =3D "v_pk_fmac_f16", + }, + [AMDGCN_INSN_TYPE_VOP1] =3D { + [GFX10_V_NOP] =3D "v_nop", + [GFX10_V_MOV_B32] =3D "v_mov_b32", + [GFX10_V_READFIRSTLANE_B32] =3D "v_readfirstlane_b32", + [GFX10_V_CVT_I32_F64] =3D "v_cvt_i32_f64", + [GFX10_V_CVT_F64_I32] =3D "v_cvt_f64_i32", + [GFX10_V_CVT_F32_I32] =3D "v_cvt_f32_i32", + [GFX10_V_CVT_F32_U32] =3D "v_cvt_f32_u32", + [GFX10_V_CVT_U32_F32] =3D "v_cvt_u32_f32", + [GFX10_V_CVT_I32_F32] =3D "v_cvt_i32_f32", + [GFX10_V_CVT_F16_F32] =3D "v_cvt_f16_f32", + [GFX10_V_CVT_F32_F16] =3D "v_cvt_f32_f16", + [GFX10_V_CVT_RPI_I32_F32] =3D "v_cvt_rpi_i32_f32", + [GFX10_V_CVT_FLR_I32_F32] =3D "v_cvt_flr_i32_f32", + [GFX10_V_CVT_OFF_F32_I4] =3D "v_cvt_off_f32_i4", + [GFX10_V_CVT_F32_F64] =3D "v_cvt_f32_f64", + [GFX10_V_CVT_F64_F32] =3D "v_cvt_f64_f32", + [GFX10_V_CVT_F32_UBYTE0] =3D "v_cvt_f32_ubyte0", + [GFX10_V_CVT_F32_UBYTE1] =3D "v_cvt_f32_ubyte1", + [GFX10_V_CVT_F32_UBYTE2] =3D "v_cvt_f32_ubyte2", + [GFX10_V_CVT_F32_UBYTE3] =3D "v_cvt_f32_ubyte3", + [GFX10_V_CVT_U32_F64] =3D "v_cvt_u32_f64", + [GFX10_V_CVT_F64_U32] =3D "v_cvt_f64_u32", + [GFX10_V_TRUNC_F64] =3D "v_trunc_f64", + [GFX10_V_CEIL_F64] =3D "v_ceil_f64", + [GFX10_V_RNDNE_F64] =3D "v_rndne_f64", + [GFX10_V_FLOOR_F64] =3D "v_floor_f64", + [GFX10_V_PIPEFLUSH] =3D "v_pipeflush", + [GFX10_V_FRACT_F32] =3D "v_fract_f32", + [GFX10_V_TRUNC_F32] =3D "v_trunc_f32", + [GFX10_V_CEIL_F32] =3D "v_ceil_f32", + [GFX10_V_RNDNE_F32] =3D "v_rndne_f32", + [GFX10_V_FLOOR_F32] =3D "v_floor_f32", + [GFX10_V_EXP_F32] =3D "v_exp_f32", + [GFX10_V_LOG_F32] =3D "v_log_f32", + [GFX10_V_RCP_F32] =3D "v_rcp_f32", + [GFX10_V_RCP_IFLAG_F32] =3D "v_rcp_iflag_f32", + [GFX10_V_RSQ_F32] =3D "v_rsq_f32", + [GFX10_V_RCP_F64] =3D "v_rcp_f64", + [GFX10_V_RSQ_F64] =3D "v_rsq_f64", + [GFX10_V_SQRT_F32] =3D "v_sqrt_f32", + [GFX10_V_SQRT_F64] =3D "v_sqrt_f64", + [GFX10_V_SIN_F32] =3D "v_sin_f32", + [GFX10_V_COS_F32] =3D "v_cos_f32", + [GFX10_V_NOT_B32] =3D "v_not_b32", + [GFX10_V_BFREV_B32] =3D "v_bfrev_b32", + [GFX10_V_FFBH_U32] =3D "v_ffbh_u32", + [GFX10_V_FFBL_B32] =3D "v_ffbl_b32", + [GFX10_V_FFBH_I32] =3D "v_ffbh_i32", + [GFX10_V_FREXP_EXP_I32_F64] =3D "v_frexp_exp_i32_f64", + [GFX10_V_FREXP_MANT_F64] =3D "v_frexp_mant_f64", + [GFX10_V_FRACT_F64] =3D "v_fract_f64", + [GFX10_V_FREXP_EXP_I32_F32] =3D "v_frexp_exp_i32_f32", + [GFX10_V_FREXP_MANT_F32] =3D "v_frexp_mant_f32", + [GFX10_V_CLREXCP] =3D "v_clrexcp", + [GFX10_V_MOVRELD_B32] =3D "v_movreld_b32", + [GFX10_V_MOVRELS_B32] =3D "v_movrels_b32", + [GFX10_V_MOVRELSD_B32] =3D "v_movrelsd_b32", + [GFX10_V_MOVRELSD_2_B32] =3D "v_movrelsd_2_b32", + [GFX10_V_CVT_F16_U16] =3D "v_cvt_f16_u16", + [GFX10_V_CVT_F16_I16] =3D "v_cvt_f16_i16", + [GFX10_V_CVT_U16_F16] =3D "v_cvt_u16_f16", + [GFX10_V_CVT_I16_F16] =3D "v_cvt_i16_f16", + [GFX10_V_RCP_F16] =3D "v_rcp_f16", + [GFX10_V_SQRT_F16] =3D "v_sqrt_f16", + [GFX10_V_RSQ_F16] =3D "v_rsq_f16", + [GFX10_V_LOG_F16] =3D "v_log_f16", + [GFX10_V_EXP_F16] =3D "v_exp_f16", + [GFX10_V_FREXP_MANT_F16] =3D "v_frexp_mant_f16", + [GFX10_V_FREXP_EXP_I16_F16] =3D "v_frexp_exp_i16_f16", + [GFX10_V_FLOOR_F16] =3D "v_floor_f16", + [GFX10_V_CEIL_F16] =3D "v_ceil_f16", + [GFX10_V_TRUNC_F16] =3D "v_trunc_f16", + [GFX10_V_RNDNE_F16] =3D "v_rndne_f16", + [GFX10_V_FRACT_F16] =3D "v_fract_f16", + [GFX10_V_SIN_F16] =3D "v_sin_f16", + [GFX10_V_COS_F16] =3D "v_cos_f16", + [GFX10_V_SAT_PK_U8_I16] =3D "v_sat_pk_u8_i16", + [GFX10_V_CVT_NORM_I16_F16] =3D "v_cvt_norm_i16_f16", + [GFX10_V_CVT_NORM_U16_F16] =3D "v_cvt_norm_u16_f16", + [GFX10_V_SWAP_B32] =3D "v_swap_b32", + [GFX10_V_SWAPREL_B32] =3D "v_swaprel_b32", + }, + [AMDGCN_INSN_TYPE_VOPC] =3D { + [GFX10_V_CMP_F_F32] =3D "v_cmp_f_f32", + [GFX10_V_CMP_LT_F32] =3D "v_cmp_lt_f32", + [GFX10_V_CMP_EQ_F32] =3D "v_cmp_eq_f32", + [GFX10_V_CMP_LE_F32] =3D "v_cmp_le_f32", + [GFX10_V_CMP_GT_F32] =3D "v_cmp_gt_f32", + [GFX10_V_CMP_LG_F32] =3D "v_cmp_lg_f32", + [GFX10_V_CMP_GE_F32] =3D "v_cmp_ge_f32", + [GFX10_V_CMP_O_F32] =3D "v_cmp_o_f32", + [GFX10_V_CMP_U_F32] =3D "v_cmp_u_f32", + [GFX10_V_CMP_NGE_F32] =3D "v_cmp_nge_f32", + [GFX10_V_CMP_NLG_F32] =3D "v_cmp_nlg_f32", + [GFX10_V_CMP_NGT_F32] =3D "v_cmp_ngt_f32", + [GFX10_V_CMP_NLE_F32] =3D "v_cmp_nle_f32", + [GFX10_V_CMP_NEQ_F32] =3D "v_cmp_neq_f32", + [GFX10_V_CMP_NLT_F32] =3D "v_cmp_nlt_f32", + [GFX10_V_CMP_TRU_F32] =3D "v_cmp_tru_f32", + [GFX10_V_CMPX_F_F32] =3D "v_cmpx_f_f32", + [GFX10_V_CMPX_LT_F32] =3D "v_cmpx_lt_f32", + [GFX10_V_CMPX_EQ_F32] =3D "v_cmpx_eq_f32", + [GFX10_V_CMPX_LE_F32] =3D "v_cmpx_le_f32", + [GFX10_V_CMPX_GT_F32] =3D "v_cmpx_gt_f32", + [GFX10_V_CMPX_LG_F32] =3D "v_cmpx_lg_f32", + [GFX10_V_CMPX_GE_F32] =3D "v_cmpx_ge_f32", + [GFX10_V_CMPX_O_F32] =3D "v_cmpx_o_f32", + [GFX10_V_CMPX_U_F32] =3D "v_cmpx_u_f32", + [GFX10_V_CMPX_NGE_F32] =3D "v_cmpx_nge_f32", + [GFX10_V_CMPX_NLG_F32] =3D "v_cmpx_nlg_f32", + [GFX10_V_CMPX_NGT_F32] =3D "v_cmpx_ngt_f32", + [GFX10_V_CMPX_NLE_F32] =3D "v_cmpx_nle_f32", + [GFX10_V_CMPX_NEQ_F32] =3D "v_cmpx_neq_f32", + [GFX10_V_CMPX_NLT_F32] =3D "v_cmpx_nlt_f32", + [GFX10_V_CMPX_TRU_F32] =3D "v_cmpx_tru_f32", + [GFX10_V_CMP_F_F64] =3D "v_cmp_f_f64", + [GFX10_V_CMP_LT_F64] =3D "v_cmp_lt_f64", + [GFX10_V_CMP_EQ_F64] =3D "v_cmp_eq_f64", + [GFX10_V_CMP_LE_F64] =3D "v_cmp_le_f64", + [GFX10_V_CMP_GT_F64] =3D "v_cmp_gt_f64", + [GFX10_V_CMP_LG_F64] =3D "v_cmp_lg_f64", + [GFX10_V_CMP_GE_F64] =3D "v_cmp_ge_f64", + [GFX10_V_CMP_O_F64] =3D "v_cmp_o_f64", + [GFX10_V_CMP_U_F64] =3D "v_cmp_u_f64", + [GFX10_V_CMP_NGE_F64] =3D "v_cmp_nge_f64", + [GFX10_V_CMP_NLG_F64] =3D "v_cmp_nlg_f64", + [GFX10_V_CMP_NGT_F64] =3D "v_cmp_ngt_f64", + [GFX10_V_CMP_NLE_F64] =3D "v_cmp_nle_f64", + [GFX10_V_CMP_NEQ_F64] =3D "v_cmp_neq_f64", + [GFX10_V_CMP_NLT_F64] =3D "v_cmp_nlt_f64", + [GFX10_V_CMP_TRU_F64] =3D "v_cmp_tru_f64", + [GFX10_V_CMPX_F_F64] =3D "v_cmpx_f_f64", + [GFX10_V_CMPX_LT_F64] =3D "v_cmpx_lt_f64", + [GFX10_V_CMPX_EQ_F64] =3D "v_cmpx_eq_f64", + [GFX10_V_CMPX_LE_F64] =3D "v_cmpx_le_f64", + [GFX10_V_CMPX_GT_F64] =3D "v_cmpx_gt_f64", + [GFX10_V_CMPX_LG_F64] =3D "v_cmpx_lg_f64", + [GFX10_V_CMPX_GE_F64] =3D "v_cmpx_ge_f64", + [GFX10_V_CMPX_O_F64] =3D "v_cmpx_o_f64", + [GFX10_V_CMPX_U_F64] =3D "v_cmpx_u_f64", + [GFX10_V_CMPX_NGE_F64] =3D "v_cmpx_nge_f64", + [GFX10_V_CMPX_NLG_F64] =3D "v_cmpx_nlg_f64", + [GFX10_V_CMPX_NGT_F64] =3D "v_cmpx_ngt_f64", + [GFX10_V_CMPX_NLE_F64] =3D "v_cmpx_nle_f64", + [GFX10_V_CMPX_NEQ_F64] =3D "v_cmpx_neq_f64", + [GFX10_V_CMPX_NLT_F64] =3D "v_cmpx_nlt_f64", + [GFX10_V_CMPX_TRU_F64] =3D "v_cmpx_tru_f64", + [GFX10_V_CMP_F_I32] =3D "v_cmp_f_i32", + [GFX10_V_CMP_LT_I32] =3D "v_cmp_lt_i32", + [GFX10_V_CMP_EQ_I32] =3D "v_cmp_eq_i32", + [GFX10_V_CMP_LE_I32] =3D "v_cmp_le_i32", + [GFX10_V_CMP_GT_I32] =3D "v_cmp_gt_i32", + [GFX10_V_CMP_NE_I32] =3D "v_cmp_ne_i32", + [GFX10_V_CMP_GE_I32] =3D "v_cmp_ge_i32", + [GFX10_V_CMP_T_I32] =3D "v_cmp_t_i32", + [GFX10_V_CMP_CLASS_F32] =3D "v_cmp_class_f32", + [GFX10_V_CMP_LT_I16] =3D "v_cmp_lt_i16", + [GFX10_V_CMP_EQ_I16] =3D "v_cmp_eq_i16", + [GFX10_V_CMP_LE_I16] =3D "v_cmp_le_i16", + [GFX10_V_CMP_GT_I16] =3D "v_cmp_gt_i16", + [GFX10_V_CMP_NE_I16] =3D "v_cmp_ne_i16", + [GFX10_V_CMP_GE_I16] =3D "v_cmp_ge_i16", + [GFX10_V_CMP_CLASS_F16] =3D "v_cmp_class_f16", + [GFX10_V_CMPX_F_I32] =3D "v_cmpx_f_i32", + [GFX10_V_CMPX_LT_I32] =3D "v_cmpx_lt_i32", + [GFX10_V_CMPX_EQ_I32] =3D "v_cmpx_eq_i32", + [GFX10_V_CMPX_LE_I32] =3D "v_cmpx_le_i32", + [GFX10_V_CMPX_GT_I32] =3D "v_cmpx_gt_i32", + [GFX10_V_CMPX_NE_I32] =3D "v_cmpx_ne_i32", + [GFX10_V_CMPX_GE_I32] =3D "v_cmpx_ge_i32", + [GFX10_V_CMPX_T_I32] =3D "v_cmpx_t_i32", + [GFX10_V_CMPX_CLASS_F32] =3D "v_cmpx_class_f32", + [GFX10_V_CMPX_LT_I16] =3D "v_cmpx_lt_i16", + [GFX10_V_CMPX_EQ_I16] =3D "v_cmpx_eq_i16", + [GFX10_V_CMPX_LE_I16] =3D "v_cmpx_le_i16", + [GFX10_V_CMPX_GT_I16] =3D "v_cmpx_gt_i16", + [GFX10_V_CMPX_NE_I16] =3D "v_cmpx_ne_i16", + [GFX10_V_CMPX_GE_I16] =3D "v_cmpx_ge_i16", + [GFX10_V_CMPX_CLASS_F16] =3D "v_cmpx_class_f16", + [GFX10_V_CMP_F_I64] =3D "v_cmp_f_i64", + [GFX10_V_CMP_LT_I64] =3D "v_cmp_lt_i64", + [GFX10_V_CMP_EQ_I64] =3D "v_cmp_eq_i64", + [GFX10_V_CMP_LE_I64] =3D "v_cmp_le_i64", + [GFX10_V_CMP_GT_I64] =3D "v_cmp_gt_i64", + [GFX10_V_CMP_NE_I64] =3D "v_cmp_ne_i64", + [GFX10_V_CMP_GE_I64] =3D "v_cmp_ge_i64", + [GFX10_V_CMP_T_I64] =3D "v_cmp_t_i64", + [GFX10_V_CMP_CLASS_F64] =3D "v_cmp_class_f64", + [GFX10_V_CMP_LT_U16] =3D "v_cmp_lt_u16", + [GFX10_V_CMP_EQ_U16] =3D "v_cmp_eq_u16", + [GFX10_V_CMP_LE_U16] =3D "v_cmp_le_u16", + [GFX10_V_CMP_GT_U16] =3D "v_cmp_gt_u16", + [GFX10_V_CMP_NE_U16] =3D "v_cmp_ne_u16", + [GFX10_V_CMP_GE_U16] =3D "v_cmp_ge_u16", + [GFX10_V_CMPX_F_I64] =3D "v_cmpx_f_i64", + [GFX10_V_CMPX_LT_I64] =3D "v_cmpx_lt_i64", + [GFX10_V_CMPX_EQ_I64] =3D "v_cmpx_eq_i64", + [GFX10_V_CMPX_LE_I64] =3D "v_cmpx_le_i64", + [GFX10_V_CMPX_GT_I64] =3D "v_cmpx_gt_i64", + [GFX10_V_CMPX_NE_I64] =3D "v_cmpx_ne_i64", + [GFX10_V_CMPX_GE_I64] =3D "v_cmpx_ge_i64", + [GFX10_V_CMPX_T_I64] =3D "v_cmpx_t_i64", + [GFX10_V_CMPX_CLASS_F64] =3D "v_cmpx_class_f64", + [GFX10_V_CMPX_LT_U16] =3D "v_cmpx_lt_u16", + [GFX10_V_CMPX_EQ_U16] =3D "v_cmpx_eq_u16", + [GFX10_V_CMPX_LE_U16] =3D "v_cmpx_le_u16", + [GFX10_V_CMPX_GT_U16] =3D "v_cmpx_gt_u16", + [GFX10_V_CMPX_NE_U16] =3D "v_cmpx_ne_u16", + [GFX10_V_CMPX_GE_U16] =3D "v_cmpx_ge_u16", + [GFX10_V_CMP_F_U32] =3D "v_cmp_f_u32", + [GFX10_V_CMP_LT_U32] =3D "v_cmp_lt_u32", + [GFX10_V_CMP_EQ_U32] =3D "v_cmp_eq_u32", + [GFX10_V_CMP_LE_U32] =3D "v_cmp_le_u32", + [GFX10_V_CMP_GT_U32] =3D "v_cmp_gt_u32", + [GFX10_V_CMP_NE_U32] =3D "v_cmp_ne_u32", + [GFX10_V_CMP_GE_U32] =3D "v_cmp_ge_u32", + [GFX10_V_CMP_T_U32] =3D "v_cmp_t_u32", + [GFX10_V_CMP_F_F16] =3D "v_cmp_f_f16", + [GFX10_V_CMP_LT_F16] =3D "v_cmp_lt_f16", + [GFX10_V_CMP_EQ_F16] =3D "v_cmp_eq_f16", + [GFX10_V_CMP_LE_F16] =3D "v_cmp_le_f16", + [GFX10_V_CMP_GT_F16] =3D "v_cmp_gt_f16", + [GFX10_V_CMP_LG_F16] =3D "v_cmp_lg_f16", + [GFX10_V_CMP_GE_F16] =3D "v_cmp_ge_f16", + [GFX10_V_CMP_O_F16] =3D "v_cmp_o_f16", + [GFX10_V_CMPX_F_U32] =3D "v_cmpx_f_u32", + [GFX10_V_CMPX_LT_U32] =3D "v_cmpx_lt_u32", + [GFX10_V_CMPX_EQ_U32] =3D "v_cmpx_eq_u32", + [GFX10_V_CMPX_LE_U32] =3D "v_cmpx_le_u32", + [GFX10_V_CMPX_GT_U32] =3D "v_cmpx_gt_u32", + [GFX10_V_CMPX_NE_U32] =3D "v_cmpx_ne_u32", + [GFX10_V_CMPX_GE_U32] =3D "v_cmpx_ge_u32", + [GFX10_V_CMPX_T_U32] =3D "v_cmpx_t_u32", + [GFX10_V_CMPX_F_F16] =3D "v_cmpx_f_f16", + [GFX10_V_CMPX_LT_F16] =3D "v_cmpx_lt_f16", + [GFX10_V_CMPX_EQ_F16] =3D "v_cmpx_eq_f16", + [GFX10_V_CMPX_LE_F16] =3D "v_cmpx_le_f16", + [GFX10_V_CMPX_GT_F16] =3D "v_cmpx_gt_f16", + [GFX10_V_CMPX_LG_F16] =3D "v_cmpx_lg_f16", + [GFX10_V_CMPX_GE_F16] =3D "v_cmpx_ge_f16", + [GFX10_V_CMPX_O_F16] =3D "v_cmpx_o_f16", + [GFX10_V_CMP_F_U64] =3D "v_cmp_f_u64", + [GFX10_V_CMP_LT_U64] =3D "v_cmp_lt_u64", + [GFX10_V_CMP_EQ_U64] =3D "v_cmp_eq_u64", + [GFX10_V_CMP_LE_U64] =3D "v_cmp_le_u64", + [GFX10_V_CMP_GT_U64] =3D "v_cmp_gt_u64", + [GFX10_V_CMP_NE_U64] =3D "v_cmp_ne_u64", + [GFX10_V_CMP_GE_U64] =3D "v_cmp_ge_u64", + [GFX10_V_CMP_T_U64] =3D "v_cmp_t_u64", + [GFX10_V_CMP_U_F16] =3D "v_cmp_u_f16", + [GFX10_V_CMP_NGE_F16] =3D "v_cmp_nge_f16", + [GFX10_V_CMP_NLG_F16] =3D "v_cmp_nlg_f16", + [GFX10_V_CMP_NGT_F16] =3D "v_cmp_ngt_f16", + [GFX10_V_CMP_NLE_F16] =3D "v_cmp_nle_f16", + [GFX10_V_CMP_NEQ_F16] =3D "v_cmp_neq_f16", + [GFX10_V_CMP_NLT_F16] =3D "v_cmp_nlt_f16", + [GFX10_V_CMP_TRU_F16] =3D "v_cmp_tru_f16", + [GFX10_V_CMPX_F_U64] =3D "v_cmpx_f_u64", + [GFX10_V_CMPX_LT_U64] =3D "v_cmpx_lt_u64", + [GFX10_V_CMPX_EQ_U64] =3D "v_cmpx_eq_u64", + [GFX10_V_CMPX_LE_U64] =3D "v_cmpx_le_u64", + [GFX10_V_CMPX_GT_U64] =3D "v_cmpx_gt_u64", + [GFX10_V_CMPX_NE_U64] =3D "v_cmpx_ne_u64", + [GFX10_V_CMPX_GE_U64] =3D "v_cmpx_ge_u64", + [GFX10_V_CMPX_T_U64] =3D "v_cmpx_t_u64", + [GFX10_V_CMPX_U_F16] =3D "v_cmpx_u_f16", + [GFX10_V_CMPX_NGE_F16] =3D "v_cmpx_nge_f16", + [GFX10_V_CMPX_NLG_F16] =3D "v_cmpx_nlg_f16", + [GFX10_V_CMPX_NGT_F16] =3D "v_cmpx_ngt_f16", + [GFX10_V_CMPX_NLE_F16] =3D "v_cmpx_nle_f16", + [GFX10_V_CMPX_NEQ_F16] =3D "v_cmpx_neq_f16", + [GFX10_V_CMPX_NLT_F16] =3D "v_cmpx_nlt_f16", + [GFX10_V_CMPX_TRU_F16] =3D "v_cmpx_tru_f16", + }, + [AMDGCN_INSN_TYPE_VOP3B] =3D { + [GFX10_V_DIV_SCALE_F32] =3D "v_div_scale_f32", + [GFX10_V_DIV_SCALE_F64] =3D "v_div_scale_f64", + [GFX10_V_MAD_U64_U32] =3D "v_mad_u64_u32", + [GFX10_V_MAD_I64_I32] =3D "v_mad_i64_i32", + [GFX10_V_ADD_CO_U32] =3D "v_add_co_u32", + [GFX10_V_SUB_CO_U32] =3D "v_sub_co_u32", + [GFX10_V_SUBREV_CO_U32] =3D "v_subrev_co_u32", + }, + [AMDGCN_INSN_TYPE_VOP3P] =3D { + [GFX10_V_PK_MAD_I16] =3D "v_pk_mad_i16", + [GFX10_V_PK_MUL_LO_U16] =3D "v_pk_mul_lo_u16", + [GFX10_V_PK_ADD_I16] =3D "v_pk_add_i16", + [GFX10_V_PK_SUB_I16] =3D "v_pk_sub_i16", + [GFX10_V_PK_LSHLREV_B16] =3D "v_pk_lshlrev_b16", + [GFX10_V_PK_LSHRREV_B16] =3D "v_pk_lshrrev_b16", + [GFX10_V_PK_ASHRREV_I16] =3D "v_pk_ashrrev_i16", + [GFX10_V_PK_MAX_I16] =3D "v_pk_max_i16", + [GFX10_V_PK_MIN_I16] =3D "v_pk_min_i16", + [GFX10_V_PK_MAD_U16] =3D "v_pk_mad_u16", + [GFX10_V_PK_ADD_U16] =3D "v_pk_add_u16", + [GFX10_V_PK_SUB_U16] =3D "v_pk_sub_u16", + [GFX10_V_PK_MAX_U16] =3D "v_pk_max_u16", + [GFX10_V_PK_MIN_U16] =3D "v_pk_min_u16", + [GFX10_V_PK_FMA_F16] =3D "v_pk_fma_f16", + [GFX10_V_PK_ADD_F16] =3D "v_pk_add_f16", + [GFX10_V_PK_MUL_F16] =3D "v_pk_mul_f16", + [GFX10_V_PK_MIN_F16] =3D "v_pk_min_f16", + [GFX10_V_PK_MAX_F16] =3D "v_pk_max_f16", + [GFX10_V_DOT2_F32_F16] =3D "v_dot2_f32_f16", + [GFX10_V_DOT2_I32_I16] =3D "v_dot2_i32_i16", + [GFX10_V_DOT2_U32_U16] =3D "v_dot2_u32_u16", + [GFX10_V_DOT4_I32_I8] =3D "v_dot4_i32_i8", + [GFX10_V_DOT4_U32_U8] =3D "v_dot4_u32_u8", + [GFX10_V_DOT8_I32_I4] =3D "v_dot8_i32_i4", + [GFX10_V_DOT8_U32_U4] =3D "v_dot8_u32_u4", + [GFX10_V_FMA_MIX_F32] =3D "v_fma_mix_f32", + [GFX10_V_FMA_MIXLO_F16] =3D "v_fma_mixlo_f16", + [GFX10_V_FMA_MIXHI_F16] =3D "v_fma_mixhi_f16", + }, + [AMDGCN_INSN_TYPE_MUBUF] =3D { + [GFX10_BUFFER_LOAD_FORMAT_X] =3D "buffer_load_format_x", + [GFX10_BUFFER_LOAD_FORMAT_XY] =3D "buffer_load_format_xy", + [GFX10_BUFFER_LOAD_FORMAT_XYZ] =3D "buffer_load_format_xyz", + [GFX10_BUFFER_LOAD_FORMAT_XYZW] =3D "buffer_load_format_xyzw", + [GFX10_BUFFER_STORE_FORMAT_X] =3D "buffer_store_format_x", + [GFX10_BUFFER_STORE_FORMAT_XY] =3D "buffer_store_format_xy", + [GFX10_BUFFER_STORE_FORMAT_XYZ] =3D "buffer_store_format_xyz", + [GFX10_BUFFER_STORE_FORMAT_XYZW] =3D "buffer_store_format_xyzw", + [GFX10_BUFFER_LOAD_UBYTE] =3D "buffer_load_ubyte", + [GFX10_BUFFER_LOAD_SBYTE] =3D "buffer_load_sbyte", + [GFX10_BUFFER_LOAD_USHORT] =3D "buffer_load_ushort", + [GFX10_BUFFER_LOAD_SSHORT] =3D "buffer_load_sshort", + [GFX10_BUFFER_LOAD_DWORD] =3D "buffer_load_dword", + [GFX10_BUFFER_LOAD_DWORDX2] =3D "buffer_load_dwordx2", + [GFX10_BUFFER_LOAD_DWORDX4] =3D "buffer_load_dwordx4", + [GFX10_BUFFER_LOAD_DWORDX3] =3D "buffer_load_dwordx3", + [GFX10_BUFFER_STORE_BYTE] =3D "buffer_store_byte", + [GFX10_BUFFER_STORE_BYTE_D16_HI] =3D "buffer_store_byte_d16_hi", + [GFX10_BUFFER_STORE_SHORT] =3D "buffer_store_short", + [GFX10_BUFFER_STORE_SHORT_D16_HI] =3D "buffer_store_short_d16_hi", + [GFX10_BUFFER_STORE_DWORD] =3D "buffer_store_dword", + [GFX10_BUFFER_STORE_DWORDX2] =3D "buffer_store_dwordx2", + [GFX10_BUFFER_STORE_DWORDX4] =3D "buffer_store_dwordx4", + [GFX10_BUFFER_STORE_DWORDX3] =3D "buffer_store_dwordx3", + [GFX10_BUFFER_LOAD_UBYTE_D16] =3D "buffer_load_ubyte_d16", + [GFX10_BUFFER_LOAD_UBYTE_D16_HI] =3D "buffer_load_ubyte_d16_hi", + [GFX10_BUFFER_LOAD_SBYTE_D16] =3D "buffer_load_sbyte_d16", + [GFX10_BUFFER_LOAD_SBYTE_D16_HI] =3D "buffer_load_sbyte_d16_hi", + [GFX10_BUFFER_LOAD_SHORT_D16] =3D "buffer_load_short_d16", + [GFX10_BUFFER_LOAD_SHORT_D16_HI] =3D "buffer_load_short_d16_hi", + [GFX10_BUFFER_LOAD_FORMAT_D16_HI_X] =3D "buffer_load_format_d16_hi_x", + [GFX10_BUFFER_STORE_FORMAT_D16_HI_X] =3D "buffer_store_format_d16_hi_x", + [GFX10_BUFFER_ATOMIC_SWAP] =3D "buffer_atomic_swap", + [GFX10_BUFFER_ATOMIC_CMPSWAP] =3D "buffer_atomic_cmpswap", + [GFX10_BUFFER_ATOMIC_ADD] =3D "buffer_atomic_add", + [GFX10_BUFFER_ATOMIC_SUB] =3D "buffer_atomic_sub", + [GFX10_BUFFER_ATOMIC_CSUB] =3D "buffer_atomic_csub", + [GFX10_BUFFER_ATOMIC_SMIN] =3D "buffer_atomic_smin", + [GFX10_BUFFER_ATOMIC_UMIN] =3D "buffer_atomic_umin", + [GFX10_BUFFER_ATOMIC_SMAX] =3D "buffer_atomic_smax", + [GFX10_BUFFER_ATOMIC_UMAX] =3D "buffer_atomic_umax", + [GFX10_BUFFER_ATOMIC_AND] =3D "buffer_atomic_and", + [GFX10_BUFFER_ATOMIC_OR] =3D "buffer_atomic_or", + [GFX10_BUFFER_ATOMIC_XOR] =3D "buffer_atomic_xor", + [GFX10_BUFFER_ATOMIC_INC] =3D "buffer_atomic_inc", + [GFX10_BUFFER_ATOMIC_DEC] =3D "buffer_atomic_dec", + [GFX10_BUFFER_ATOMIC_FCMPSWAP] =3D "buffer_atomic_fcmpswap", + [GFX10_BUFFER_ATOMIC_FMIN] =3D "buffer_atomic_fmin", + [GFX10_BUFFER_ATOMIC_FMAX] =3D "buffer_atomic_fmax", + [GFX10_BUFFER_ATOMIC_SWAP_X2] =3D "buffer_atomic_swap_x2", + [GFX10_BUFFER_ATOMIC_CMPSWAP_X2] =3D "buffer_atomic_cmpswap_x2", + [GFX10_BUFFER_ATOMIC_ADD_X2] =3D "buffer_atomic_add_x2", + [GFX10_BUFFER_ATOMIC_SUB_X2] =3D "buffer_atomic_sub_x2", + [GFX10_BUFFER_ATOMIC_SMIN_X2] =3D "buffer_atomic_smin_x2", + [GFX10_BUFFER_ATOMIC_UMIN_X2] =3D "buffer_atomic_umin_x2", + [GFX10_BUFFER_ATOMIC_SMAX_X2] =3D "buffer_atomic_smax_x2", + [GFX10_BUFFER_ATOMIC_UMAX_X2] =3D "buffer_atomic_umax_x2", + [GFX10_BUFFER_ATOMIC_AND_X2] =3D "buffer_atomic_and_x2", + [GFX10_BUFFER_ATOMIC_OR_X2] =3D "buffer_atomic_or_x2", + [GFX10_BUFFER_ATOMIC_XOR_X2] =3D "buffer_atomic_xor_x2", + [GFX10_BUFFER_ATOMIC_INC_X2] =3D "buffer_atomic_inc_x2", + [GFX10_BUFFER_ATOMIC_DEC_X2] =3D "buffer_atomic_dec_x2", + [GFX10_BUFFER_ATOMIC_FCMPSWAP_X2] =3D "buffer_atomic_fcmpswap_x2", + [GFX10_BUFFER_ATOMIC_FMIN_X2] =3D "buffer_atomic_fmin_x2", + [GFX10_BUFFER_ATOMIC_FMAX_X2] =3D "buffer_atomic_fmax_x2", + [GFX10_BUFFER_GL0_INV] =3D "buffer_gl0_inv", + [GFX10_BUFFER_GL1_INV] =3D "buffer_gl1_inv", + [GFX10_BUFFER_LOAD_FORMAT_D16_X] =3D "buffer_load_format_d16_x", + [GFX10_BUFFER_LOAD_FORMAT_D16_XY] =3D "buffer_load_format_d16_xy", + [GFX10_BUFFER_LOAD_FORMAT_D16_XYZ] =3D "buffer_load_format_d16_xyz", + [GFX10_BUFFER_LOAD_FORMAT_D16_XYZW] =3D "buffer_load_format_d16_xyzw", + [GFX10_BUFFER_STORE_FORMAT_D16_X] =3D "buffer_store_format_d16_x", + [GFX10_BUFFER_STORE_FORMAT_D16_XY] =3D "buffer_store_format_d16_xy", + [GFX10_BUFFER_STORE_FORMAT_D16_XYZ] =3D "buffer_store_format_d16_xyz", + [GFX10_BUFFER_STORE_FORMAT_D16_XYZW] =3D "buffer_store_format_d16_xyzw", + }, + [AMDGCN_INSN_TYPE_FLAT] =3D { + [GFX10_GLOBAL_LOAD_UBYTE] =3D "global_load_ubyte", + [GFX10_GLOBAL_LOAD_SBYTE] =3D "global_load_sbyte", + [GFX10_GLOBAL_LOAD_USHORT] =3D "global_load_ushort", + [GFX10_GLOBAL_LOAD_SSHORT] =3D "global_load_sshort", + [GFX10_GLOBAL_LOAD_DWORD] =3D "global_load_dword", + [GFX10_GLOBAL_LOAD_DWORDX2] =3D "global_load_dwordx2", + [GFX10_GLOBAL_LOAD_DWORDX4] =3D "global_load_dwordx4", + [GFX10_GLOBAL_LOAD_DWORDX3] =3D "global_load_dwordx3", + [GFX10_GLOBAL_LOAD_DWORD_ADDTID] =3D "global_load_dword_addtid", + [GFX10_GLOBAL_STORE_DWORD_ADDTID] =3D "global_store_dword_addtid", + [GFX10_GLOBAL_STORE_BYTE] =3D "global_store_byte", + [GFX10_GLOBAL_STORE_BYTE_D16_HI] =3D "global_store_byte_d16_hi", + [GFX10_GLOBAL_STORE_SHORT] =3D "global_store_short", + [GFX10_GLOBAL_STORE_SHORT_D16_HI] =3D "global_store_short_d16_hi", + [GFX10_GLOBAL_STORE_DWORD] =3D "global_store_dword", + [GFX10_GLOBAL_STORE_DWORDX2] =3D "global_store_dwordx2", + [GFX10_GLOBAL_STORE_DWORDX4] =3D "global_store_dwordx4", + [GFX10_GLOBAL_STORE_DWORDX3] =3D "global_store_dwordx3", + [GFX10_GLOBAL_LOAD_UBYTE_D16] =3D "global_load_ubyte_d16", + [GFX10_GLOBAL_LOAD_UBYTE_D16_HI] =3D "global_load_ubyte_d16_hi", + [GFX10_GLOBAL_LOAD_SBYTE_D16] =3D "global_load_sbyte_d16", + [GFX10_GLOBAL_LOAD_SBYTE_D16_HI] =3D "global_load_sbyte_d16_hi", + [GFX10_GLOBAL_LOAD_SHORT_D16] =3D "global_load_short_d16", + [GFX10_GLOBAL_LOAD_SHORT_D16_HI] =3D "global_load_short_d16_hi", + [GFX10_GLOBAL_ATOMIC_SWAP] =3D "global_atomic_swap", + [GFX10_GLOBAL_ATOMIC_CMPSWAP] =3D "global_atomic_cmpswap", + [GFX10_GLOBAL_ATOMIC_ADD] =3D "global_atomic_add", + [GFX10_GLOBAL_ATOMIC_SUB] =3D "global_atomic_sub", + [GFX10_GLOBAL_ATOMIC_CSUB] =3D "global_atomic_csub", + [GFX10_GLOBAL_ATOMIC_SMIN] =3D "global_atomic_smin", + [GFX10_GLOBAL_ATOMIC_UMIN] =3D "global_atomic_umin", + [GFX10_GLOBAL_ATOMIC_SMAX] =3D "global_atomic_smax", + [GFX10_GLOBAL_ATOMIC_UMAX] =3D "global_atomic_umax", + [GFX10_GLOBAL_ATOMIC_AND] =3D "global_atomic_and", + [GFX10_GLOBAL_ATOMIC_OR] =3D "global_atomic_or", + [GFX10_GLOBAL_ATOMIC_XOR] =3D "global_atomic_xor", + [GFX10_GLOBAL_ATOMIC_INC] =3D "global_atomic_inc", + [GFX10_GLOBAL_ATOMIC_DEC] =3D "global_atomic_dec", + [GFX10_GLOBAL_ATOMIC_FCMPSWAP] =3D "global_atomic_fcmpswap", + [GFX10_GLOBAL_ATOMIC_FMIN] =3D "global_atomic_fmin", + [GFX10_GLOBAL_ATOMIC_FMAX] =3D "global_atomic_fmax", + [GFX10_GLOBAL_ATOMIC_SWAP_X2] =3D "global_atomic_swap_x2", + [GFX10_GLOBAL_ATOMIC_CMPSWAP_X2] =3D "global_atomic_cmpswap_x2", + [GFX10_GLOBAL_ATOMIC_ADD_X2] =3D "global_atomic_add_x2", + [GFX10_GLOBAL_ATOMIC_SUB_X2] =3D "global_atomic_sub_x2", + [GFX10_GLOBAL_ATOMIC_SMIN_X2] =3D "global_atomic_smin_x2", + [GFX10_GLOBAL_ATOMIC_UMIN_X2] =3D "global_atomic_umin_x2", + [GFX10_GLOBAL_ATOMIC_SMAX_X2] =3D "global_atomic_smax_x2", + [GFX10_GLOBAL_ATOMIC_UMAX_X2] =3D "global_atomic_umax_x2", + [GFX10_GLOBAL_ATOMIC_AND_X2] =3D "global_atomic_and_x2", + [GFX10_GLOBAL_ATOMIC_OR_X2] =3D "global_atomic_or_x2", + [GFX10_GLOBAL_ATOMIC_XOR_X2] =3D "global_atomic_xor_x2", + [GFX10_GLOBAL_ATOMIC_INC_X2] =3D "global_atomic_inc_x2", + [GFX10_GLOBAL_ATOMIC_DEC_X2] =3D "global_atomic_dec_x2", + [GFX10_GLOBAL_ATOMIC_FCMPSWAP_X2] =3D "global_atomic_fcmpswap_x2", + [GFX10_GLOBAL_ATOMIC_FMIN_X2] =3D "global_atomic_fmin_x2", + [GFX10_GLOBAL_ATOMIC_FMAX_X2] =3D "global_atomic_fmax_x2", + }, +}; + +static const char opnames_gfx9[__AMDGCN_INSN_TYPE_MAX][900][30] =3D { + [AMDGCN_INSN_TYPE_SOP2] =3D { + [GFX9_S_ADD_U32] =3D "s_add_u32", + [GFX9_S_SUB_U32] =3D "s_sub_u32", + [GFX9_S_ADD_I32] =3D "s_add_i32", + [GFX9_S_SUB_I32] =3D "s_sub_i32", + [GFX9_S_ADDC_U32] =3D "s_addc_u32", + [GFX9_S_SUBB_U32] =3D "s_subb_u32", + [GFX9_S_MIN_I32] =3D "s_min_i32", + [GFX9_S_MIN_U32] =3D "s_min_u32", + [GFX9_S_MAX_I32] =3D "s_max_i32", + [GFX9_S_MAX_U32] =3D "s_max_u32", + [GFX9_S_CSELECT_B32] =3D "s_cselect_b32", + [GFX9_S_CSELECT_B64] =3D "s_cselect_b64", + [GFX9_S_AND_B32] =3D "s_and_b32", + [GFX9_S_AND_B64] =3D "s_and_b64", + [GFX9_S_OR_B32] =3D "s_or_b32", + [GFX9_S_OR_B64] =3D "s_or_b64", + [GFX9_S_XOR_B32] =3D "s_xor_b32", + [GFX9_S_XOR_B64] =3D "s_xor_b64", + [GFX9_S_ANDN2_B32] =3D "s_andn2_b32", + [GFX9_S_ANDN2_B64] =3D "s_andn2_b64", + [GFX9_S_ORN2_B32] =3D "s_orn2_b32", + [GFX9_S_ORN2_B64] =3D "s_orn2_b64", + [GFX9_S_NAND_B32] =3D "s_nand_b32", + [GFX9_S_NAND_B64] =3D "s_nand_b64", + [GFX9_S_NOR_B32] =3D "s_nor_b32", + [GFX9_S_NOR_B64] =3D "s_nor_b64", + [GFX9_S_XNOR_B32] =3D "s_xnor_b32", + [GFX9_S_XNOR_B64] =3D "s_xnor_b64", + [GFX9_S_LSHL_B32] =3D "s_lshl_b32", + [GFX9_S_LSHL_B64] =3D "s_lshl_b64", + [GFX9_S_LSHR_B32] =3D "s_lshr_b32", + [GFX9_S_LSHR_B64] =3D "s_lshr_b64", + [GFX9_S_ASHR_I32] =3D "s_ashr_i32", + [GFX9_S_ASHR_I64] =3D "s_ashr_i64", + [GFX9_S_BFM_B32] =3D "s_bfm_b32", + [GFX9_S_BFM_B64] =3D "s_bfm_b64", + [GFX9_S_MUL_I32] =3D "s_mul_i32", + [GFX9_S_BFE_U32] =3D "s_bfe_u32", + [GFX9_S_BFE_I32] =3D "s_bfe_i32", + [GFX9_S_BFE_U64] =3D "s_bfe_u64", + [GFX9_S_BFE_I64] =3D "s_bfe_i64", + [GFX9_S_CBRANCH_G_FORK] =3D "s_cbranch_g_fork", + [GFX9_S_ABSDIFF_I32] =3D "s_absdiff_i32", + [GFX9_S_RFE_RESTORE_B64] =3D "s_rfe_restore_b64", + [GFX9_S_MUL_HI_U32] =3D "s_mul_hi_u32", + [GFX9_S_MUL_HI_I32] =3D "s_mul_hi_i32", + [GFX9_S_LSHL1_ADD_U32] =3D "s_lshl1_add_u32", + [GFX9_S_LSHL2_ADD_U32] =3D "s_lshl2_add_u32", + [GFX9_S_LSHL3_ADD_U32] =3D "s_lshl3_add_u32", + [GFX9_S_LSHL4_ADD_U32] =3D "s_lshl4_add_u32", + [GFX9_S_PACK_LL_B32_B16] =3D "s_pack_ll_b32_b16", + [GFX9_S_PACK_LH_B32_B16] =3D "s_pack_lh_b32_b16", + [GFX9_S_PACK_HH_B32_B16] =3D "s_pack_hh_b32_b16", + }, + [AMDGCN_INSN_TYPE_SOPK] =3D { + [GFX9_S_MOVK_I32] =3D "s_movk_i32", + [GFX9_S_CMOVK_I32] =3D "s_cmovk_i32", + [GFX9_S_CMPK_EQ_I32] =3D "s_cmpk_eq_i32", + [GFX9_S_CMPK_LG_I32] =3D "s_cmpk_lg_i32", + [GFX9_S_CMPK_GT_I32] =3D "s_cmpk_gt_i32", + [GFX9_S_CMPK_GE_I32] =3D "s_cmpk_ge_i32", + [GFX9_S_CMPK_LT_I32] =3D "s_cmpk_lt_i32", + [GFX9_S_CMPK_LE_I32] =3D "s_cmpk_le_i32", + [GFX9_S_CMPK_EQ_U32] =3D "s_cmpk_eq_u32", + [GFX9_S_CMPK_LG_U32] =3D "s_cmpk_lg_u32", + [GFX9_S_CMPK_GT_U32] =3D "s_cmpk_gt_u32", + [GFX9_S_CMPK_GE_U32] =3D "s_cmpk_ge_u32", + [GFX9_S_CMPK_LT_U32] =3D "s_cmpk_lt_u32", + [GFX9_S_CMPK_LE_U32] =3D "s_cmpk_le_u32", + [GFX9_S_ADDK_I32] =3D "s_addk_i32", + [GFX9_S_MULK_I32] =3D "s_mulk_i32", + [GFX9_S_CBRANCH_I_FORK] =3D "s_cbranch_i_fork", + [GFX9_S_GETREG_B32] =3D "s_getreg_b32", + [GFX9_S_SETREG_B32] =3D "s_setreg_b32", + [GFX9_S_SETREG_IMM32_B32] =3D "s_setreg_imm32_b32", + [GFX9_S_CALL_B64] =3D "s_call_b64", + }, + [AMDGCN_INSN_TYPE_SOP1] =3D { + [GFX9_S_MOV_B32] =3D "s_mov_b32", + [GFX9_S_MOV_B64] =3D "s_mov_b64", + [GFX9_S_CMOV_B32] =3D "s_cmov_b32", + [GFX9_S_CMOV_B64] =3D "s_cmov_b64", + [GFX9_S_NOT_B32] =3D "s_not_b32", + [GFX9_S_NOT_B64] =3D "s_not_b64", + [GFX9_S_WQM_B32] =3D "s_wqm_b32", + [GFX9_S_WQM_B64] =3D "s_wqm_b64", + [GFX9_S_BREV_B32] =3D "s_brev_b32", + [GFX9_S_BREV_B64] =3D "s_brev_b64", + [GFX9_S_BCNT0_I32_B32] =3D "s_bcnt0_i32_b32", + [GFX9_S_BCNT0_I32_B64] =3D "s_bcnt0_i32_b64", + [GFX9_S_BCNT1_I32_B32] =3D "s_bcnt1_i32_b32", + [GFX9_S_BCNT1_I32_B64] =3D "s_bcnt1_i32_b64", + [GFX9_S_FF0_I32_B32] =3D "s_ff0_i32_b32", + [GFX9_S_FF0_I32_B64] =3D "s_ff0_i32_b64", + [GFX9_S_FF1_I32_B32] =3D "s_ff1_i32_b32", + [GFX9_S_FF1_I32_B64] =3D "s_ff1_i32_b64", + [GFX9_S_FLBIT_I32_B32] =3D "s_flbit_i32_b32", + [GFX9_S_FLBIT_I32_B64] =3D "s_flbit_i32_b64", + [GFX9_S_FLBIT_I32] =3D "s_flbit_i32", + [GFX9_S_FLBIT_I32_I64] =3D "s_flbit_i32_i64", + [GFX9_S_SEXT_I32_I8] =3D "s_sext_i32_i8", + [GFX9_S_SEXT_I32_I16] =3D "s_sext_i32_i16", + [GFX9_S_BITSET0_B32] =3D "s_bitset0_b32", + [GFX9_S_BITSET0_B64] =3D "s_bitset0_b64", + [GFX9_S_BITSET1_B32] =3D "s_bitset1_b32", + [GFX9_S_BITSET1_B64] =3D "s_bitset1_b64", + [GFX9_S_GETPC_B64] =3D "s_getpc_b64", + [GFX9_S_SETPC_B64] =3D "s_setpc_b64", + [GFX9_S_SWAPPC_B64] =3D "s_swappc_b64", + [GFX9_S_RFE_B64] =3D "s_rfe_b64", + [GFX9_S_AND_SAVEEXEC_B64] =3D "s_and_saveexec_b64", + [GFX9_S_XNOR_SAVEEXEC_B64] =3D "s_xnor_saveexec_b64", + [GFX9_S_QUADMASK_B32] =3D "s_quadmask_b32", + [GFX9_S_QUADMASK_B64] =3D "s_quadmask_b64", + [GFX9_S_MOVRELS_B32] =3D "s_movrels_b32", + [GFX9_S_MOVRELS_B64] =3D "s_movrels_b64", + [GFX9_S_MOVRELD_B32] =3D "s_movreld_b32", + [GFX9_S_MOVRELD_B64] =3D "s_movreld_b64", + [GFX9_S_ABS_I32] =3D "s_abs_i32", + [GFX9_S_ANDN1_SAVEEXEC_B64] =3D "s_andn1_saveexec_b64", + [GFX9_S_ORN1_SAVEEXEC_B64] =3D "s_orn1_saveexec_b64", + [GFX9_S_ANDN1_WREXEC_B64] =3D "s_andn1_wrexec_b64", + [GFX9_S_ANDN2_WREXEC_B64] =3D "s_andn2_wrexec_b64", + [GFX9_S_BITREPLICATE_B64_B32] =3D "s_bitreplicate_b64_b32", + /* SOP1 opcodes missing from the table (Vega ISA 12.3). */ + [GFX9_S_OR_SAVEEXEC_B64] =3D "s_or_saveexec_b64", + [GFX9_S_XOR_SAVEEXEC_B64] =3D "s_xor_saveexec_b64", + [GFX9_S_ANDN2_SAVEEXEC_B64] =3D "s_andn2_saveexec_b64", + [GFX9_S_ORN2_SAVEEXEC_B64] =3D "s_orn2_saveexec_b64", + [GFX9_S_NAND_SAVEEXEC_B64] =3D "s_nand_saveexec_b64", + [GFX9_S_NOR_SAVEEXEC_B64] =3D "s_nor_saveexec_b64", + [GFX9_S_SET_GPR_IDX_IDX] =3D "s_set_gpr_idx_idx", + }, + [AMDGCN_INSN_TYPE_SOPC] =3D { + [GFX9_S_CMP_EQ_I32] =3D "s_cmp_eq_i32", + [GFX9_S_CMP_LG_I32] =3D "s_cmp_lg_i32", + [GFX9_S_CMP_GT_I32] =3D "s_cmp_gt_i32", + [GFX9_S_CMP_GE_I32] =3D "s_cmp_ge_i32", + [GFX9_S_CMP_LT_I32] =3D "s_cmp_lt_i32", + [GFX9_S_CMP_LE_I32] =3D "s_cmp_le_i32", + [GFX9_S_CMP_EQ_U32] =3D "s_cmp_eq_u32", + [GFX9_S_CMP_LG_U32] =3D "s_cmp_lg_u32", + [GFX9_S_CMP_GT_U32] =3D "s_cmp_gt_u32", + [GFX9_S_CMP_GE_U32] =3D "s_cmp_ge_u32", + [GFX9_S_CMP_LT_U32] =3D "s_cmp_lt_u32", + [GFX9_S_CMP_LE_U32] =3D "s_cmp_le_u32", + [GFX9_S_BITCMP0_B32] =3D "s_bitcmp0_b32", + [GFX9_S_BITCMP1_B32] =3D "s_bitcmp1_b32", + [GFX9_S_BITCMP0_B64] =3D "s_bitcmp0_b64", + [GFX9_S_BITCMP1_B64] =3D "s_bitcmp1_b64", + [GFX9_S_SETVSKIP] =3D "s_setvskip", + [GFX9_S_SET_GPR_IDX_ON] =3D "s_set_gpr_idx_on", + [GFX9_S_CMP_EQ_U64] =3D "s_cmp_eq_u64", + [GFX9_S_CMP_LG_U64] =3D "s_cmp_lg_u64", + }, + [AMDGCN_INSN_TYPE_SOPP] =3D { + [GFX9_S_NOP] =3D "s_nop", + [GFX9_S_ENDPGM] =3D "s_endpgm", + [GFX9_S_BRANCH] =3D "s_branch", + [GFX9_S_WAKEUP] =3D "s_wakeup", + [GFX9_S_CBRANCH_SCC0] =3D "s_cbranch_scc0", + [GFX9_S_CBRANCH_SCC1] =3D "s_cbranch_scc1", + [GFX9_S_CBRANCH_VCCZ] =3D "s_cbranch_vccz", + [GFX9_S_CBRANCH_VCCNZ] =3D "s_cbranch_vccnz", + [GFX9_S_CBRANCH_EXECZ] =3D "s_cbranch_execz", + [GFX9_S_CBRANCH_EXECNZ] =3D "s_cbranch_execnz", + [GFX9_S_BARRIER] =3D "s_barrier", + [GFX9_S_SETKILL] =3D "s_setkill", + [GFX9_S_WAITCNT] =3D "s_waitcnt", + [GFX9_S_SETHALT] =3D "s_sethalt", + [GFX9_S_SLEEP] =3D "s_sleep", + [GFX9_S_SETPRIO] =3D "s_setprio", + [GFX9_S_SENDMSG] =3D "s_sendmsg", + [GFX9_S_SENDMSGHALT] =3D "s_sendmsghalt", + [GFX9_S_TRAP] =3D "s_trap", + [GFX9_S_ICACHE_INV] =3D "s_icache_inv", + [GFX9_S_INCPERFLEVEL] =3D "s_incperflevel", + [GFX9_S_DECPERFLEVEL] =3D "s_decperflevel", + [GFX9_S_TTRACEDATA] =3D "s_ttracedata", + [GFX9_S_CBRANCH_CDBGSYS] =3D "s_cbranch_cdbgsys", + [GFX9_S_CBRANCH_CDBGUSER] =3D "s_cbranch_cdbguser", + [GFX9_S_CBRANCH_CDBGSYS_OR_USER] =3D "s_cbranch_cdbgsys_or_user", + [GFX9_S_CBRANCH_CDBGSYS_AND_USER] =3D "s_cbranch_cdbgsys_and_user", + [GFX9_S_ENDPGM_SAVED] =3D "s_endpgm_saved", + [GFX9_S_SET_GPR_IDX_OFF] =3D "s_set_gpr_idx_off", + [GFX9_S_SET_GPR_IDX_MODE] =3D "s_set_gpr_idx_mode", + [GFX9_S_ENDPGM_ORDERED_PS_DONE] =3D "s_endpgm_ordered_ps_done", + }, + [AMDGCN_INSN_TYPE_SMEM] =3D { + [GFX9_S_LOAD_DWORD] =3D "s_load_dword", + [GFX9_S_LOAD_DWORDX2] =3D "s_load_dwordx2", + [GFX9_S_LOAD_DWORDX4] =3D "s_load_dwordx4", + [GFX9_S_LOAD_DWORDX8] =3D "s_load_dwordx8", + [GFX9_S_LOAD_DWORDX16] =3D "s_load_dwordx16", + [GFX9_S_SCRATCH_LOAD_DWORD] =3D "s_scratch_load_dword", + [GFX9_S_SCRATCH_LOAD_DWORDX2] =3D "s_scratch_load_dwordx2", + [GFX9_S_SCRATCH_LOAD_DWORDX4] =3D "s_scratch_load_dwordx4", + [GFX9_S_BUFFER_LOAD_DWORD] =3D "s_buffer_load_dword", + [GFX9_S_BUFFER_LOAD_DWORDX2] =3D "s_buffer_load_dwordx2", + [GFX9_S_BUFFER_LOAD_DWORDX4] =3D "s_buffer_load_dwordx4", + [GFX9_S_BUFFER_LOAD_DWORDX8] =3D "s_buffer_load_dwordx8", + [GFX9_S_BUFFER_LOAD_DWORDX16] =3D "s_buffer_load_dwordx16", + [GFX9_S_STORE_DWORD] =3D "s_store_dword", + [GFX9_S_STORE_DWORDX2] =3D "s_store_dwordx2", + [GFX9_S_STORE_DWORDX4] =3D "s_store_dwordx4", + [GFX9_S_SCRATCH_STORE_DWORD] =3D "s_scratch_store_dword", + [GFX9_S_SCRATCH_STORE_DWORDX2] =3D "s_scratch_store_dwordx2", + [GFX9_S_SCRATCH_STORE_DWORDX4] =3D "s_scratch_store_dwordx4", + [GFX9_S_BUFFER_STORE_DWORD] =3D "s_buffer_store_dword", + [GFX9_S_BUFFER_STORE_DWORDX2] =3D "s_buffer_store_dwordx2", + [GFX9_S_BUFFER_STORE_DWORDX4] =3D "s_buffer_store_dwordx4", + [GFX9_S_DCACHE_INV] =3D "s_dcache_inv", + [GFX9_S_DCACHE_WB] =3D "s_dcache_wb", + [GFX9_S_DCACHE_INV_VOL] =3D "s_dcache_inv_vol", + [GFX9_S_DCACHE_WB_VOL] =3D "s_dcache_wb_vol", + [GFX9_S_MEMTIME] =3D "s_memtime", + [GFX9_S_MEMREALTIME] =3D "s_memrealtime", + [GFX9_S_ATC_PROBE] =3D "s_atc_probe", + [GFX9_S_ATC_PROBE_BUFFER] =3D "s_atc_probe_buffer", + [GFX9_S_DCACHE_DISCARD] =3D "s_dcache_discard", + [GFX9_S_DCACHE_DISCARD_X2] =3D "s_dcache_discard_x2", + [GFX9_S_BUFFER_ATOMIC_SWAP] =3D "s_buffer_atomic_swap", + [GFX9_S_BUFFER_ATOMIC_CMPSWAP] =3D "s_buffer_atomic_cmpswap", + [GFX9_S_BUFFER_ATOMIC_ADD] =3D "s_buffer_atomic_add", + [GFX9_S_BUFFER_ATOMIC_SUB] =3D "s_buffer_atomic_sub", + [GFX9_S_BUFFER_ATOMIC_SMIN] =3D "s_buffer_atomic_smin", + [GFX9_S_BUFFER_ATOMIC_UMIN] =3D "s_buffer_atomic_umin", + [GFX9_S_BUFFER_ATOMIC_SMAX] =3D "s_buffer_atomic_smax", + [GFX9_S_BUFFER_ATOMIC_UMAX] =3D "s_buffer_atomic_umax", + [GFX9_S_BUFFER_ATOMIC_AND] =3D "s_buffer_atomic_and", + [GFX9_S_BUFFER_ATOMIC_OR] =3D "s_buffer_atomic_or", + [GFX9_S_BUFFER_ATOMIC_XOR] =3D "s_buffer_atomic_xor", + [GFX9_S_BUFFER_ATOMIC_INC] =3D "s_buffer_atomic_inc", + [GFX9_S_BUFFER_ATOMIC_DEC] =3D "s_buffer_atomic_dec", + [GFX9_S_BUFFER_ATOMIC_SWAP_X2] =3D "s_buffer_atomic_swap_x2", + [GFX9_S_BUFFER_ATOMIC_CMPSWAP_X2] =3D "s_buffer_atomic_cmpswap_x2", + [GFX9_S_BUFFER_ATOMIC_ADD_X2] =3D "s_buffer_atomic_add_x2", + [GFX9_S_BUFFER_ATOMIC_SUB_X2] =3D "s_buffer_atomic_sub_x2", + [GFX9_S_BUFFER_ATOMIC_SMIN_X2] =3D "s_buffer_atomic_smin_x2", + [GFX9_S_BUFFER_ATOMIC_UMIN_X2] =3D "s_buffer_atomic_umin_x2", + [GFX9_S_BUFFER_ATOMIC_SMAX_X2] =3D "s_buffer_atomic_smax_x2", + [GFX9_S_BUFFER_ATOMIC_UMAX_X2] =3D "s_buffer_atomic_umax_x2", + [GFX9_S_BUFFER_ATOMIC_AND_X2] =3D "s_buffer_atomic_and_x2", + [GFX9_S_BUFFER_ATOMIC_OR_X2] =3D "s_buffer_atomic_or_x2", + [GFX9_S_BUFFER_ATOMIC_XOR_X2] =3D "s_buffer_atomic_xor_x2", + [GFX9_S_BUFFER_ATOMIC_INC_X2] =3D "s_buffer_atomic_inc_x2", + [GFX9_S_BUFFER_ATOMIC_DEC_X2] =3D "s_buffer_atomic_dec_x2", + [GFX9_S_ATOMIC_SWAP] =3D "s_atomic_swap", + [GFX9_S_ATOMIC_CMPSWAP] =3D "s_atomic_cmpswap", + [GFX9_S_ATOMIC_ADD] =3D "s_atomic_add", + [GFX9_S_ATOMIC_SUB] =3D "s_atomic_sub", + [GFX9_S_ATOMIC_SMIN] =3D "s_atomic_smin", + [GFX9_S_ATOMIC_UMIN] =3D "s_atomic_umin", + [GFX9_S_ATOMIC_SMAX] =3D "s_atomic_smax", + [GFX9_S_ATOMIC_UMAX] =3D "s_atomic_umax", + [GFX9_S_ATOMIC_AND] =3D "s_atomic_and", + [GFX9_S_ATOMIC_OR] =3D "s_atomic_or", + [GFX9_S_ATOMIC_XOR] =3D "s_atomic_xor", + [GFX9_S_ATOMIC_INC] =3D "s_atomic_inc", + [GFX9_S_ATOMIC_DEC] =3D "s_atomic_dec", + [GFX9_S_ATOMIC_SWAP_X2] =3D "s_atomic_swap_x2", + [GFX9_S_ATOMIC_CMPSWAP_X2] =3D "s_atomic_cmpswap_x2", + [GFX9_S_ATOMIC_ADD_X2] =3D "s_atomic_add_x2", + [GFX9_S_ATOMIC_SUB_X2] =3D "s_atomic_sub_x2", + [GFX9_S_ATOMIC_SMIN_X2] =3D "s_atomic_smin_x2", + [GFX9_S_ATOMIC_UMIN_X2] =3D "s_atomic_umin_x2", + [GFX9_S_ATOMIC_SMAX_X2] =3D "s_atomic_smax_x2", + [GFX9_S_ATOMIC_UMAX_X2] =3D "s_atomic_umax_x2", + [GFX9_S_ATOMIC_AND_X2] =3D "s_atomic_and_x2", + [GFX9_S_ATOMIC_OR_X2] =3D "s_atomic_or_x2", + [GFX9_S_ATOMIC_XOR_X2] =3D "s_atomic_xor_x2", + [GFX9_S_ATOMIC_INC_X2] =3D "s_atomic_inc_x2", + [GFX9_S_ATOMIC_DEC_X2] =3D "s_atomic_dec_x2", + }, + [AMDGCN_INSN_TYPE_VOP2] =3D { + [GFX9_V_CNDMASK_B32] =3D "v_cndmask_b32", + [GFX9_V_ADD_F32] =3D "v_add_f32", + [GFX9_V_SUB_F32] =3D "v_sub_f32", + [GFX9_V_SUBREV_F32] =3D "v_subrev_f32", + [GFX9_V_MUL_LEGACY_F32] =3D "v_mul_legacy_f32", + [GFX9_V_MUL_F32] =3D "v_mul_f32", + [GFX9_V_MUL_I32_I24] =3D "v_mul_i32_i24", + [GFX9_V_MUL_HI_I32_I24] =3D "v_mul_hi_i32_i24", + [GFX9_V_MUL_U32_U24] =3D "v_mul_u32_u24", + [GFX9_V_MUL_HI_U32_U24] =3D "v_mul_hi_u32_u24", + [GFX9_V_MIN_F32] =3D "v_min_f32", + [GFX9_V_MAX_F32] =3D "v_max_f32", + [GFX9_V_MIN_I32] =3D "v_min_i32", + [GFX9_V_MAX_I32] =3D "v_max_i32", + [GFX9_V_MIN_U32] =3D "v_min_u32", + [GFX9_V_MAX_U32] =3D "v_max_u32", + [GFX9_V_LSHRREV_B32] =3D "v_lshrrev_b32", + [GFX9_V_ASHRREV_I32] =3D "v_ashrrev_i32", + [GFX9_V_LSHLREV_B32] =3D "v_lshlrev_b32", + [GFX9_V_AND_B32] =3D "v_and_b32", + [GFX9_V_OR_B32] =3D "v_or_b32", + [GFX9_V_XOR_B32] =3D "v_xor_b32", + [GFX9_V_MAC_F32] =3D "v_mac_f32", + [GFX9_V_MADMK_F32] =3D "v_madmk_f32", + [GFX9_V_MADAK_F32] =3D "v_madak_f32", + [GFX9_V_ADD_CO_U32] =3D "v_add_co_u32", + [GFX9_V_SUB_CO_U32] =3D "v_sub_co_u32", + [GFX9_V_SUBREV_CO_U32] =3D "v_subrev_co_u32", + [GFX9_V_ADDC_CO_U32] =3D "v_addc_co_u32", + [GFX9_V_SUBB_CO_U32] =3D "v_subb_co_u32", + [GFX9_V_SUBBREV_CO_U32] =3D "v_subbrev_co_u32", + [GFX9_V_ADD_F16] =3D "v_add_f16", + [GFX9_V_SUB_F16] =3D "v_sub_f16", + [GFX9_V_SUBREV_F16] =3D "v_subrev_f16", + [GFX9_V_MUL_F16] =3D "v_mul_f16", + [GFX9_V_MAC_F16] =3D "v_mac_f16", + [GFX9_V_MADMK_F16] =3D "v_madmk_f16", + [GFX9_V_MADAK_F16] =3D "v_madak_f16", + [GFX9_V_ADD_U16] =3D "v_add_u16", + [GFX9_V_SUB_U16] =3D "v_sub_u16", + [GFX9_V_SUBREV_U16] =3D "v_subrev_u16", + [GFX9_V_MUL_LO_U16] =3D "v_mul_lo_u16", + [GFX9_V_LSHLREV_B16] =3D "v_lshlrev_b16", + [GFX9_V_LSHRREV_B16] =3D "v_lshrrev_b16", + [GFX9_V_ASHRREV_I16] =3D "v_ashrrev_i16", + [GFX9_V_MAX_F16] =3D "v_max_f16", + [GFX9_V_MIN_F16] =3D "v_min_f16", + [GFX9_V_MAX_U16] =3D "v_max_u16", + [GFX9_V_MAX_I16] =3D "v_max_i16", + [GFX9_V_MIN_U16] =3D "v_min_u16", + [GFX9_V_MIN_I16] =3D "v_min_i16", + [GFX9_V_LDEXP_F16] =3D "v_ldexp_f16", + [GFX9_V_ADD_U32] =3D "v_add_u32", + [GFX9_V_SUB_U32] =3D "v_sub_u32", + [GFX9_V_SUBREV_U32] =3D "v_subrev_u32", + }, + [AMDGCN_INSN_TYPE_VOP1] =3D { + [GFX9_V_NOP] =3D "v_nop", + [GFX9_V_MOV_B32] =3D "v_mov_b32", + [GFX9_V_READFIRSTLANE_B32] =3D "v_readfirstlane_b32", + [GFX9_V_CVT_I32_F64] =3D "v_cvt_i32_f64", + [GFX9_V_CVT_F64_I32] =3D "v_cvt_f64_i32", + [GFX9_V_CVT_F32_I32] =3D "v_cvt_f32_i32", + [GFX9_V_CVT_F32_U32] =3D "v_cvt_f32_u32", + [GFX9_V_CVT_U32_F32] =3D "v_cvt_u32_f32", + [GFX9_V_CVT_I32_F32] =3D "v_cvt_i32_f32", + [GFX9_V_CVT_F16_F32] =3D "v_cvt_f16_f32", + [GFX9_V_CVT_F32_F16] =3D "v_cvt_f32_f16", + [GFX9_V_CVT_RPI_I32_F32] =3D "v_cvt_rpi_i32_f32", + [GFX9_V_CVT_FLR_I32_F32] =3D "v_cvt_flr_i32_f32", + [GFX9_V_CVT_OFF_F32_I4] =3D "v_cvt_off_f32_i4", + [GFX9_V_CVT_F32_F64] =3D "v_cvt_f32_f64", + [GFX9_V_CVT_F64_F32] =3D "v_cvt_f64_f32", + [GFX9_V_CVT_F32_UBYTE0] =3D "v_cvt_f32_ubyte0", + [GFX9_V_CVT_F32_UBYTE1] =3D "v_cvt_f32_ubyte1", + [GFX9_V_CVT_F32_UBYTE2] =3D "v_cvt_f32_ubyte2", + [GFX9_V_CVT_F32_UBYTE3] =3D "v_cvt_f32_ubyte3", + [GFX9_V_CVT_U32_F64] =3D "v_cvt_u32_f64", + [GFX9_V_CVT_F64_U32] =3D "v_cvt_f64_u32", + [GFX9_V_TRUNC_F64] =3D "v_trunc_f64", + [GFX9_V_CEIL_F64] =3D "v_ceil_f64", + [GFX9_V_RNDNE_F64] =3D "v_rndne_f64", + [GFX9_V_FLOOR_F64] =3D "v_floor_f64", + [GFX9_V_FRACT_F32] =3D "v_fract_f32", + [GFX9_V_TRUNC_F32] =3D "v_trunc_f32", + [GFX9_V_CEIL_F32] =3D "v_ceil_f32", + [GFX9_V_RNDNE_F32] =3D "v_rndne_f32", + [GFX9_V_FLOOR_F32] =3D "v_floor_f32", + [GFX9_V_EXP_F32] =3D "v_exp_f32", + [GFX9_V_LOG_F32] =3D "v_log_f32", + [GFX9_V_RCP_F32] =3D "v_rcp_f32", + [GFX9_V_RCP_IFLAG_F32] =3D "v_rcp_iflag_f32", + [GFX9_V_RSQ_F32] =3D "v_rsq_f32", + [GFX9_V_RCP_F64] =3D "v_rcp_f64", + [GFX9_V_RSQ_F64] =3D "v_rsq_f64", + [GFX9_V_SQRT_F32] =3D "v_sqrt_f32", + [GFX9_V_SQRT_F64] =3D "v_sqrt_f64", + [GFX9_V_SIN_F32] =3D "v_sin_f32", + [GFX9_V_COS_F32] =3D "v_cos_f32", + [GFX9_V_NOT_B32] =3D "v_not_b32", + [GFX9_V_BFREV_B32] =3D "v_bfrev_b32", + [GFX9_V_FFBH_U32] =3D "v_ffbh_u32", + [GFX9_V_FFBL_B32] =3D "v_ffbl_b32", + [GFX9_V_FFBH_I32] =3D "v_ffbh_i32", + [GFX9_V_FREXP_EXP_I32_F64] =3D "v_frexp_exp_i32_f64", + [GFX9_V_FREXP_MANT_F64] =3D "v_frexp_mant_f64", + [GFX9_V_FRACT_F64] =3D "v_fract_f64", + [GFX9_V_FREXP_EXP_I32_F32] =3D "v_frexp_exp_i32_f32", + [GFX9_V_FREXP_MANT_F32] =3D "v_frexp_mant_f32", + [GFX9_V_CLREXCP] =3D "v_clrexcp", + [GFX9_V_SCREEN_PARTITION_4SE_B32] =3D "v_screen_partition_4se_b32", + [GFX9_V_CVT_F16_U16] =3D "v_cvt_f16_u16", + [GFX9_V_CVT_F16_I16] =3D "v_cvt_f16_i16", + [GFX9_V_CVT_U16_F16] =3D "v_cvt_u16_f16", + [GFX9_V_CVT_I16_F16] =3D "v_cvt_i16_f16", + [GFX9_V_RCP_F16] =3D "v_rcp_f16", + [GFX9_V_SQRT_F16] =3D "v_sqrt_f16", + [GFX9_V_RSQ_F16] =3D "v_rsq_f16", + [GFX9_V_LOG_F16] =3D "v_log_f16", + [GFX9_V_EXP_F16] =3D "v_exp_f16", + [GFX9_V_FREXP_MANT_F16] =3D "v_frexp_mant_f16", + [GFX9_V_FREXP_EXP_I16_F16] =3D "v_frexp_exp_i16_f16", + [GFX9_V_FLOOR_F16] =3D "v_floor_f16", + [GFX9_V_CEIL_F16] =3D "v_ceil_f16", + [GFX9_V_TRUNC_F16] =3D "v_trunc_f16", + [GFX9_V_RNDNE_F16] =3D "v_rndne_f16", + [GFX9_V_FRACT_F16] =3D "v_fract_f16", + [GFX9_V_SIN_F16] =3D "v_sin_f16", + [GFX9_V_COS_F16] =3D "v_cos_f16", + [GFX9_V_EXP_LEGACY_F32] =3D "v_exp_legacy_f32", + [GFX9_V_LOG_LEGACY_F32] =3D "v_log_legacy_f32", + [GFX9_V_CVT_NORM_I16_F16] =3D "v_cvt_norm_i16_f16", + [GFX9_V_CVT_NORM_U16_F16] =3D "v_cvt_norm_u16_f16", + [GFX9_V_SAT_PK_U8_I16] =3D "v_sat_pk_u8_i16", + [GFX9_V_SWAP_B32] =3D "v_swap_b32", + }, + [AMDGCN_INSN_TYPE_VOPC] =3D { + [GFX9_V_CMP_CLASS_F32] =3D "v_cmp_class_f32", + [GFX9_V_CMPX_CLASS_F32] =3D "v_cmpx_class_f32", + [GFX9_V_CMP_CLASS_F64] =3D "v_cmp_class_f64", + [GFX9_V_CMPX_CLASS_F64] =3D "v_cmpx_class_f64", + [GFX9_V_CMP_CLASS_F16] =3D "v_cmp_class_f16", + [GFX9_V_CMPX_CLASS_F16] =3D "v_cmpx_class_f16", + [GFX9_V_CMP_F_F16] =3D "v_cmp_f_f16", + [GFX9_V_CMP_LT_F16] =3D "v_cmp_lt_f16", + [GFX9_V_CMP_EQ_F16] =3D "v_cmp_eq_f16", + [GFX9_V_CMP_LE_F16] =3D "v_cmp_le_f16", + [GFX9_V_CMP_GT_F16] =3D "v_cmp_gt_f16", + [GFX9_V_CMP_LG_F16] =3D "v_cmp_lg_f16", + [GFX9_V_CMP_GE_F16] =3D "v_cmp_ge_f16", + [GFX9_V_CMP_O_F16] =3D "v_cmp_o_f16", + [GFX9_V_CMP_U_F16] =3D "v_cmp_u_f16", + [GFX9_V_CMP_NGE_F16] =3D "v_cmp_nge_f16", + [GFX9_V_CMP_NLG_F16] =3D "v_cmp_nlg_f16", + [GFX9_V_CMP_NGT_F16] =3D "v_cmp_ngt_f16", + [GFX9_V_CMP_NLE_F16] =3D "v_cmp_nle_f16", + [GFX9_V_CMP_NEQ_F16] =3D "v_cmp_neq_f16", + [GFX9_V_CMP_NLT_F16] =3D "v_cmp_nlt_f16", + [GFX9_V_CMP_TRU_F16] =3D "v_cmp_tru_f16", + [GFX9_V_CMPX_F_F16] =3D "v_cmpx_f_f16", + [GFX9_V_CMPX_LT_F16] =3D "v_cmpx_lt_f16", + [GFX9_V_CMPX_EQ_F16] =3D "v_cmpx_eq_f16", + [GFX9_V_CMPX_LE_F16] =3D "v_cmpx_le_f16", + [GFX9_V_CMPX_GT_F16] =3D "v_cmpx_gt_f16", + [GFX9_V_CMPX_LG_F16] =3D "v_cmpx_lg_f16", + [GFX9_V_CMPX_GE_F16] =3D "v_cmpx_ge_f16", + [GFX9_V_CMPX_O_F16] =3D "v_cmpx_o_f16", + [GFX9_V_CMPX_U_F16] =3D "v_cmpx_u_f16", + [GFX9_V_CMPX_NGE_F16] =3D "v_cmpx_nge_f16", + [GFX9_V_CMPX_NLG_F16] =3D "v_cmpx_nlg_f16", + [GFX9_V_CMPX_NGT_F16] =3D "v_cmpx_ngt_f16", + [GFX9_V_CMPX_NLE_F16] =3D "v_cmpx_nle_f16", + [GFX9_V_CMPX_NEQ_F16] =3D "v_cmpx_neq_f16", + [GFX9_V_CMPX_NLT_F16] =3D "v_cmpx_nlt_f16", + [GFX9_V_CMPX_TRU_F16] =3D "v_cmpx_tru_f16", + [GFX9_V_CMP_F_F32] =3D "v_cmp_f_f32", + [GFX9_V_CMP_LT_F32] =3D "v_cmp_lt_f32", + [GFX9_V_CMP_EQ_F32] =3D "v_cmp_eq_f32", + [GFX9_V_CMP_LE_F32] =3D "v_cmp_le_f32", + [GFX9_V_CMP_GT_F32] =3D "v_cmp_gt_f32", + [GFX9_V_CMP_LG_F32] =3D "v_cmp_lg_f32", + [GFX9_V_CMP_GE_F32] =3D "v_cmp_ge_f32", + [GFX9_V_CMP_O_F32] =3D "v_cmp_o_f32", + [GFX9_V_CMP_U_F32] =3D "v_cmp_u_f32", + [GFX9_V_CMP_NGE_F32] =3D "v_cmp_nge_f32", + [GFX9_V_CMP_NLG_F32] =3D "v_cmp_nlg_f32", + [GFX9_V_CMP_NGT_F32] =3D "v_cmp_ngt_f32", + [GFX9_V_CMP_NLE_F32] =3D "v_cmp_nle_f32", + [GFX9_V_CMP_NEQ_F32] =3D "v_cmp_neq_f32", + [GFX9_V_CMP_NLT_F32] =3D "v_cmp_nlt_f32", + [GFX9_V_CMP_TRU_F32] =3D "v_cmp_tru_f32", + [GFX9_V_CMPX_F_F32] =3D "v_cmpx_f_f32", + [GFX9_V_CMPX_LT_F32] =3D "v_cmpx_lt_f32", + [GFX9_V_CMPX_EQ_F32] =3D "v_cmpx_eq_f32", + [GFX9_V_CMPX_LE_F32] =3D "v_cmpx_le_f32", + [GFX9_V_CMPX_GT_F32] =3D "v_cmpx_gt_f32", + [GFX9_V_CMPX_LG_F32] =3D "v_cmpx_lg_f32", + [GFX9_V_CMPX_GE_F32] =3D "v_cmpx_ge_f32", + [GFX9_V_CMPX_O_F32] =3D "v_cmpx_o_f32", + [GFX9_V_CMPX_U_F32] =3D "v_cmpx_u_f32", + [GFX9_V_CMPX_NGE_F32] =3D "v_cmpx_nge_f32", + [GFX9_V_CMPX_NLG_F32] =3D "v_cmpx_nlg_f32", + [GFX9_V_CMPX_NGT_F32] =3D "v_cmpx_ngt_f32", + [GFX9_V_CMPX_NLE_F32] =3D "v_cmpx_nle_f32", + [GFX9_V_CMPX_NEQ_F32] =3D "v_cmpx_neq_f32", + [GFX9_V_CMPX_NLT_F32] =3D "v_cmpx_nlt_f32", + [GFX9_V_CMPX_TRU_F32] =3D "v_cmpx_tru_f32", + [GFX9_V_CMP_F_F64] =3D "v_cmp_f_f64", + [GFX9_V_CMP_LT_F64] =3D "v_cmp_lt_f64", + [GFX9_V_CMP_EQ_F64] =3D "v_cmp_eq_f64", + [GFX9_V_CMP_LE_F64] =3D "v_cmp_le_f64", + [GFX9_V_CMP_GT_F64] =3D "v_cmp_gt_f64", + [GFX9_V_CMP_LG_F64] =3D "v_cmp_lg_f64", + [GFX9_V_CMP_GE_F64] =3D "v_cmp_ge_f64", + [GFX9_V_CMP_O_F64] =3D "v_cmp_o_f64", + [GFX9_V_CMP_U_F64] =3D "v_cmp_u_f64", + [GFX9_V_CMP_NGE_F64] =3D "v_cmp_nge_f64", + [GFX9_V_CMP_NLG_F64] =3D "v_cmp_nlg_f64", + [GFX9_V_CMP_NGT_F64] =3D "v_cmp_ngt_f64", + [GFX9_V_CMP_NLE_F64] =3D "v_cmp_nle_f64", + [GFX9_V_CMP_NEQ_F64] =3D "v_cmp_neq_f64", + [GFX9_V_CMP_NLT_F64] =3D "v_cmp_nlt_f64", + [GFX9_V_CMP_TRU_F64] =3D "v_cmp_tru_f64", + [GFX9_V_CMPX_F_F64] =3D "v_cmpx_f_f64", + [GFX9_V_CMPX_LT_F64] =3D "v_cmpx_lt_f64", + [GFX9_V_CMPX_EQ_F64] =3D "v_cmpx_eq_f64", + [GFX9_V_CMPX_LE_F64] =3D "v_cmpx_le_f64", + [GFX9_V_CMPX_GT_F64] =3D "v_cmpx_gt_f64", + [GFX9_V_CMPX_LG_F64] =3D "v_cmpx_lg_f64", + [GFX9_V_CMPX_GE_F64] =3D "v_cmpx_ge_f64", + [GFX9_V_CMPX_O_F64] =3D "v_cmpx_o_f64", + [GFX9_V_CMPX_U_F64] =3D "v_cmpx_u_f64", + [GFX9_V_CMPX_NGE_F64] =3D "v_cmpx_nge_f64", + [GFX9_V_CMPX_NLG_F64] =3D "v_cmpx_nlg_f64", + [GFX9_V_CMPX_NGT_F64] =3D "v_cmpx_ngt_f64", + [GFX9_V_CMPX_NLE_F64] =3D "v_cmpx_nle_f64", + [GFX9_V_CMPX_NEQ_F64] =3D "v_cmpx_neq_f64", + [GFX9_V_CMPX_NLT_F64] =3D "v_cmpx_nlt_f64", + [GFX9_V_CMPX_TRU_F64] =3D "v_cmpx_tru_f64", + [GFX9_V_CMP_F_I16] =3D "v_cmp_f_i16", + [GFX9_V_CMP_LT_I16] =3D "v_cmp_lt_i16", + [GFX9_V_CMP_EQ_I16] =3D "v_cmp_eq_i16", + [GFX9_V_CMP_LE_I16] =3D "v_cmp_le_i16", + [GFX9_V_CMP_GT_I16] =3D "v_cmp_gt_i16", + [GFX9_V_CMP_NE_I16] =3D "v_cmp_ne_i16", + [GFX9_V_CMP_GE_I16] =3D "v_cmp_ge_i16", + [GFX9_V_CMP_T_I16] =3D "v_cmp_t_i16", + [GFX9_V_CMP_F_U16] =3D "v_cmp_f_u16", + [GFX9_V_CMP_LT_U16] =3D "v_cmp_lt_u16", + [GFX9_V_CMP_EQ_U16] =3D "v_cmp_eq_u16", + [GFX9_V_CMP_LE_U16] =3D "v_cmp_le_u16", + [GFX9_V_CMP_GT_U16] =3D "v_cmp_gt_u16", + [GFX9_V_CMP_NE_U16] =3D "v_cmp_ne_u16", + [GFX9_V_CMP_GE_U16] =3D "v_cmp_ge_u16", + [GFX9_V_CMP_T_U16] =3D "v_cmp_t_u16", + [GFX9_V_CMPX_F_I16] =3D "v_cmpx_f_i16", + [GFX9_V_CMPX_LT_I16] =3D "v_cmpx_lt_i16", + [GFX9_V_CMPX_EQ_I16] =3D "v_cmpx_eq_i16", + [GFX9_V_CMPX_LE_I16] =3D "v_cmpx_le_i16", + [GFX9_V_CMPX_GT_I16] =3D "v_cmpx_gt_i16", + [GFX9_V_CMPX_NE_I16] =3D "v_cmpx_ne_i16", + [GFX9_V_CMPX_GE_I16] =3D "v_cmpx_ge_i16", + [GFX9_V_CMPX_T_I16] =3D "v_cmpx_t_i16", + [GFX9_V_CMPX_F_U16] =3D "v_cmpx_f_u16", + [GFX9_V_CMPX_LT_U16] =3D "v_cmpx_lt_u16", + [GFX9_V_CMPX_EQ_U16] =3D "v_cmpx_eq_u16", + [GFX9_V_CMPX_LE_U16] =3D "v_cmpx_le_u16", + [GFX9_V_CMPX_GT_U16] =3D "v_cmpx_gt_u16", + [GFX9_V_CMPX_NE_U16] =3D "v_cmpx_ne_u16", + [GFX9_V_CMPX_GE_U16] =3D "v_cmpx_ge_u16", + [GFX9_V_CMPX_T_U16] =3D "v_cmpx_t_u16", + [GFX9_V_CMP_F_I32] =3D "v_cmp_f_i32", + [GFX9_V_CMP_LT_I32] =3D "v_cmp_lt_i32", + [GFX9_V_CMP_EQ_I32] =3D "v_cmp_eq_i32", + [GFX9_V_CMP_LE_I32] =3D "v_cmp_le_i32", + [GFX9_V_CMP_GT_I32] =3D "v_cmp_gt_i32", + [GFX9_V_CMP_NE_I32] =3D "v_cmp_ne_i32", + [GFX9_V_CMP_GE_I32] =3D "v_cmp_ge_i32", + [GFX9_V_CMP_T_I32] =3D "v_cmp_t_i32", + [GFX9_V_CMP_F_U32] =3D "v_cmp_f_u32", + [GFX9_V_CMP_LT_U32] =3D "v_cmp_lt_u32", + [GFX9_V_CMP_EQ_U32] =3D "v_cmp_eq_u32", + [GFX9_V_CMP_LE_U32] =3D "v_cmp_le_u32", + [GFX9_V_CMP_GT_U32] =3D "v_cmp_gt_u32", + [GFX9_V_CMP_NE_U32] =3D "v_cmp_ne_u32", + [GFX9_V_CMP_GE_U32] =3D "v_cmp_ge_u32", + [GFX9_V_CMP_T_U32] =3D "v_cmp_t_u32", + [GFX9_V_CMPX_F_I32] =3D "v_cmpx_f_i32", + [GFX9_V_CMPX_LT_I32] =3D "v_cmpx_lt_i32", + [GFX9_V_CMPX_EQ_I32] =3D "v_cmpx_eq_i32", + [GFX9_V_CMPX_LE_I32] =3D "v_cmpx_le_i32", + [GFX9_V_CMPX_GT_I32] =3D "v_cmpx_gt_i32", + [GFX9_V_CMPX_NE_I32] =3D "v_cmpx_ne_i32", + [GFX9_V_CMPX_GE_I32] =3D "v_cmpx_ge_i32", + [GFX9_V_CMPX_T_I32] =3D "v_cmpx_t_i32", + [GFX9_V_CMPX_F_U32] =3D "v_cmpx_f_u32", + [GFX9_V_CMPX_LT_U32] =3D "v_cmpx_lt_u32", + [GFX9_V_CMPX_EQ_U32] =3D "v_cmpx_eq_u32", + [GFX9_V_CMPX_LE_U32] =3D "v_cmpx_le_u32", + [GFX9_V_CMPX_GT_U32] =3D "v_cmpx_gt_u32", + [GFX9_V_CMPX_NE_U32] =3D "v_cmpx_ne_u32", + [GFX9_V_CMPX_GE_U32] =3D "v_cmpx_ge_u32", + [GFX9_V_CMPX_T_U32] =3D "v_cmpx_t_u32", + [GFX9_V_CMP_F_I64] =3D "v_cmp_f_i64", + [GFX9_V_CMP_LT_I64] =3D "v_cmp_lt_i64", + [GFX9_V_CMP_EQ_I64] =3D "v_cmp_eq_i64", + [GFX9_V_CMP_LE_I64] =3D "v_cmp_le_i64", + [GFX9_V_CMP_GT_I64] =3D "v_cmp_gt_i64", + [GFX9_V_CMP_NE_I64] =3D "v_cmp_ne_i64", + [GFX9_V_CMP_GE_I64] =3D "v_cmp_ge_i64", + [GFX9_V_CMP_T_I64] =3D "v_cmp_t_i64", + [GFX9_V_CMP_F_U64] =3D "v_cmp_f_u64", + [GFX9_V_CMP_LT_U64] =3D "v_cmp_lt_u64", + [GFX9_V_CMP_EQ_U64] =3D "v_cmp_eq_u64", + [GFX9_V_CMP_LE_U64] =3D "v_cmp_le_u64", + [GFX9_V_CMP_GT_U64] =3D "v_cmp_gt_u64", + [GFX9_V_CMP_NE_U64] =3D "v_cmp_ne_u64", + [GFX9_V_CMP_GE_U64] =3D "v_cmp_ge_u64", + [GFX9_V_CMP_T_U64] =3D "v_cmp_t_u64", + [GFX9_V_CMPX_F_I64] =3D "v_cmpx_f_i64", + [GFX9_V_CMPX_LT_I64] =3D "v_cmpx_lt_i64", + [GFX9_V_CMPX_EQ_I64] =3D "v_cmpx_eq_i64", + [GFX9_V_CMPX_LE_I64] =3D "v_cmpx_le_i64", + [GFX9_V_CMPX_GT_I64] =3D "v_cmpx_gt_i64", + [GFX9_V_CMPX_NE_I64] =3D "v_cmpx_ne_i64", + [GFX9_V_CMPX_GE_I64] =3D "v_cmpx_ge_i64", + [GFX9_V_CMPX_T_I64] =3D "v_cmpx_t_i64", + [GFX9_V_CMPX_F_U64] =3D "v_cmpx_f_u64", + [GFX9_V_CMPX_LT_U64] =3D "v_cmpx_lt_u64", + [GFX9_V_CMPX_EQ_U64] =3D "v_cmpx_eq_u64", + [GFX9_V_CMPX_LE_U64] =3D "v_cmpx_le_u64", + [GFX9_V_CMPX_GT_U64] =3D "v_cmpx_gt_u64", + [GFX9_V_CMPX_NE_U64] =3D "v_cmpx_ne_u64", + [GFX9_V_CMPX_GE_U64] =3D "v_cmpx_ge_u64", + [GFX9_V_CMPX_T_U64] =3D "v_cmpx_t_u64", + }, + [AMDGCN_INSN_TYPE_VOP3A] =3D { + [GFX9_V_MAD_LEGACY_F32] =3D "v_mad_legacy_f32", + [GFX9_V_MAD_F32] =3D "v_mad_f32", + [GFX9_V_MAD_I32_I24] =3D "v_mad_i32_i24", + [GFX9_V_MAD_U32_U24] =3D "v_mad_u32_u24", + [GFX9_V_CUBEID_F32] =3D "v_cubeid_f32", + [GFX9_V_CUBESC_F32] =3D "v_cubesc_f32", + [GFX9_V_CUBETC_F32] =3D "v_cubetc_f32", + [GFX9_V_CUBEMA_F32] =3D "v_cubema_f32", + [GFX9_V_BFE_U32] =3D "v_bfe_u32", + [GFX9_V_BFE_I32] =3D "v_bfe_i32", + [GFX9_V_BFI_B32] =3D "v_bfi_b32", + [GFX9_V_FMA_F32] =3D "v_fma_f32", + [GFX9_V_FMA_F64] =3D "v_fma_f64", + [GFX9_V_LERP_U8] =3D "v_lerp_u8", + [GFX9_V_ALIGNBIT_B32] =3D "v_alignbit_b32", + [GFX9_V_ALIGNBYTE_B32] =3D "v_alignbyte_b32", + [GFX9_V_MIN3_F32] =3D "v_min3_f32", + [GFX9_V_MIN3_I32] =3D "v_min3_i32", + [GFX9_V_MIN3_U32] =3D "v_min3_u32", + [GFX9_V_MAX3_F32] =3D "v_max3_f32", + [GFX9_V_MAX3_I32] =3D "v_max3_i32", + [GFX9_V_MAX3_U32] =3D "v_max3_u32", + [GFX9_V_MED3_F32] =3D "v_med3_f32", + [GFX9_V_MED3_I32] =3D "v_med3_i32", + [GFX9_V_MED3_U32] =3D "v_med3_u32", + [GFX9_V_SAD_U8] =3D "v_sad_u8", + [GFX9_V_SAD_HI_U8] =3D "v_sad_hi_u8", + [GFX9_V_SAD_U16] =3D "v_sad_u16", + [GFX9_V_SAD_U32] =3D "v_sad_u32", + [GFX9_V_CVT_PK_U8_F32] =3D "v_cvt_pk_u8_f32", + [GFX9_V_DIV_FIXUP_F32] =3D "v_div_fixup_f32", + [GFX9_V_DIV_FIXUP_F64] =3D "v_div_fixup_f64", + [GFX9_V_DIV_FMAS_F32] =3D "v_div_fmas_f32", + [GFX9_V_DIV_FMAS_F64] =3D "v_div_fmas_f64", + [GFX9_V_MSAD_U8] =3D "v_msad_u8", + [GFX9_V_QSAD_PK_U16_U8] =3D "v_qsad_pk_u16_u8", + [GFX9_V_MQSAD_PK_U16_U8] =3D "v_mqsad_pk_u16_u8", + [GFX9_V_MQSAD_U32_U8] =3D "v_mqsad_u32_u8", + [GFX9_V_MAD_LEGACY_F16] =3D "v_mad_legacy_f16", + [GFX9_V_MAD_LEGACY_U16] =3D "v_mad_legacy_u16", + [GFX9_V_MAD_LEGACY_I16] =3D "v_mad_legacy_i16", + [GFX9_V_PERM_B32] =3D "v_perm_b32", + [GFX9_V_FMA_LEGACY_F16] =3D "v_fma_legacy_f16", + [GFX9_V_DIV_FIXUP_LEGACY_F16] =3D "v_div_fixup_legacy_f16", + [GFX9_V_CVT_PKACCUM_U8_F32] =3D "v_cvt_pkaccum_u8_f32", + [GFX9_V_MAD_U32_U16] =3D "v_mad_u32_u16", + [GFX9_V_MAD_I32_I16] =3D "v_mad_i32_i16", + [GFX9_V_XAD_U32] =3D "v_xad_u32", + [GFX9_V_MIN3_F16] =3D "v_min3_f16", + [GFX9_V_MIN3_I16] =3D "v_min3_i16", + [GFX9_V_MIN3_U16] =3D "v_min3_u16", + [GFX9_V_MAX3_F16] =3D "v_max3_f16", + [GFX9_V_MAX3_I16] =3D "v_max3_i16", + [GFX9_V_MAX3_U16] =3D "v_max3_u16", + [GFX9_V_MED3_F16] =3D "v_med3_f16", + [GFX9_V_MED3_I16] =3D "v_med3_i16", + [GFX9_V_MED3_U16] =3D "v_med3_u16", + [GFX9_V_LSHL_ADD_U32] =3D "v_lshl_add_u32", + [GFX9_V_ADD_LSHL_U32] =3D "v_add_lshl_u32", + [GFX9_V_ADD3_U32] =3D "v_add3_u32", + [GFX9_V_LSHL_OR_B32] =3D "v_lshl_or_b32", + [GFX9_V_AND_OR_B32] =3D "v_and_or_b32", + [GFX9_V_OR3_B32] =3D "v_or3_b32", + [GFX9_V_MAD_F16] =3D "v_mad_f16", + [GFX9_V_MAD_U16] =3D "v_mad_u16", + [GFX9_V_MAD_I16] =3D "v_mad_i16", + [GFX9_V_FMA_F16] =3D "v_fma_f16", + [GFX9_V_DIV_FIXUP_F16] =3D "v_div_fixup_f16", + [GFX9_V_INTERP_P1LL_F16] =3D "v_interp_p1ll_f16", + [GFX9_V_INTERP_P1LV_F16] =3D "v_interp_p1lv_f16", + [GFX9_V_INTERP_P2_LEGACY_F16] =3D "v_interp_p2_legacy_f16", + [GFX9_V_INTERP_P2_F16] =3D "v_interp_p2_f16", + [GFX9_V_ADD_F64] =3D "v_add_f64", + [GFX9_V_MUL_F64] =3D "v_mul_f64", + [GFX9_V_MIN_F64] =3D "v_min_f64", + [GFX9_V_MAX_F64] =3D "v_max_f64", + [GFX9_V_LDEXP_F64] =3D "v_ldexp_f64", + [GFX9_V_MUL_LO_U32] =3D "v_mul_lo_u32", + [GFX9_V_MUL_HI_U32] =3D "v_mul_hi_u32", + [GFX9_V_MUL_HI_I32] =3D "v_mul_hi_i32", + [GFX9_V_LDEXP_F32] =3D "v_ldexp_f32", + [GFX9_V_READLANE_B32] =3D "v_readlane_b32", + [GFX9_V_WRITELANE_B32] =3D "v_writelane_b32", + [GFX9_V_BCNT_U32_B32] =3D "v_bcnt_u32_b32", + [GFX9_V_MBCNT_LO_U32_B32] =3D "v_mbcnt_lo_u32_b32", + [GFX9_V_MBCNT_HI_U32_B32] =3D "v_mbcnt_hi_u32_b32", + [GFX9_V_LSHLREV_B64] =3D "v_lshlrev_b64", + [GFX9_V_LSHRREV_B64] =3D "v_lshrrev_b64", + [GFX9_V_ASHRREV_I64] =3D "v_ashrrev_i64", + [GFX9_V_TRIG_PREOP_F64] =3D "v_trig_preop_f64", + [GFX9_V_BFM_B32] =3D "v_bfm_b32", + [GFX9_V_CVT_PKNORM_I16_F32] =3D "v_cvt_pknorm_i16_f32", + [GFX9_V_CVT_PKNORM_U16_F32] =3D "v_cvt_pknorm_u16_f32", + [GFX9_V_CVT_PKRTZ_F16_F32] =3D "v_cvt_pkrtz_f16_f32", + [GFX9_V_CVT_PK_U16_U32] =3D "v_cvt_pk_u16_u32", + [GFX9_V_CVT_PK_I16_I32] =3D "v_cvt_pk_i16_i32", + [GFX9_V_CVT_PKNORM_I16_F16] =3D "v_cvt_pknorm_i16_f16", + [GFX9_V_CVT_PKNORM_U16_F16] =3D "v_cvt_pknorm_u16_f16", + [GFX9_V_ADD_I32] =3D "v_add_i32", + [GFX9_V_SUB_I32] =3D "v_sub_i32", + [GFX9_V_ADD_I16] =3D "v_add_i16", + [GFX9_V_SUB_I16] =3D "v_sub_i16", + [GFX9_V_PACK_B32_F16] =3D "v_pack_b32_f16", + }, + [AMDGCN_INSN_TYPE_VOP3B] =3D { + [GFX9_V_DIV_SCALE_F64] =3D "v_div_scale_f64", + [GFX9_V_MAD_U64_U32] =3D "v_mad_u64_u32", + [GFX9_V_MAD_I64_I32] =3D "v_mad_i64_i32", + }, + [AMDGCN_INSN_TYPE_VOP3P] =3D { + [GFX9_V_PK_MAD_I16] =3D "v_pk_mad_i16", + [GFX9_V_PK_MUL_LO_U16] =3D "v_pk_mul_lo_u16", + [GFX9_V_PK_ADD_I16] =3D "v_pk_add_i16", + [GFX9_V_PK_SUB_I16] =3D "v_pk_sub_i16", + [GFX9_V_PK_LSHLREV_B16] =3D "v_pk_lshlrev_b16", + [GFX9_V_PK_LSHRREV_B16] =3D "v_pk_lshrrev_b16", + [GFX9_V_PK_ASHRREV_I16] =3D "v_pk_ashrrev_i16", + [GFX9_V_PK_MAX_I16] =3D "v_pk_max_i16", + [GFX9_V_PK_MIN_I16] =3D "v_pk_min_i16", + [GFX9_V_PK_MAD_U16] =3D "v_pk_mad_u16", + [GFX9_V_PK_ADD_U16] =3D "v_pk_add_u16", + [GFX9_V_PK_SUB_U16] =3D "v_pk_sub_u16", + [GFX9_V_PK_MAX_U16] =3D "v_pk_max_u16", + [GFX9_V_PK_MIN_U16] =3D "v_pk_min_u16", + [GFX9_V_PK_FMA_F16] =3D "v_pk_fma_f16", + [GFX9_V_PK_ADD_F16] =3D "v_pk_add_f16", + [GFX9_V_PK_MUL_F16] =3D "v_pk_mul_f16", + [GFX9_V_PK_MIN_F16] =3D "v_pk_min_f16", + [GFX9_V_PK_MAX_F16] =3D "v_pk_max_f16", + [GFX9_V_MAD_MIX_F32] =3D "v_mad_mix_f32", + [GFX9_V_MAD_MIXLO_F16] =3D "v_mad_mixlo_f16", + [GFX9_V_MAD_MIXHI_F16] =3D "v_mad_mixhi_f16", + }, + [AMDGCN_INSN_TYPE_MUBUF] =3D { + [GFX9_BUFFER_LOAD_FORMAT_X] =3D "buffer_load_format_x", + [GFX9_BUFFER_LOAD_FORMAT_XY] =3D "buffer_load_format_xy", + [GFX9_BUFFER_LOAD_FORMAT_XYZ] =3D "buffer_load_format_xyz", + [GFX9_BUFFER_LOAD_FORMAT_XYZW] =3D "buffer_load_format_xyzw", + [GFX9_BUFFER_STORE_FORMAT_X] =3D "buffer_store_format_x", + [GFX9_BUFFER_STORE_FORMAT_XY] =3D "buffer_store_format_xy", + [GFX9_BUFFER_STORE_FORMAT_XYZ] =3D "buffer_store_format_xyz", + [GFX9_BUFFER_STORE_FORMAT_XYZW] =3D "buffer_store_format_xyzw", + [GFX9_BUFFER_LOAD_FORMAT_D16_X] =3D "buffer_load_format_d16_x", + [GFX9_BUFFER_LOAD_FORMAT_D16_XY] =3D "buffer_load_format_d16_xy", + [GFX9_BUFFER_LOAD_FORMAT_D16_XYZ] =3D "buffer_load_format_d16_xyz", + [GFX9_BUFFER_LOAD_FORMAT_D16_XYZW] =3D "buffer_load_format_d16_xyzw", + [GFX9_BUFFER_STORE_FORMAT_D16_X] =3D "buffer_store_format_d16_x", + [GFX9_BUFFER_STORE_FORMAT_D16_XY] =3D "buffer_store_format_d16_xy", + [GFX9_BUFFER_STORE_FORMAT_D16_XYZ] =3D "buffer_store_format_d16_xyz", + [GFX9_BUFFER_STORE_FORMAT_D16_XYZW] =3D "buffer_store_format_d16_xyzw", + [GFX9_BUFFER_LOAD_UBYTE] =3D "buffer_load_ubyte", + [GFX9_BUFFER_LOAD_SBYTE] =3D "buffer_load_sbyte", + [GFX9_BUFFER_LOAD_USHORT] =3D "buffer_load_ushort", + [GFX9_BUFFER_LOAD_SSHORT] =3D "buffer_load_sshort", + [GFX9_BUFFER_LOAD_DWORD] =3D "buffer_load_dword", + [GFX9_BUFFER_LOAD_DWORDX2] =3D "buffer_load_dwordx2", + [GFX9_BUFFER_LOAD_DWORDX3] =3D "buffer_load_dwordx3", + [GFX9_BUFFER_LOAD_DWORDX4] =3D "buffer_load_dwordx4", + [GFX9_BUFFER_STORE_BYTE] =3D "buffer_store_byte", + [GFX9_BUFFER_STORE_BYTE_D16_HI] =3D "buffer_store_byte_d16_hi", + [GFX9_BUFFER_STORE_SHORT] =3D "buffer_store_short", + [GFX9_BUFFER_STORE_SHORT_D16_HI] =3D "buffer_store_short_d16_hi", + [GFX9_BUFFER_STORE_DWORD] =3D "buffer_store_dword", + [GFX9_BUFFER_STORE_DWORDX2] =3D "buffer_store_dwordx2", + [GFX9_BUFFER_STORE_DWORDX3] =3D "buffer_store_dwordx3", + [GFX9_BUFFER_STORE_DWORDX4] =3D "buffer_store_dwordx4", + [GFX9_BUFFER_LOAD_UBYTE_D16] =3D "buffer_load_ubyte_d16", + [GFX9_BUFFER_LOAD_UBYTE_D16_HI] =3D "buffer_load_ubyte_d16_hi", + [GFX9_BUFFER_LOAD_SBYTE_D16] =3D "buffer_load_sbyte_d16", + [GFX9_BUFFER_LOAD_SBYTE_D16_HI] =3D "buffer_load_sbyte_d16_hi", + [GFX9_BUFFER_LOAD_SHORT_D16] =3D "buffer_load_short_d16", + [GFX9_BUFFER_LOAD_SHORT_D16_HI] =3D "buffer_load_short_d16_hi", + [GFX9_BUFFER_LOAD_FORMAT_D16_HI_X] =3D "buffer_load_format_d16_hi_x", + [GFX9_BUFFER_STORE_FORMAT_D16_HI_X] =3D "buffer_store_format_d16_hi_x", + [GFX9_BUFFER_STORE_LDS_DWORD] =3D "buffer_store_lds_dword", + [GFX9_BUFFER_WBINVL1] =3D "buffer_wbinvl1", + [GFX9_BUFFER_WBINVL1_VOL] =3D "buffer_wbinvl1_vol", + [GFX9_BUFFER_ATOMIC_SWAP] =3D "buffer_atomic_swap", + [GFX9_BUFFER_ATOMIC_CMPSWAP] =3D "buffer_atomic_cmpswap", + [GFX9_BUFFER_ATOMIC_ADD] =3D "buffer_atomic_add", + [GFX9_BUFFER_ATOMIC_SUB] =3D "buffer_atomic_sub", + [GFX9_BUFFER_ATOMIC_SMIN] =3D "buffer_atomic_smin", + [GFX9_BUFFER_ATOMIC_UMIN] =3D "buffer_atomic_umin", + [GFX9_BUFFER_ATOMIC_SMAX] =3D "buffer_atomic_smax", + [GFX9_BUFFER_ATOMIC_UMAX] =3D "buffer_atomic_umax", + [GFX9_BUFFER_ATOMIC_AND] =3D "buffer_atomic_and", + [GFX9_BUFFER_ATOMIC_OR] =3D "buffer_atomic_or", + [GFX9_BUFFER_ATOMIC_XOR] =3D "buffer_atomic_xor", + [GFX9_BUFFER_ATOMIC_INC] =3D "buffer_atomic_inc", + [GFX9_BUFFER_ATOMIC_DEC] =3D "buffer_atomic_dec", + [GFX9_BUFFER_ATOMIC_SWAP_X2] =3D "buffer_atomic_swap_x2", + [GFX9_BUFFER_ATOMIC_CMPSWAP_X2] =3D "buffer_atomic_cmpswap_x2", + [GFX9_BUFFER_ATOMIC_ADD_X2] =3D "buffer_atomic_add_x2", + [GFX9_BUFFER_ATOMIC_SUB_X2] =3D "buffer_atomic_sub_x2", + [GFX9_BUFFER_ATOMIC_SMIN_X2] =3D "buffer_atomic_smin_x2", + [GFX9_BUFFER_ATOMIC_UMIN_X2] =3D "buffer_atomic_umin_x2", + [GFX9_BUFFER_ATOMIC_SMAX_X2] =3D "buffer_atomic_smax_x2", + [GFX9_BUFFER_ATOMIC_UMAX_X2] =3D "buffer_atomic_umax_x2", + [GFX9_BUFFER_ATOMIC_AND_X2] =3D "buffer_atomic_and_x2", + [GFX9_BUFFER_ATOMIC_OR_X2] =3D "buffer_atomic_or_x2", + [GFX9_BUFFER_ATOMIC_XOR_X2] =3D "buffer_atomic_xor_x2", + [GFX9_BUFFER_ATOMIC_INC_X2] =3D "buffer_atomic_inc_x2", + [GFX9_BUFFER_ATOMIC_DEC_X2] =3D "buffer_atomic_dec_x2", + }, + [AMDGCN_INSN_TYPE_FLAT] =3D { + [GFX9_GLOBAL_LOAD_UBYTE] =3D "global_load_ubyte", + [GFX9_GLOBAL_LOAD_SBYTE] =3D "global_load_sbyte", + [GFX9_GLOBAL_LOAD_USHORT] =3D "global_load_ushort", + [GFX9_GLOBAL_LOAD_SSHORT] =3D "global_load_sshort", + [GFX9_GLOBAL_LOAD_DWORD] =3D "global_load_dword", + [GFX9_GLOBAL_LOAD_DWORDX2] =3D "global_load_dwordx2", + [GFX9_GLOBAL_LOAD_DWORDX3] =3D "global_load_dwordx3", + [GFX9_GLOBAL_LOAD_DWORDX4] =3D "global_load_dwordx4", + [GFX9_GLOBAL_STORE_BYTE] =3D "global_store_byte", + [GFX9_GLOBAL_STORE_BYTE_D16_HI] =3D "global_store_byte_d16_hi", + [GFX9_GLOBAL_STORE_SHORT] =3D "global_store_short", + [GFX9_GLOBAL_STORE_SHORT_D16_HI] =3D "global_store_short_d16_hi", + [GFX9_GLOBAL_STORE_DWORD] =3D "global_store_dword", + [GFX9_GLOBAL_STORE_DWORDX2] =3D "global_store_dwordx2", + [GFX9_GLOBAL_STORE_DWORDX3] =3D "global_store_dwordx3", + [GFX9_GLOBAL_STORE_DWORDX4] =3D "global_store_dwordx4", + [GFX9_GLOBAL_LOAD_UBYTE_D16] =3D "global_load_ubyte_d16", + [GFX9_GLOBAL_LOAD_UBYTE_D16_HI] =3D "global_load_ubyte_d16_hi", + [GFX9_GLOBAL_LOAD_SBYTE_D16] =3D "global_load_sbyte_d16", + [GFX9_GLOBAL_LOAD_SBYTE_D16_HI] =3D "global_load_sbyte_d16_hi", + [GFX9_GLOBAL_LOAD_SHORT_D16] =3D "global_load_short_d16", + [GFX9_GLOBAL_LOAD_SHORT_D16_HI] =3D "global_load_short_d16_hi", + [GFX9_GLOBAL_ATOMIC_SWAP] =3D "global_atomic_swap", + [GFX9_GLOBAL_ATOMIC_CMPSWAP] =3D "global_atomic_cmpswap", + [GFX9_GLOBAL_ATOMIC_ADD] =3D "global_atomic_add", + [GFX9_GLOBAL_ATOMIC_SUB] =3D "global_atomic_sub", + [GFX9_GLOBAL_ATOMIC_SMIN] =3D "global_atomic_smin", + [GFX9_GLOBAL_ATOMIC_UMIN] =3D "global_atomic_umin", + [GFX9_GLOBAL_ATOMIC_SMAX] =3D "global_atomic_smax", + [GFX9_GLOBAL_ATOMIC_UMAX] =3D "global_atomic_umax", + [GFX9_GLOBAL_ATOMIC_AND] =3D "global_atomic_and", + [GFX9_GLOBAL_ATOMIC_OR] =3D "global_atomic_or", + [GFX9_GLOBAL_ATOMIC_XOR] =3D "global_atomic_xor", + [GFX9_GLOBAL_ATOMIC_INC] =3D "global_atomic_inc", + [GFX9_GLOBAL_ATOMIC_DEC] =3D "global_atomic_dec", + [GFX9_GLOBAL_ATOMIC_SWAP_X2] =3D "global_atomic_swap_x2", + [GFX9_GLOBAL_ATOMIC_CMPSWAP_X2] =3D "global_atomic_cmpswap_x2", + [GFX9_GLOBAL_ATOMIC_ADD_X2] =3D "global_atomic_add_x2", + [GFX9_GLOBAL_ATOMIC_SUB_X2] =3D "global_atomic_sub_x2", + [GFX9_GLOBAL_ATOMIC_SMIN_X2] =3D "global_atomic_smin_x2", + [GFX9_GLOBAL_ATOMIC_UMIN_X2] =3D "global_atomic_umin_x2", + [GFX9_GLOBAL_ATOMIC_SMAX_X2] =3D "global_atomic_smax_x2", + [GFX9_GLOBAL_ATOMIC_UMAX_X2] =3D "global_atomic_umax_x2", + [GFX9_GLOBAL_ATOMIC_AND_X2] =3D "global_atomic_and_x2", + [GFX9_GLOBAL_ATOMIC_OR_X2] =3D "global_atomic_or_x2", + [GFX9_GLOBAL_ATOMIC_XOR_X2] =3D "global_atomic_xor_x2", + [GFX9_GLOBAL_ATOMIC_INC_X2] =3D "global_atomic_inc_x2", + [GFX9_GLOBAL_ATOMIC_DEC_X2] =3D "global_atomic_dec_x2", + }, + [AMDGCN_INSN_TYPE_DS] =3D { + /* DS opcode names (Vega ISA 12.13, opcodes 0-255). The DS + * table was entirely unpopulated, so DS instructions + * disassembled with a blank mnemonic. Enum values were + * ISA-verified (one known value bug at DS_CONDXCHG32_RTN_B64 + * is tracked separately). + */ + [GFX9_DS_ADD_U32] =3D "ds_add_u32", + [GFX9_DS_SUB_U32] =3D "ds_sub_u32", + [GFX9_DS_RSUB_U32] =3D "ds_rsub_u32", + [GFX9_DS_INC_U32] =3D "ds_inc_u32", + [GFX9_DS_DEC_U32] =3D "ds_dec_u32", + [GFX9_DS_MIN_I32] =3D "ds_min_i32", + [GFX9_DS_MAX_I32] =3D "ds_max_i32", + [GFX9_DS_MIN_U32] =3D "ds_min_u32", + [GFX9_DS_MAX_U32] =3D "ds_max_u32", + [GFX9_DS_AND_B32] =3D "ds_and_b32", + [GFX9_DS_OR_B32] =3D "ds_or_b32", + [GFX9_DS_XOR_B32] =3D "ds_xor_b32", + [GFX9_DS_MSKOR_B32] =3D "ds_mskor_b32", + [GFX9_DS_WRITE_B32] =3D "ds_write_b32", + [GFX9_DS_WRITE2_B32] =3D "ds_write2_b32", + [GFX9_DS_WRITE2ST64_B32] =3D "ds_write2st64_b32", + [GFX9_DS_CMPST_B32] =3D "ds_cmpst_b32", + [GFX9_DS_CMPST_F32] =3D "ds_cmpst_f32", + [GFX9_DS_MIN_F32] =3D "ds_min_f32", + [GFX9_DS_MAX_F32] =3D "ds_max_f32", + [GFX9_DS_NOP] =3D "ds_nop", + [GFX9_DS_ADD_F32] =3D "ds_add_f32", + [GFX9_DS_WRITE_ADDTID_B32] =3D "ds_write_addtid_b32", + [GFX9_DS_WRITE_B8] =3D "ds_write_b8", + [GFX9_DS_WRITE_B16] =3D "ds_write_b16", + [GFX9_DS_ADD_RTN_U32] =3D "ds_add_rtn_u32", + [GFX9_DS_SUB_RTN_U32] =3D "ds_sub_rtn_u32", + [GFX9_DS_RSUB_RTN_U32] =3D "ds_rsub_rtn_u32", + [GFX9_DS_INC_RTN_U32] =3D "ds_inc_rtn_u32", + [GFX9_DS_DEC_RTN_U32] =3D "ds_dec_rtn_u32", + [GFX9_DS_MIN_RTN_I32] =3D "ds_min_rtn_i32", + [GFX9_DS_MAX_RTN_I32] =3D "ds_max_rtn_i32", + [GFX9_DS_MIN_RTN_U32] =3D "ds_min_rtn_u32", + [GFX9_DS_MAX_RTN_U32] =3D "ds_max_rtn_u32", + [GFX9_DS_AND_RTN_B32] =3D "ds_and_rtn_b32", + [GFX9_DS_OR_RTN_B32] =3D "ds_or_rtn_b32", + [GFX9_DS_XOR_RTN_B32] =3D "ds_xor_rtn_b32", + [GFX9_DS_MSKOR_RTN_B32] =3D "ds_mskor_rtn_b32", + [GFX9_DS_WRXCHG_RTN_B32] =3D "ds_wrxchg_rtn_b32", + [GFX9_DS_WRXCHG2_RTN_B32] =3D "ds_wrxchg2_rtn_b32", + [GFX9_DS_WRXCHG2ST64_RTN_B32] =3D "ds_wrxchg2st64_rtn_b32", + [GFX9_DS_CMPST_RTN_B32] =3D "ds_cmpst_rtn_b32", + [GFX9_DS_CMPST_RTN_F32] =3D "ds_cmpst_rtn_f32", + [GFX9_DS_MIN_RTN_F32] =3D "ds_min_rtn_f32", + [GFX9_DS_MAX_RTN_F32] =3D "ds_max_rtn_f32", + [GFX9_DS_WRAP_RTN_B32] =3D "ds_wrap_rtn_b32", + [GFX9_DS_ADD_RTN_F32] =3D "ds_add_rtn_f32", + [GFX9_DS_READ_B32] =3D "ds_read_b32", + [GFX9_DS_READ2_B32] =3D "ds_read2_b32", + [GFX9_DS_READ2ST64_B32] =3D "ds_read2st64_b32", + [GFX9_DS_READ_I8] =3D "ds_read_i8", + [GFX9_DS_READ_U8] =3D "ds_read_u8", + [GFX9_DS_READ_I16] =3D "ds_read_i16", + [GFX9_DS_READ_U16] =3D "ds_read_u16", + [GFX9_DS_SWIZZLE_B32] =3D "ds_swizzle_b32", + [GFX9_DS_PERMUTE_B32] =3D "ds_permute_b32", + [GFX9_DS_BPERMUTE_B32] =3D "ds_bpermute_b32", + [GFX9_DS_ADD_U64] =3D "ds_add_u64", + [GFX9_DS_SUB_U64] =3D "ds_sub_u64", + [GFX9_DS_RSUB_U64] =3D "ds_rsub_u64", + [GFX9_DS_INC_U64] =3D "ds_inc_u64", + [GFX9_DS_DEC_U64] =3D "ds_dec_u64", + [GFX9_DS_MIN_I64] =3D "ds_min_i64", + [GFX9_DS_MAX_I64] =3D "ds_max_i64", + [GFX9_DS_MIN_U64] =3D "ds_min_u64", + [GFX9_DS_MAX_U64] =3D "ds_max_u64", + [GFX9_DS_AND_B64] =3D "ds_and_b64", + [GFX9_DS_OR_B64] =3D "ds_or_b64", + [GFX9_DS_XOR_B64] =3D "ds_xor_b64", + [GFX9_DS_MSKOR_B64] =3D "ds_mskor_b64", + [GFX9_DS_WRITE_B64] =3D "ds_write_b64", + [GFX9_DS_WRITE2_B64] =3D "ds_write2_b64", + [GFX9_DS_WRITE2ST64_B64] =3D "ds_write2st64_b64", + [GFX9_DS_CMPST_B64] =3D "ds_cmpst_b64", + [GFX9_DS_CMPST_F64] =3D "ds_cmpst_f64", + [GFX9_DS_MIN_F64] =3D "ds_min_f64", + [GFX9_DS_MAX_F64] =3D "ds_max_f64", + [GFX9_DS_WRITE_B8_D16_HI] =3D "ds_write_b8_d16_hi", + [GFX9_DS_WRITE_B16_D16_HI] =3D "ds_write_b16_d16_hi", + [GFX9_DS_READ_U8_D16] =3D "ds_read_u8_d16", + [GFX9_DS_READ_U8_D16_HI] =3D "ds_read_u8_d16_hi", + [GFX9_DS_READ_I8_D16] =3D "ds_read_i8_d16", + [GFX9_DS_READ_I8_D16_HI] =3D "ds_read_i8_d16_hi", + [GFX9_DS_READ_U16_D16] =3D "ds_read_u16_d16", + [GFX9_DS_READ_U16_D16_HI] =3D "ds_read_u16_d16_hi", + [GFX9_DS_ADD_RTN_U64] =3D "ds_add_rtn_u64", + [GFX9_DS_SUB_RTN_U64] =3D "ds_sub_rtn_u64", + [GFX9_DS_RSUB_RTN_U64] =3D "ds_rsub_rtn_u64", + [GFX9_DS_INC_RTN_U64] =3D "ds_inc_rtn_u64", + [GFX9_DS_DEC_RTN_U64] =3D "ds_dec_rtn_u64", + [GFX9_DS_MIN_RTN_I64] =3D "ds_min_rtn_i64", + [GFX9_DS_MAX_RTN_I64] =3D "ds_max_rtn_i64", + [GFX9_DS_MIN_RTN_U64] =3D "ds_min_rtn_u64", + [GFX9_DS_MAX_RTN_U64] =3D "ds_max_rtn_u64", + [GFX9_DS_AND_RTN_B64] =3D "ds_and_rtn_b64", + [GFX9_DS_OR_RTN_B64] =3D "ds_or_rtn_b64", + [GFX9_DS_XOR_RTN_B64] =3D "ds_xor_rtn_b64", + [GFX9_DS_MSKOR_RTN_B64] =3D "ds_mskor_rtn_b64", + [GFX9_DS_WRXCHG_RTN_B64] =3D "ds_wrxchg_rtn_b64", + [GFX9_DS_WRXCHG2_RTN_B64] =3D "ds_wrxchg2_rtn_b64", + [GFX9_DS_WRXCHG2ST64_RTN_B64] =3D "ds_wrxchg2st64_rtn_b64", + [GFX9_DS_CMPST_RTN_B64] =3D "ds_cmpst_rtn_b64", + [GFX9_DS_CMPST_RTN_F64] =3D "ds_cmpst_rtn_f64", + [GFX9_DS_MIN_RTN_F64] =3D "ds_min_rtn_f64", + [GFX9_DS_MAX_RTN_F64] =3D "ds_max_rtn_f64", + [GFX9_DS_READ_B64] =3D "ds_read_b64", + [GFX9_DS_READ2_B64] =3D "ds_read2_b64", + [GFX9_DS_CONDXCHG32_RTN_B64] =3D "ds_condxchg32_rtn_b64", + [GFX9_DS_ADD_SRC2_U32] =3D "ds_add_src2_u32", + [GFX9_DS_SUB_SRC2_U32] =3D "ds_sub_src2_u32", + [GFX9_DS_RSUB_SRC2_U32] =3D "ds_rsub_src2_u32", + [GFX9_DS_INC_SRC2_U32] =3D "ds_inc_src2_u32", + [GFX9_DS_DEC_SRC2_U32] =3D "ds_dec_src2_u32", + [GFX9_DS_MIN_SRC2_I32] =3D "ds_min_src2_i32", + [GFX9_DS_MAX_SRC2_I32] =3D "ds_max_src2_i32", + [GFX9_DS_MIN_SRC2_U32] =3D "ds_min_src2_u32", + [GFX9_DS_MAX_SRC2_U32] =3D "ds_max_src2_u32", + [GFX9_DS_AND_SRC2_B32] =3D "ds_and_src2_b32", + [GFX9_DS_OR_SRC2_B32] =3D "ds_or_src2_b32", + [GFX9_DS_XOR_SRC2_B32] =3D "ds_xor_src2_b32", + [GFX9_DS_WRITE_SRC2_B32] =3D "ds_write_src2_b32", + [GFX9_DS_MIN_SRC2_F32] =3D "ds_min_src2_f32", + [GFX9_DS_MAX_SRC2_F32] =3D "ds_max_src2_f32", + [GFX9_DS_ADD_SRC2_F32] =3D "ds_add_src2_f32", + [GFX9_DS_GWS_SEMA_RELEASE_ALL] =3D "ds_gws_sema_release_all", + [GFX9_DS_ADD_SRC2_U64] =3D "ds_add_src2_u64", + [GFX9_DS_SUB_SRC2_U64] =3D "ds_sub_src2_u64", + [GFX9_DS_RSUB_SRC2_U64] =3D "ds_rsub_src2_u64", + [GFX9_DS_INC_SRC2_U64] =3D "ds_inc_src2_u64", + [GFX9_DS_DEC_SRC2_U64] =3D "ds_dec_src2_u64", + [GFX9_DS_MIN_SRC2_I64] =3D "ds_min_src2_i64", + [GFX9_DS_MAX_SRC2_I64] =3D "ds_max_src2_i64", + [GFX9_DS_MIN_SRC2_U64] =3D "ds_min_src2_u64", + [GFX9_DS_MAX_SRC2_U64] =3D "ds_max_src2_u64", + [GFX9_DS_AND_SRC2_B64] =3D "ds_and_src2_b64", + [GFX9_DS_OR_SRC2_B64] =3D "ds_or_src2_b64", + [GFX9_DS_XOR_SRC2_B64] =3D "ds_xor_src2_b64", + [GFX9_DS_WRITE_SRC2_B64] =3D "ds_write_src2_b64", + [GFX9_DS_MIN_SRC2_F64] =3D "ds_min_src2_f64", + [GFX9_DS_MAX_SRC2_F64] =3D "ds_max_src2_f64", + [GFX9_DS_WRITE_B96] =3D "ds_write_b96", + [GFX9_DS_WRITE_B128] =3D "ds_write_b128", + [GFX9_DS_READ_B96] =3D "ds_read_b96", + [GFX9_DS_READ_B128] =3D "ds_read_b128", + }, +}; + +struct amdgcn_insn { + union { + union amdgcn_gfx10_insn gfx10; + union amdgcn_gfx9_insn gfx9; + }; + u32 size; + u32 idx; + enum amdgcn_insn_type type; +}; + +static inline void emit_s_load_dwordx2(int version, struct amdgcn_insn *in= sn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int offset) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_load_dwordx2(&insn->gfx10, dst, + src, + offset); + insn->type =3D AMDGCN_INSN_TYPE_SMEM; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_load_dwordx2(&insn->gfx9, dst, + src, + offset); + insn->type =3D AMDGCN_INSN_TYPE_SMEM; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_load_dwordx2_soff(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + int offset, u8 soffset) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_load_dwordx2(&insn->gfx10, dst, + src, offset); + insn->gfx10.smem.soffset =3D soffset; + insn->type =3D AMDGCN_INSN_TYPE_SMEM; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_load_dwordx2(&insn->gfx9, dst, + src, offset); + insn->gfx9.smem.soe =3D 1; + insn->gfx9.smem.soffset =3D soffset; + insn->type =3D AMDGCN_INSN_TYPE_SMEM; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_lshl_b32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_lshl_b32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_lshl_b32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_bfe_i32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(knod_param_is_literal(src0) || + knod_param_is_literal(src1) || + knod_param_is_literal(src2)); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_bfe_i32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_bfe_i32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_bfe_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(knod_param_is_literal(src0) || + knod_param_is_literal(src1) || + knod_param_is_literal(src2)); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_bfe_u32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_bfe_u32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_bfi_b32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(knod_param_is_literal(src0) || + knod_param_is_literal(src1) || + knod_param_is_literal(src2)); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_bfi_b32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_bfi_b32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshl_add_u32(int version, struct amdgcn_insn *in= sn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_lshl_add_u32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_lshl_add_u32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshl_or_b32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_lshl_or_b32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_lshl_or_b32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_alignbit_b32(int version, struct amdgcn_insn *in= sn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_alignbit_b32(&insn->gfx10, + dst, src0, + src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_alignbit_b32(&insn->gfx9, + dst, src0, + src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_perm_b32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(knod_param_is_literal(src0) || + knod_param_is_literal(src1) || + knod_param_is_literal(src2)); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_perm_b32(&insn->gfx10, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_perm_b32(&insn->gfx9, + dst, src0, src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mad_u64_u32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param64 dst, + struct amdgcn_param32 dst2, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param64 src2) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mad_u64_u32(&insn->gfx10, dst, + dst2, src0, + src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3B; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mad_u64_u32(&insn->gfx9, dst, + dst2, src0, + src1, src2); + insn->type =3D AMDGCN_INSN_TYPE_VOP3B; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_mov_b32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, struct amdgcn_param32 src) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_mov_b32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_mov_b32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mov_b32_e32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mov_b32_e32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mov_b32_e32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_readfirstlane_b32(int version, + struct amdgcn_insn *insn, + u8 sdst, u8 vsrc) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_readfirstlane_b32(&insn->gfx10, + sdst, vsrc); + insn->type =3D AMDGCN_INSN_TYPE_VOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_readfirstlane_b32(&insn->gfx9, + sdst, vsrc); + insn->type =3D AMDGCN_INSN_TYPE_VOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_add_co_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_add_co_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3B; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_add_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_add_co_ci_u32_e32(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_add_co_ci_u32_e32(&insn->gfx10, + dst, src0, + src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_addc_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +/* No carry in/out */ +static inline void emit_v_add_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_add_nc_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_add_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +/* No carry in/out */ +static inline void emit_v_sub_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_sub_nc_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_sub_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_xor_b32_e32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_xor_b32_e32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_xor_b32_e32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_or_b32_e32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_or_b32_e32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_or_b32_e32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cndmask_b32_e32(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_cndmask_b32_e32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_cndmask_b32_e32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_and_b32_e32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_and_b32_e32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_and_b32_e32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_sub_co_ci_u32_e32(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_sub_co_ci_u32_e32(&insn->gfx10, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_subb_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_subrev_co_ci_u32_e32(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_subrev_co_ci_u32_e32(&insn->gfx10, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_subbrev_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_sub_co_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_sub_co_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3B; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_sub_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_subrev_co_u32(int version, struct amdgcn_insn *i= nsn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_subrev_co_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3B; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_subrev_co_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mul_lo_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mul_lo_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mul_lo_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mbcnt_lo_u32_b32(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mbcnt_lo_u32_b32(&insn->gfx10, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mbcnt_lo_u32_b32(&insn->gfx9, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mbcnt_hi_u32_b32(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mbcnt_hi_u32_b32(&insn->gfx10, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mbcnt_hi_u32_b32(&insn->gfx9, + dst, src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_mul_hi_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_mul_hi_u32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_mul_hi_u32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshlrev_b64(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* D.u64 =3D S1.u64 << S0.u[5:0]. */ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_lshlrev_b64(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_lshlrev_b64(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshrrev_b64(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + WARN_ON(src1.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_lshrrev_b64(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_lshrrev_b64(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_ashrrev_i64(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_ashrrev_i64(&insn->gfx10, dst, src0, + src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_ashrrev_i64(&insn->gfx9, dst, src0, + src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_ashrrev_i32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_ashrrev_i32(&insn->gfx10, dst, src0, + src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_ashrrev_i32(&insn->gfx9, dst, src0, + src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP3A; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshlrev_b32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_lshlrev_b32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_lshlrev_b32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_lshrrev_b32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + if (version =3D=3D 10) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx10_v_lshrrev_b32(&insn->gfx10, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->size =3D emit_gfx9_v_lshrrev_b32(&insn->gfx9, dst, + src0, src1); + insn->type =3D AMDGCN_INSN_TYPE_VOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_eq_u64(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_eq_u64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_eq_u64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_eq_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_eq_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_eq_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_gt_u64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_gt_u64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_gt_u64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_gt_i64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_gt_i64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_gt_i64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_ge_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_ge_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_ge_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_gt_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_gt_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_gt_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_lt_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_lt_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_lt_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_le_u32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_le_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_le_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_gt_i32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_gt_i32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_gt_i32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_ge_i32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_ge_i32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_ge_i32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_lt_i32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_lt_i32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_lt_i32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_le_i32(int version, struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_le_i32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_le_i32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +/* EXEC &=3D (src0 < src1), per-lane mask update */ +static inline void emit_v_cmpx_lt_u32(int version, struct amdgcn_insn *ins= n, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + WARN_ON_ONCE(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmpx_lt_u32(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmpx_lt_u32(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_ge_u64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_ge_u64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_ge_u64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_ge_i64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_ge_i64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_ge_i64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_lt_u64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_lt_u64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_lt_u64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_lt_i64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_lt_i64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_lt_i64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_le_u64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_le_u64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_le_u64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_v_cmp_le_i64(int version, struct amdgcn_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + WARN_ON_ONCE(dst.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_v_cmp_le_i64(&insn->gfx10, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_v_cmp_le_i64(&insn->gfx9, dst, src); + insn->type =3D AMDGCN_INSN_TYPE_VOPC; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_buffer_load_ubyte(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_buffer_load_ubyte(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_buffer_load_ubyte(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_buffer_load_ushort(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_buffer_load_ushort(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_buffer_load_ushort(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_buffer_load_dword(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_buffer_load_dword(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_buffer_load_dword(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_buffer_load_dwordx2(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_buffer_load_dwordx2(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_buffer_load_dwordx2(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_buffer_load_dwordx4(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_buffer_load_dwordx4(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_buffer_load_dwordx4(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_MUBUF; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_load_ubyte(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_load_ubyte(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_load_ubyte(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_load_ushort(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_load_ushort(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_load_ushort(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_load_dword(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_load_dword(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_load_dword(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_load_dwordx2(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_load_dwordx2(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_load_dwordx2(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_load_dwordx4(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_load_dwordx4(&insn->gfx10, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_load_dwordx4(&insn->gfx9, + dst, src, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_store_byte(int version, struct amdgcn_insn = *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_store_byte(&insn->gfx10, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_store_byte(&insn->gfx9, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +#define DEFINE_EMIT_GLOBAL_ATOMIC(name) \ +static inline void emit_global_atomic_##name(int version, \ + struct amdgcn_insn *insn, \ + struct amdgcn_param32 vdst, \ + struct amdgcn_param32 addr, \ + struct amdgcn_param32 data, \ + int off, int glc) \ +{ \ + if (version =3D=3D 10) { \ + insn->size =3D emit_gfx10_global_atomic_##name( \ + &insn->gfx10, vdst, addr, data, off, glc); \ + insn->type =3D AMDGCN_INSN_TYPE_FLAT; \ + } else if (version =3D=3D 9) { \ + insn->size =3D emit_gfx9_global_atomic_##name( \ + &insn->gfx9, vdst, addr, data, off, glc); \ + insn->type =3D AMDGCN_INSN_TYPE_FLAT; \ + } else { \ + WARN_ON_ONCE(1); \ + } \ +} + +DEFINE_EMIT_GLOBAL_ATOMIC(add) +DEFINE_EMIT_GLOBAL_ATOMIC(and) +DEFINE_EMIT_GLOBAL_ATOMIC(or) +DEFINE_EMIT_GLOBAL_ATOMIC(xor) +DEFINE_EMIT_GLOBAL_ATOMIC(swap) +DEFINE_EMIT_GLOBAL_ATOMIC(cmpswap) +DEFINE_EMIT_GLOBAL_ATOMIC(add_x2) +DEFINE_EMIT_GLOBAL_ATOMIC(and_x2) +DEFINE_EMIT_GLOBAL_ATOMIC(or_x2) +DEFINE_EMIT_GLOBAL_ATOMIC(xor_x2) +DEFINE_EMIT_GLOBAL_ATOMIC(swap_x2) +DEFINE_EMIT_GLOBAL_ATOMIC(cmpswap_x2) + +static inline void emit_global_store_short(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_store_short(&insn->gfx10, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_store_short(&insn->gfx9, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_store_dword(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_store_dword(&insn->gfx10, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_store_dword(&insn->gfx9, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_store_dwordx2(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_store_dwordx2(&insn->gfx10, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_store_dwordx2(&insn->gfx9, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_global_store_dwordx4(int version, + struct amdgcn_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_global_store_dwordx4(&insn->gfx10, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_global_store_dwordx4(&insn->gfx9, + src, dst, off); + insn->type =3D AMDGCN_INSN_TYPE_FLAT; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_branch(int version, struct amdgcn_insn *insn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_branch(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_branch(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_cbranch_vccz(int version, struct amdgcn_insn *in= sn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_cbranch_vccz(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_cbranch_vccz(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_cbranch_vccnz(int version, struct amdgcn_insn *i= nsn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_cbranch_vccnz(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_cbranch_vccnz(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +/* Structurized CFG wrapper functions. + * Use raw SGPR indices; EXEC=3D126, VCC=3D106, integer_0=3D128. + */ + +static inline void emit_s_and_saveexec_b64(int version, + struct amdgcn_insn *insn, + u8 sdst, u8 ssrc) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_and_saveexec_b64(&insn->gfx10, + sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_and_saveexec_b64(&insn->gfx9, + sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_bcnt1_i32_b64(int version, struct amdgcn_insn *i= nsn, + u8 sdst, u8 ssrc) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_bcnt1_i32_b64(&insn->gfx10, + sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_bcnt1_i32_b64(&insn->gfx9, + sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_mov_b64(int version, struct amdgcn_insn *insn, + u8 sdst, u8 ssrc) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_mov_b64(&insn->gfx10, sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_mov_b64(&insn->gfx9, sdst, ssrc); + insn->type =3D AMDGCN_INSN_TYPE_SOP1; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_and_b64(int version, struct amdgcn_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_and_b64(&insn->gfx10, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_and_b64(&insn->gfx9, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_or_b64(int version, struct amdgcn_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_or_b64(&insn->gfx10, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_or_b64(&insn->gfx9, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_andn2_b64(int version, struct amdgcn_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_andn2_b64(&insn->gfx10, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_andn2_b64(&insn->gfx9, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_cbranch_execz(int version, struct amdgcn_insn *i= nsn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_cbranch_execz(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_cbranch_execz(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_cbranch_execnz(int version, struct amdgcn_insn *= insn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_cbranch_execnz(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_cbranch_execnz(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_sub_u32(int version, struct amdgcn_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_sub_u32(&insn->gfx10, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_sub_u32(&insn->gfx9, + sdst, ssrc0, ssrc1); + insn->type =3D AMDGCN_INSN_TYPE_SOP2; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_cbranch_scc0(int version, struct amdgcn_insn *in= sn, + short off) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_cbranch_scc0(&insn->gfx10, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_cbranch_scc0(&insn->gfx9, off); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_branch_fixup(int version, struct amdgcn_insn *insn, + short off) +{ + if (version =3D=3D 10) + insn->size =3D emit_gfx10_branch_fixup(&insn->gfx10, off); + else if (version =3D=3D 9) + insn->size =3D emit_gfx9_branch_fixup(&insn->gfx9, off); + else + WARN_ON_ONCE(1); +} + +static inline void emit_s_waitcnt_lgkmcnt(int version, struct amdgcn_insn = *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_waitcnt_lgkmcnt(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_waitcnt_lgkmcnt(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_waitcnt_vmcnt(int version, struct amdgcn_insn *i= nsn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_waitcnt_vmcnt(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_waitcnt_vmcnt(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_waitcnt_vmcnt_lgkmcnt(int version, + struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_waitcnt_vmcnt_lgkmcnt(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_waitcnt_vmcnt_lgkmcnt(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_nop(int version, struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_nop(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_nop(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_endpgm(int version, struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_endpgm(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_endpgm(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_code_end(int version, struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_code_end(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + WARN_ON_ONCE(1); + } else { + WARN_ON_ONCE(1); + } +} + +static inline void emit_s_icache_inv(int version, struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) { + insn->size =3D emit_gfx10_s_icache_inv(&insn->gfx10); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else if (version =3D=3D 9) { + insn->size =3D emit_gfx9_s_icache_inv(&insn->gfx9); + insn->type =3D AMDGCN_INSN_TYPE_SOPP; + } else { + WARN_ON_ONCE(1); + } +} + +static inline void gfx10_debug_vop3a(union amdgcn_gfx10_insn *insn) +{ + char src0[20]; + char src1[20]; + char src2[20]; + + if (insn->vop3a.src0 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3a.src0); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3a.src0 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3a.src0 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src0 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3a.src0 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src0 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3a.src0 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src1 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3a.src1); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3a.src1 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3a.src1 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src1 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3a.src1 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src1 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3a.src1 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src2 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3a.src2); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3a.src2 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3a.src2 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src2 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3a.src2 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src2 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3a.src2 - GFX10_VOP3A_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3A][insn->vop3a.op], + insn->vop3a.vdst, src0, src1, src2); +} + +static inline void gfx10_debug_vop3b(union amdgcn_gfx10_insn *insn) +{ + char src0[20]; + char src1[20]; + char src2[20]; + char sdst[20]; + + if (insn->vop3b.sdst < GFX10_VOP3A_SRC_VCC_LO) + sprintf(sdst, "s%d", insn->vop3b.sdst); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(sdst, "%s", "vcc_lo"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(sdst, "%s", "vcc_hi"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(sdst, "%s", "null"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(sdst, "%s", "exec_lo"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(sdst, "%s", "exec_hi"); + else if (insn->vop3b.sdst < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(sdst, "0x%x", + insn->vop3b.sdst - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.sdst < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(sdst, "-0x%x", + insn->vop3b.sdst - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(sdst, "%s", "vcc"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(sdst, "%s", "exec"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(sdst, "%s", "scc"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(sdst, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.sdst >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(sdst, "v%d", + insn->vop3b.sdst - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src0 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3b.src0); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3b.src0 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3b.src0 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src0 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3b.src0 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src0 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3b.src0 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src1 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3b.src1); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3b.src1 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3b.src1 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src1 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3b.src1 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src1 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3b.src1 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src2 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3b.src2); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3b.src2 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3b.src2 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src2 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3b.src2 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src2 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3b.src2 - GFX10_VOP3A_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3B][insn->vop3b.op], + insn->vop3b.vdst, sdst, src0, src1, src2); +} + +static inline void gfx10_debug_vop1(union amdgcn_gfx10_insn *insn) +{ + char src0[20]; + + if (insn->vop1.src0 < GFX10_VOP1_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop1.src0); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop1.src0 < GFX10_VOP1_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop1.src0 - GFX10_VOP1_SRC_INTEGER_0); + else if (insn->vop1.src0 < GFX10_VOP1_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop1.src0 - GFX10_VOP1_SRC_INTEGER_MINUS_1); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop1.literal); + else if (insn->vop1.src0 >=3D GFX10_VOP1_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop1.src0 - GFX10_VOP1_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP1][insn->vop1.op], + insn->vop1.vdst, src0); +} + +static inline void gfx10_debug_vopc(union amdgcn_gfx10_insn *insn) +{ + char src0[20]; + + if (insn->vopc.src0 < GFX10_VOPC_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vopc.src0); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vopc.src0 < GFX10_VOPC_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vopc.src0 - GFX10_VOPC_SRC_INTEGER_0); + else if (insn->vopc.src0 < GFX10_VOPC_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vopc.src0 - GFX10_VOPC_SRC_INTEGER_MINUS_1); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vopc.literal); + else if (insn->vopc.src0 >=3D GFX10_VOPC_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vopc.src0 - GFX10_VOPC_SRC_VGPR_BASE); + + pr_debug("knod_asm %s %s, v%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOPC][insn->vopc.op], + src0, insn->vopc.vsrc1); +} + +static inline void gfx10_debug_vop2(union amdgcn_gfx10_insn *insn) +{ + char src0[20]; + + if (insn->vop2.src0 < GFX10_VOP2_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop2.src0); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop2.src0 < GFX10_VOP2_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop2.src0 - GFX10_VOP2_SRC_INTEGER_0); + else if (insn->vop2.src0 < GFX10_VOP2_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop2.src0 - GFX10_VOP2_SRC_INTEGER_MINUS_1); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop2.literal); + else if (insn->vop2.src0 >=3D GFX10_VOP2_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop2.src0 - GFX10_VOP2_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s v%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP2][insn->vop2.op], + insn->vop2.vdst, + src0, + insn->vop2.vsrc1); +} + +static inline void gfx10_debug_sop1(union amdgcn_gfx10_insn *insn) +{ + char ssrc0[20]; + + if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "s%d", insn->sop1.ssrc0); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "%s", "vcc_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCC_HI) + sprintf(ssrc0, "%s", "vcc_hi"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_NULL) + sprintf(ssrc0, "%s", "null"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXEC_LO) + sprintf(ssrc0, "%s", "exec_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXEC_HI) + sprintf(ssrc0, "%s", "exec_hi"); + else if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_INTEGER_MINUS_1) + sprintf(ssrc0, "0x%x", + insn->sop1.ssrc0 - GFX10_SOP1_SSRC_INTEGER_0); + else if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_SHARED_BASE) + sprintf(ssrc0, "-0x%x", + insn->sop1.ssrc0 - GFX10_SOP1_SSRC_INTEGER_MINUS_1); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCCZ) + sprintf(ssrc0, "%s", "vcc"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXECZ) + sprintf(ssrc0, "%s", "exec"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_SCC) + sprintf(ssrc0, "%s", "scc"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_LITERAL_CONST) + sprintf(ssrc0, "0x%x", insn->sop1.literal); + + pr_debug("knod_asm %s s%d, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOP1][insn->sop1.op], + insn->sop1.sdst, + ssrc0); +} + +static inline void gfx10_debug_sopp(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm %s 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPP][insn->sopp.op], + insn->sopp.simm16); +} + +static inline void gfx10_debug_smem(union amdgcn_gfx10_insn *insn) +{ + pr_debug("%s s[%d:%d], s[%d:%d], 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SMEM][insn->smem.op], + insn->smem.sdata, + insn->smem.sdata + 1, + (insn->smem.sbase * 2), + (insn->smem.sbase * 2) + 1, + insn->smem.offset); +} + +static inline void gfx10_debug_mubuf(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, s[%d:%d], 0x%x offen offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_MUBUF][insn->mubuf.op], + insn->mubuf.vdata, + insn->mubuf.vaddr, + insn->mubuf.srsrc, + insn->mubuf.srsrc + 3, + insn->mubuf.soffset, + insn->mubuf.offset); +} + +static inline void gfx10_debug_global(union amdgcn_gfx10_insn *insn) +{ + if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_BYTE) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_SHORT) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORD) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORDX2) { + pr_debug("knod_asm %s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.data + 1, + insn->flat.offset); + } else { + pr_debug("knod_asm %s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.vdst, + insn->flat.vdst + 1, + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.offset); + } +} + +static inline void decode_ssrc8(int ssrc, u32 literal, char *buf) +{ + if (ssrc < 106) + sprintf(buf, "s%d", ssrc); + else if (ssrc =3D=3D 106) + sprintf(buf, "vcc_lo"); + else if (ssrc =3D=3D 107) + sprintf(buf, "vcc_hi"); + else if (ssrc =3D=3D 125) + sprintf(buf, "null"); + else if (ssrc =3D=3D 126) + sprintf(buf, "exec_lo"); + else if (ssrc =3D=3D 127) + sprintf(buf, "exec_hi"); + else if (ssrc < 193) + sprintf(buf, "0x%x", ssrc - 128); + else if (ssrc < 235) + sprintf(buf, "-0x%x", ssrc - 193); + else if (ssrc =3D=3D 251) + sprintf(buf, "vcc"); + else if (ssrc =3D=3D 252) + sprintf(buf, "exec"); + else if (ssrc =3D=3D 253) + sprintf(buf, "scc"); + else if (ssrc =3D=3D 255) + sprintf(buf, "0x%x", literal); + else + sprintf(buf, "?%d", ssrc); +} + +static inline void decode_vsrc9(int src, u32 literal, char *buf) +{ + if (src >=3D 256) + sprintf(buf, "v%d", src - 256); + else + decode_ssrc8(src, literal, buf); +} + +static inline void gfx10_debug_sop2(union amdgcn_gfx10_insn *insn) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sop2.ssrc0, insn->sop2.literal, ssrc0); + decode_ssrc8(insn->sop2.ssrc1, insn->sop2.literal, ssrc1); + pr_debug("knod_asm %s s%d, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOP2][insn->sop2.op], + insn->sop2.sdst, ssrc0, ssrc1); +} + +static inline void gfx10_debug_sopk(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm %s s%d, 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPK][insn->sopk.op], + insn->sopk.sdst, insn->sopk.simm16); +} + +static inline void gfx10_debug_sopc(union amdgcn_gfx10_insn *insn) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sopc.ssrc0, insn->sopc.literal, ssrc0); + decode_ssrc8(insn->sopc.ssrc1, insn->sopc.literal, ssrc1); + pr_debug("knod_asm %s %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPC][insn->sopc.op], + ssrc0, ssrc1); +} + +static inline void gfx10_debug_vop3p(union amdgcn_gfx10_insn *insn) +{ + char src0[20], src1[20], src2[20]; + + decode_vsrc9(insn->vop3p.src0, insn->vop3p.literal, src0); + decode_vsrc9(insn->vop3p.src1, insn->vop3p.literal, src1); + decode_vsrc9(insn->vop3p.src2, insn->vop3p.literal, src2); + pr_debug("knod_asm %s v%d, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3P][insn->vop3p.op], + (int)insn->vop3p.vdst, src0, src1, src2); +} + +static inline void gfx10_debug_sdwa(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm sdwa src0:%d dst_sel:%d src0_sel:%d src1_sel:%d\n", + insn->sdwa.src0, insn->sdwa.dst_sel, + insn->sdwa.src0_sel, insn->sdwa.src1_sel); +} + +static inline void gfx10_debug_sdwab(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm sdwab src0:%d sdst:s%d src0_sel:%d src1_sel:%d\n", + insn->sdwab.src0, insn->sdwab.sdst, + insn->sdwab.src0_sel, insn->sdwab.src1_sel); +} + +static inline void gfx10_debug_dpp16(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm dpp16 (not decoded)\n"); +} + +static inline void gfx10_debug_dpp8(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm dpp8 (not decoded)\n"); +} + +static inline void gfx10_debug_vintrp(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm vintrp (not decoded)\n"); +} + +static inline void gfx10_debug_ds(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, v%d, v%d offset0:%d offset1:%d%s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_DS][insn->ds.op], + (int)insn->ds.vdst, (int)insn->ds.addr, + (int)insn->ds.data0, (int)insn->ds.data1, + (int)insn->ds.offset0, (int)insn->ds.offset1, + insn->ds.gds ? " gds" : ""); +} + +static inline void gfx10_debug_mtbuf(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, s[%d:%d], s%d format:%d offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_MTBUF][insn->mtbuf.op], + (int)insn->mtbuf.vdata, (int)insn->mtbuf.vaddr, + (int)insn->mtbuf.srsrc * 4, (int)insn->mtbuf.srsrc * 4 + 3, + (int)insn->mtbuf.soffset, (int)insn->mtbuf.foamat, + (int)insn->mtbuf.offset); +} + +static inline void gfx10_debug_mimg(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm mimg (not decoded)\n"); +} + +static inline void gfx10_debug_exp(union amdgcn_gfx10_insn *insn) +{ + pr_debug("knod_asm exp (not decoded)\n"); +} + +static inline void gfx10_debug_insn(struct amdgcn_insn *insn) +{ + u32 *ptr; + + switch (insn->type) { + case AMDGCN_INSN_TYPE_SOP2: + gfx10_debug_sop2(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SOPK: + gfx10_debug_sopk(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SOP1: + gfx10_debug_sop1(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SOPC: + gfx10_debug_sopc(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SOPP: + gfx10_debug_sopp(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SMEM: + gfx10_debug_smem(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOP2: + gfx10_debug_vop2(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOP1: + gfx10_debug_vop1(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOPC: + gfx10_debug_vopc(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOP3A: + gfx10_debug_vop3a(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOP3B: + gfx10_debug_vop3b(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VOP3P: + gfx10_debug_vop3p(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SDWA: + gfx10_debug_sdwa(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_SDWAB: + gfx10_debug_sdwab(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_DPP16: + gfx10_debug_dpp16(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_DPP8: + gfx10_debug_dpp8(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_VINTRP: + gfx10_debug_vintrp(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_DS: + gfx10_debug_ds(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_MTBUF: + gfx10_debug_mtbuf(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_MUBUF: + gfx10_debug_mubuf(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_MIMG: + gfx10_debug_mimg(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_FLAT: + gfx10_debug_global(&insn->gfx10); + break; + case AMDGCN_INSN_TYPE_EXP: + gfx10_debug_exp(&insn->gfx10); + break; + default: + WARN_ON_ONCE(1); + break; + } + + ptr =3D (u32 *)&insn->gfx10; + if (insn->size =3D=3D 4) { + pr_debug("knod_asm %s %u %.8X\n", __func__, __LINE__, ptr[0]); + } else if (insn->size =3D=3D 8) { + pr_debug("knod_asm %s %u %.8X %.8X\n", + __func__, __LINE__, ptr[0], ptr[1]); + } else if (insn->size =3D=3D 12) { + pr_debug("knod_asm %s %u %.8X %.8X %.8X\n", + __func__, __LINE__, ptr[0], ptr[1], ptr[2]); + } else { + WARN_ON_ONCE(1); + } +} + +static inline void gfx9_debug_vop3a(union amdgcn_gfx9_insn *insn) +{ + char src0[20]; + char src1[20]; + char src2[20]; + + if (insn->vop3a.src0 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3a.src0); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3a.src0 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3a.src0 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src0 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3a.src0 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src0 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3a.src0 - GFX9_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src1 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3a.src1); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3a.src1 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3a.src1 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src1 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3a.src1 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src1 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3a.src1 - GFX9_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src2 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3a.src2); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3a.src2 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3a.src2 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src2 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3a.src2 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src2 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3a.src2 - GFX9_VOP3A_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3A][insn->vop3a.op], + insn->vop3a.vdst, src0, src1, src2); +} + +static inline void gfx9_debug_vop3b(union amdgcn_gfx9_insn *insn) +{ + char src0[20]; + char src1[20]; + char src2[20]; + char sdst[20]; + + if (insn->vop3b.sdst < GFX9_VOP3B_SRC_VCC_LO) + sprintf(sdst, "s%d", insn->vop3b.sdst); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(sdst, "%s", "vcc_lo"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(sdst, "%s", "vcc_hi"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(sdst, "%s", "null"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(sdst, "%s", "exec_lo"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(sdst, "%s", "exec_hi"); + else if (insn->vop3b.sdst < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(sdst, "0x%x", + insn->vop3b.sdst - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.sdst < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(sdst, "-0x%x", + insn->vop3b.sdst - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(sdst, "%s", "vcc"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(sdst, "%s", "exec"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(sdst, "%s", "scc"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(sdst, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.sdst >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(sdst, "v%d", + insn->vop3b.sdst - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src0 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3b.src0); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3b.src0 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3b.src0 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src0 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3b.src0 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src0 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3b.src0 - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src1 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3b.src1); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3b.src1 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3b.src1 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src1 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3b.src1 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src1 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3b.src1 - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src2 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3b.src2); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3b.src2 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3b.src2 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src2 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3b.src2 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src2 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3b.src2 - GFX9_VOP3B_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3B][insn->vop3b.op], + insn->vop3b.vdst, sdst, src0, src1, src2); +} + +static inline void gfx9_debug_vop1(union amdgcn_gfx9_insn *insn) +{ + char src0[20]; + + if (insn->vop1.src0 < GFX9_VOP1_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop1.src0); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop1.src0 < GFX9_VOP1_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop1.src0 - GFX9_VOP1_SRC_INTEGER_0); + else if (insn->vop1.src0 < GFX9_VOP1_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop1.src0 - GFX9_VOP1_SRC_INTEGER_MINUS_1); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop1.literal); + else if (insn->vop1.src0 >=3D GFX9_VOP1_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vop1.src0 - GFX9_VOP1_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP1][insn->vop1.op], + insn->vop1.vdst, src0); +} + +static inline void gfx9_debug_vopc(union amdgcn_gfx9_insn *insn) +{ + char src0[20]; + + if (insn->vopc.src0 < GFX9_VOPC_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vopc.src0); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vopc.src0 < GFX9_VOPC_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vopc.src0 - GFX9_VOPC_SRC_INTEGER_0); + else if (insn->vopc.src0 < GFX9_VOPC_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vopc.src0 - GFX9_VOPC_SRC_INTEGER_MINUS_1); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vopc.literal); + else if (insn->vopc.src0 >=3D GFX9_VOPC_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vopc.src0 - GFX9_VOPC_SRC_VGPR_BASE); + + pr_debug("knod_asm %s %s, v%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOPC][insn->vopc.op], + src0, + insn->vopc.vsrc1); +} + +static inline void gfx9_debug_vop2(union amdgcn_gfx9_insn *insn) +{ + char src0[20]; + + if (insn->vop2.src0 < GFX9_VOP2_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop2.src0); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop2.src0 < GFX9_VOP2_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop2.src0 - GFX9_VOP2_SRC_INTEGER_0); + else if (insn->vop2.src0 < GFX9_VOP2_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop2.src0 - GFX9_VOP2_SRC_INTEGER_MINUS_1); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop2.literal); + else if (insn->vop2.src0 >=3D GFX9_VOP2_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vop2.src0 - GFX9_VOP2_SRC_VGPR_BASE); + + pr_debug("knod_asm %s v%d, %s v%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP2][insn->vop2.op], + insn->vop2.vdst, + src0, + insn->vop2.vsrc1); +} + +static inline void gfx9_debug_sop1(union amdgcn_gfx9_insn *insn) +{ + char ssrc0[20]; + + if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "s%d", insn->sop1.ssrc0); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "%s", "vcc_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCC_HI) + sprintf(ssrc0, "%s", "vcc_hi"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_NULL) + sprintf(ssrc0, "%s", "null"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXEC_LO) + sprintf(ssrc0, "%s", "exec_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXEC_HI) + sprintf(ssrc0, "%s", "exec_hi"); + else if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_INTEGER_MINUS_1) + sprintf(ssrc0, "0x%x", + insn->sop1.ssrc0 - GFX9_SOP1_SSRC_INTEGER_0); + else if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_SHARED_BASE) + sprintf(ssrc0, "-0x%x", + insn->sop1.ssrc0 - GFX9_SOP1_SSRC_INTEGER_MINUS_1); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCCZ) + sprintf(ssrc0, "%s", "vcc"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXECZ) + sprintf(ssrc0, "%s", "exec"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_SCC) + sprintf(ssrc0, "%s", "scc"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_LITERAL_CONST) + sprintf(ssrc0, "0x%x", insn->sop1.literal); + + pr_debug("knod_asm %s s%d, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOP1][insn->sop1.op], + insn->sop1.sdst, + ssrc0); +} + +static inline void gfx9_debug_sopp(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm %s 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPP][insn->sopp.op], + insn->sopp.simm16); +} + +static inline void gfx9_debug_smem(union amdgcn_gfx9_insn *insn) +{ + pr_debug("%s s[%d:%d], s[%d:%d], 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SMEM][insn->smem.op], + insn->smem.sdata, + insn->smem.sdata + 1, + (insn->smem.sbase * 2), + (insn->smem.sbase * 2) + 1, + insn->smem.offset); +} + +static inline void gfx9_debug_mubuf(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, s[%d:%d], 0x%x offen offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_MUBUF][insn->mubuf.op], + insn->mubuf.vdata, + insn->mubuf.vaddr, + insn->mubuf.srsrc, + insn->mubuf.srsrc + 3, + insn->mubuf.soffset, + insn->mubuf.offset); +} + +static inline void gfx9_debug_global(union amdgcn_gfx9_insn *insn) +{ + if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_BYTE) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_SHORT) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORD) { + pr_debug("knod_asm %s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORDX2) { + pr_debug("knod_asm %s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.data + 1, + insn->flat.offset); + } else { + pr_debug("knod_asm %s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.vdst, + insn->flat.vdst + 1, + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.offset); + } +} + +static inline void gfx9_debug_sop2(union amdgcn_gfx9_insn *insn) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sop2.ssrc0, insn->sop2.literal, ssrc0); + decode_ssrc8(insn->sop2.ssrc1, insn->sop2.literal, ssrc1); + pr_debug("knod_asm %s s%d, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOP2][insn->sop2.op], + insn->sop2.sdst, ssrc0, ssrc1); +} + +static inline void gfx9_debug_sopk(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm %s s%d, 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPK][insn->sopk.op], + insn->sopk.sdst, insn->sopk.simm16); +} + +static inline void gfx9_debug_sopc(union amdgcn_gfx9_insn *insn) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sopc.ssrc0, insn->sopc.literal, ssrc0); + decode_ssrc8(insn->sopc.ssrc1, insn->sopc.literal, ssrc1); + pr_debug("knod_asm %s %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPC][insn->sopc.op], + ssrc0, ssrc1); +} + +static inline void gfx9_debug_vop3p(union amdgcn_gfx9_insn *insn) +{ + char src0[20], src1[20], src2[20]; + + decode_vsrc9(insn->vop3p.src0, insn->vop3p.literal, src0); + decode_vsrc9(insn->vop3p.src1, insn->vop3p.literal, src1); + decode_vsrc9(insn->vop3p.src2, insn->vop3p.literal, src2); + pr_debug("knod_asm %s v%d, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3P][insn->vop3p.op], + (int)insn->vop3p.vdst, src0, src1, src2); +} + +static inline void gfx9_debug_sdwa(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm sdwa src0:%d dst_sel:%d src0_sel:%d src1_sel:%d\n", + insn->sdwa.src0, insn->sdwa.dst_sel, + insn->sdwa.src0_sel, insn->sdwa.src1_sel); +} + +static inline void gfx9_debug_sdwab(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm sdwab src0:%d sdst:s%d src0_sel:%d src1_sel:%d\n", + insn->sdwab.src0, insn->sdwab.sdst, + insn->sdwab.src0_sel, insn->sdwab.src1_sel); +} + +static inline void gfx9_debug_dpp16(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm dpp16 (not decoded)\n"); +} + +static inline void gfx9_debug_dpp8(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm dpp8 (not decoded)\n"); +} + +static inline void gfx9_debug_vintrp(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm vintrp (not decoded)\n"); +} + +static inline void gfx9_debug_ds(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, v%d, v%d offset0:%d offset1:%d%s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_DS][insn->ds.op], + (int)insn->ds.vdst, (int)insn->ds.addr, + (int)insn->ds.data0, (int)insn->ds.data1, + (int)insn->ds.offset0, (int)insn->ds.offset1, + insn->ds.gds ? " gds" : ""); +} + +static inline void gfx9_debug_mtbuf(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm %s v%d, v%d, s[%d:%d], s%d dfmt:%d nfmt:%d offset:%d\n= ", + opnames_gfx9[AMDGCN_INSN_TYPE_MTBUF][insn->mtbuf.op], + (int)insn->mtbuf.vdata, (int)insn->mtbuf.vaddr, + (int)insn->mtbuf.srsrc * 4, (int)insn->mtbuf.srsrc * 4 + 3, + (int)insn->mtbuf.soffset, (int)insn->mtbuf.dfmt, + (int)insn->mtbuf.nfmt, (int)insn->mtbuf.offset); +} + +static inline void gfx9_debug_mimg(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm mimg (not decoded)\n"); +} + +static inline void gfx9_debug_exp(union amdgcn_gfx9_insn *insn) +{ + pr_debug("knod_asm exp (not decoded)\n"); +} + +static inline void gfx9_debug_insn(struct amdgcn_insn *insn) +{ + u32 *ptr; + + switch (insn->type) { + case AMDGCN_INSN_TYPE_SOP2: + gfx9_debug_sop2(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SOPK: + gfx9_debug_sopk(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SOP1: + gfx9_debug_sop1(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SOPC: + gfx9_debug_sopc(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SOPP: + gfx9_debug_sopp(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SMEM: + gfx9_debug_smem(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOP2: + gfx9_debug_vop2(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOP1: + gfx9_debug_vop1(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOPC: + gfx9_debug_vopc(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOP3A: + gfx9_debug_vop3a(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOP3B: + gfx9_debug_vop3b(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VOP3P: + gfx9_debug_vop3p(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SDWA: + gfx9_debug_sdwa(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_SDWAB: + gfx9_debug_sdwab(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_DPP16: + gfx9_debug_dpp16(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_DPP8: + gfx9_debug_dpp8(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_VINTRP: + gfx9_debug_vintrp(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_DS: + gfx9_debug_ds(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_MTBUF: + gfx9_debug_mtbuf(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_MUBUF: + gfx9_debug_mubuf(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_MIMG: + gfx9_debug_mimg(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_FLAT: + gfx9_debug_global(&insn->gfx9); + break; + case AMDGCN_INSN_TYPE_EXP: + gfx9_debug_exp(&insn->gfx9); + break; + default: + WARN_ON_ONCE(1); + break; + } + + ptr =3D (u32 *)&insn->gfx9; + if (insn->size =3D=3D 4) { + pr_debug("knod_asm %s %u %.8X\n", __func__, __LINE__, ptr[0]); + } else if (insn->size =3D=3D 8) { + pr_debug("knod_asm %s %u %.8X %.8X\n", + __func__, __LINE__, ptr[0], ptr[1]); + } else if (insn->size =3D=3D 12) { + pr_debug("knod_asm %s %u %.8X %.8X %.8X\n", + __func__, __LINE__, ptr[0], ptr[1], ptr[2]); + } else { + WARN_ON_ONCE(1); + } +} + +static inline void debug_insn(int version, struct amdgcn_insn *insn) +{ + if (version =3D=3D 10) + gfx10_debug_insn(insn); + else if (version =3D=3D 9) + gfx9_debug_insn(insn); + else + WARN_ON_ONCE(1); +} + +static inline void gfx10_debugfs_vop3a(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20]; + char src1[20]; + char src2[20]; + + if (insn->vop3a.src0 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3a.src0); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3a.src0 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3a.src0 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src0 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3a.src0 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3a.src0 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src0 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3a.src0 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src1 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3a.src1); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3a.src1 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3a.src1 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src1 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3a.src1 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3a.src1 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src1 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3a.src1 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src2 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3a.src2); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3a.src2 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3a.src2 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src2 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3a.src2 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3a.src2 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src2 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3a.src2 - GFX10_VOP3A_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3A][insn->vop3a.op], + insn->vop3a.vdst, src0, src1, src2); +} + +static inline void gfx10_debugfs_vop3b(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20]; + char src1[20]; + char src2[20]; + char sdst[20]; + + if (insn->vop3b.sdst < GFX10_VOP3A_SRC_VCC_LO) + sprintf(sdst, "s%d", insn->vop3b.sdst); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(sdst, "%s", "vcc_lo"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(sdst, "%s", "vcc_hi"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(sdst, "%s", "null"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(sdst, "%s", "exec_lo"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(sdst, "%s", "exec_hi"); + else if (insn->vop3b.sdst < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(sdst, "0x%x", + insn->vop3b.sdst - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.sdst < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(sdst, "-0x%x", + insn->vop3b.sdst - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(sdst, "%s", "vcc"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(sdst, "%s", "exec"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(sdst, "%s", "scc"); + else if (insn->vop3b.sdst =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(sdst, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.sdst >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(sdst, "v%d", + insn->vop3b.sdst - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src0 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3b.src0); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3b.src0 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3b.src0 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src0 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3b.src0 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3b.src0 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src0 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3b.src0 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src1 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3b.src1); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3b.src1 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3b.src1 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src1 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3b.src1 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3b.src1 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src1 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3b.src1 - GFX10_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3b.src2 < GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3b.src2); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3b.src2 < GFX10_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3b.src2 - GFX10_VOP3A_SRC_INTEGER_0); + else if (insn->vop3b.src2 < GFX10_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3b.src2 - GFX10_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3b.src2 =3D=3D GFX10_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src2 >=3D GFX10_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3b.src2 - GFX10_VOP3A_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3B][insn->vop3b.op], + insn->vop3b.vdst, sdst, src0, src1, src2); +} + +static inline void gfx10_debugfs_vop1(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vop1.src0 < GFX10_VOP1_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop1.src0); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop1.src0 < GFX10_VOP1_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop1.src0 - GFX10_VOP1_SRC_INTEGER_0); + else if (insn->vop1.src0 < GFX10_VOP1_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop1.src0 - GFX10_VOP1_SRC_INTEGER_MINUS_1); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop1.src0 =3D=3D GFX10_VOP1_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop1.literal); + else if (insn->vop1.src0 >=3D GFX10_VOP1_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop1.src0 - GFX10_VOP1_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP1][insn->vop1.op], + insn->vop1.vdst, src0); +} + +static inline void gfx10_debugfs_vopc(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vopc.src0 < GFX10_VOPC_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vopc.src0); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vopc.src0 < GFX10_VOPC_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vopc.src0 - GFX10_VOPC_SRC_INTEGER_0); + else if (insn->vopc.src0 < GFX10_VOPC_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vopc.src0 - GFX10_VOPC_SRC_INTEGER_MINUS_1); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vopc.src0 =3D=3D GFX10_VOPC_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vopc.literal); + else if (insn->vopc.src0 >=3D GFX10_VOPC_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vopc.src0 - GFX10_VOPC_SRC_VGPR_BASE); + + seq_printf(m, "%s %s, v%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOPC][insn->vopc.op], + src0, insn->vopc.vsrc1); +} + +static inline void gfx10_debugfs_vop2(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vop2.src0 < GFX10_VOP2_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop2.src0); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop2.src0 < GFX10_VOP2_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop2.src0 - GFX10_VOP2_SRC_INTEGER_0); + else if (insn->vop2.src0 < GFX10_VOP2_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop2.src0 - GFX10_VOP2_SRC_INTEGER_MINUS_1); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop2.src0 =3D=3D GFX10_VOP2_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop2.literal); + else if (insn->vop2.src0 >=3D GFX10_VOP2_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop2.src0 - GFX10_VOP2_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s v%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP2][insn->vop2.op], + insn->vop2.vdst, + src0, + insn->vop2.vsrc1); +} + +static inline void gfx10_debugfs_sop1(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char ssrc0[20]; + + if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "s%d", insn->sop1.ssrc0); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "%s", "vcc_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCC_HI) + sprintf(ssrc0, "%s", "vcc_hi"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_NULL) + sprintf(ssrc0, "%s", "null"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXEC_LO) + sprintf(ssrc0, "%s", "exec_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXEC_HI) + sprintf(ssrc0, "%s", "exec_hi"); + else if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_INTEGER_MINUS_1) + sprintf(ssrc0, "0x%x", + insn->sop1.ssrc0 - GFX10_SOP1_SSRC_INTEGER_0); + else if (insn->sop1.ssrc0 < GFX10_SOP1_SSRC_SHARED_BASE) + sprintf(ssrc0, "-0x%x", + insn->sop1.ssrc0 - GFX10_SOP1_SSRC_INTEGER_MINUS_1); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_VCCZ) + sprintf(ssrc0, "%s", "vcc"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_EXECZ) + sprintf(ssrc0, "%s", "exec"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_SCC) + sprintf(ssrc0, "%s", "scc"); + else if (insn->sop1.ssrc0 =3D=3D GFX10_SOP1_SSRC_LITERAL_CONST) + sprintf(ssrc0, "0x%x", insn->sop1.literal); + + seq_printf(m, "%s s%d, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOP1][insn->sop1.op], + insn->sop1.sdst, + ssrc0); +} + +static inline void gfx10_debugfs_sopp(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPP][insn->sopp.op], + insn->sopp.simm16); +} + +static inline void gfx10_debugfs_smem(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s s[%d:%d], s[%d:%d], 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SMEM][insn->smem.op], + insn->smem.sdata, + insn->smem.sdata + 1, + (insn->smem.sbase * 2), + (insn->smem.sbase * 2) + 1, + insn->smem.offset); +} + +static inline void gfx10_debugfs_mubuf(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, s[%d:%d], 0x%x offen offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_MUBUF][insn->mubuf.op], + insn->mubuf.vdata, + insn->mubuf.vaddr, + insn->mubuf.srsrc, + insn->mubuf.srsrc + 3, + insn->mubuf.soffset, + insn->mubuf.offset); +} + +static inline void gfx10_debugfs_global(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_BYTE) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_SHORT) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORD) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORDX2) { + seq_printf(m, "%s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.data + 1, + insn->flat.offset); + } else { + seq_printf(m, "%s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.vdst, + insn->flat.vdst + 1, + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.offset); + } +} + +static inline void gfx10_debugfs_sop2(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sop2.ssrc0, insn->sop2.literal, ssrc0); + decode_ssrc8(insn->sop2.ssrc1, insn->sop2.literal, ssrc1); + seq_printf(m, "%s s%d, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOP2][insn->sop2.op], + insn->sop2.sdst, ssrc0, ssrc1); +} + +static inline void gfx10_debugfs_sopk(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s s%d, 0x%x\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPK][insn->sopk.op], + insn->sopk.sdst, insn->sopk.simm16); +} + +static inline void gfx10_debugfs_sopc(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sopc.ssrc0, insn->sopc.literal, ssrc0); + decode_ssrc8(insn->sopc.ssrc1, insn->sopc.literal, ssrc1); + seq_printf(m, "%s %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_SOPC][insn->sopc.op], + ssrc0, ssrc1); +} + +static inline void gfx10_debugfs_vop3p(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + char src0[20], src1[20], src2[20]; + + decode_vsrc9(insn->vop3p.src0, insn->vop3p.literal, src0); + decode_vsrc9(insn->vop3p.src1, insn->vop3p.literal, src1); + decode_vsrc9(insn->vop3p.src2, insn->vop3p.literal, src2); + seq_printf(m, "%s v%d, %s, %s, %s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_VOP3P][insn->vop3p.op], + (int)insn->vop3p.vdst, src0, src1, src2); +} + +static inline void gfx10_debugfs_sdwa(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "sdwa src0:%d dst_sel:%d src0_sel:%d src1_sel:%d\n", + insn->sdwa.src0, insn->sdwa.dst_sel, + insn->sdwa.src0_sel, insn->sdwa.src1_sel); +} + +static inline void gfx10_debugfs_sdwab(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "sdwab src0:%d sdst:s%d src0_sel:%d src1_sel:%d\n", + insn->sdwab.src0, insn->sdwab.sdst, + insn->sdwab.src0_sel, insn->sdwab.src1_sel); +} + +static inline void gfx10_debugfs_dpp16(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "dpp16 (not decoded)\n"); +} + +static inline void gfx10_debugfs_dpp8(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "dpp8 (not decoded)\n"); +} + +static inline void gfx10_debugfs_vintrp(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "vintrp (not decoded)\n"); +} + +static inline void gfx10_debugfs_ds(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, v%d, v%d offset0:%d offset1:%d%s\n", + opnames_gfx10[AMDGCN_INSN_TYPE_DS][insn->ds.op], + (int)insn->ds.vdst, (int)insn->ds.addr, + (int)insn->ds.data0, (int)insn->ds.data1, + (int)insn->ds.offset0, (int)insn->ds.offset1, + insn->ds.gds ? " gds" : ""); +} + +static inline void gfx10_debugfs_mtbuf(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, s[%d:%d], s%d format:%d offset:%d\n", + opnames_gfx10[AMDGCN_INSN_TYPE_MTBUF][insn->mtbuf.op], + (int)insn->mtbuf.vdata, (int)insn->mtbuf.vaddr, + (int)insn->mtbuf.srsrc * 4, (int)insn->mtbuf.srsrc * 4 + 3, + (int)insn->mtbuf.soffset, (int)insn->mtbuf.foamat, + (int)insn->mtbuf.offset); +} + +static inline void gfx10_debugfs_mimg(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "mimg (not decoded)\n"); +} + +static inline void gfx10_debugfs_exp(union amdgcn_gfx10_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "exp (not decoded)\n"); +} + +static inline void gfx10_debugfs_insn(struct amdgcn_insn *insn, + struct seq_file *m) +{ + u32 *ptr =3D (u32 *)&insn->gfx10; + + if (insn->size =3D=3D 4) + seq_printf(m, "%.8X\t\t\t", ptr[0]); + else if (insn->size =3D=3D 8) + seq_printf(m, "%.8X %.8X\t\t", ptr[0], ptr[1]); + else if (insn->size =3D=3D 12) + seq_printf(m, "%.8X %.8X %.8X\t\t\t", ptr[0], ptr[1], ptr[2]); + else + WARN_ON_ONCE(1); + + switch (insn->type) { + case AMDGCN_INSN_TYPE_SOP2: + gfx10_debugfs_sop2(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SOPK: + gfx10_debugfs_sopk(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SOP1: + gfx10_debugfs_sop1(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SOPC: + gfx10_debugfs_sopc(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SOPP: + gfx10_debugfs_sopp(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SMEM: + gfx10_debugfs_smem(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOP2: + gfx10_debugfs_vop2(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOP1: + gfx10_debugfs_vop1(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOPC: + gfx10_debugfs_vopc(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOP3A: + gfx10_debugfs_vop3a(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOP3B: + gfx10_debugfs_vop3b(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VOP3P: + gfx10_debugfs_vop3p(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SDWA: + gfx10_debugfs_sdwa(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_SDWAB: + gfx10_debugfs_sdwab(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_DPP16: + gfx10_debugfs_dpp16(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_DPP8: + gfx10_debugfs_dpp8(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_VINTRP: + gfx10_debugfs_vintrp(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_DS: + gfx10_debugfs_ds(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_MTBUF: + gfx10_debugfs_mtbuf(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_MUBUF: + gfx10_debugfs_mubuf(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_MIMG: + gfx10_debugfs_mimg(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_FLAT: + gfx10_debugfs_global(&insn->gfx10, m); + break; + case AMDGCN_INSN_TYPE_EXP: + gfx10_debugfs_exp(&insn->gfx10, m); + break; + default: + WARN_ON_ONCE(1); + break; + } +} + +static inline void gfx9_debugfs_vop3a(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20]; + char src1[20]; + char src2[20]; + + if (insn->vop3a.src0 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3a.src0); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3a.src0 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3a.src0 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src0 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3a.src0 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3a.src0 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src0 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3a.src0 - GFX9_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src1 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3a.src1); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3a.src1 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3a.src1 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src1 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3a.src1 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3a.src1 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src1 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3a.src1 - GFX9_VOP3A_SRC_VGPR_BASE); + + if (insn->vop3a.src2 < GFX9_VOP3A_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3a.src2); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3a.src2 < GFX9_VOP3A_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3a.src2 - GFX9_VOP3A_SRC_INTEGER_0); + else if (insn->vop3a.src2 < GFX9_VOP3A_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3a.src2 - GFX9_VOP3A_SRC_INTEGER_MINUS_1); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3a.src2 =3D=3D GFX9_VOP3A_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3a.literal); + else if (insn->vop3a.src2 >=3D GFX9_VOP3A_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3a.src2 - GFX9_VOP3A_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3A][insn->vop3a.op], + insn->vop3a.vdst, src0, src1, src2); +} + +static inline void gfx9_debugfs_vop3b(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20]; + char src1[20]; + char src2[20]; + char sdst[20]; + + if (insn->vop3b.sdst < GFX9_VOP3B_SRC_VCC_LO) + sprintf(sdst, "s%d", insn->vop3b.sdst); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(sdst, "%s", "vcc_lo"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(sdst, "%s", "vcc_hi"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(sdst, "%s", "null"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(sdst, "%s", "exec_lo"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(sdst, "%s", "exec_hi"); + else if (insn->vop3b.sdst < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(sdst, "0x%x", + insn->vop3b.sdst - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.sdst < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(sdst, "-0x%x", + insn->vop3b.sdst - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(sdst, "%s", "vcc"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(sdst, "%s", "exec"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(sdst, "%s", "scc"); + else if (insn->vop3b.sdst =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(sdst, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.sdst >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(sdst, "v%d", + insn->vop3b.sdst - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src0 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop3b.src0); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop3b.src0 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop3b.src0 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src0 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop3b.src0 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop3b.src0 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src0 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src0, "v%d", + insn->vop3b.src0 - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src1 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src1, "s%d", insn->vop3b.src1); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src1, "%s", "vcc_lo"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src1, "%s", "vcc_hi"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src1, "%s", "null"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src1, "%s", "exec_lo"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src1, "%s", "exec_hi"); + else if (insn->vop3b.src1 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src1, "0x%x", + insn->vop3b.src1 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src1 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src1, "-0x%x", + insn->vop3b.src1 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src1, "%s", "vcc"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src1, "%s", "exec"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src1, "%s", "scc"); + else if (insn->vop3b.src1 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src1, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src1 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src1, "v%d", + insn->vop3b.src1 - GFX9_VOP3B_SRC_VGPR_BASE); + + if (insn->vop3b.src2 < GFX9_VOP3B_SRC_VCC_LO) + sprintf(src2, "s%d", insn->vop3b.src2); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCC_LO) + sprintf(src2, "%s", "vcc_lo"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCC_HI) + sprintf(src2, "%s", "vcc_hi"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_NULL) + sprintf(src2, "%s", "null"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXEC_LO) + sprintf(src2, "%s", "exec_lo"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXEC_HI) + sprintf(src2, "%s", "exec_hi"); + else if (insn->vop3b.src2 < GFX9_VOP3B_SRC_INTEGER_MINUS_1) + sprintf(src2, "0x%x", + insn->vop3b.src2 - GFX9_VOP3B_SRC_INTEGER_0); + else if (insn->vop3b.src2 < GFX9_VOP3B_SRC_SHARED_BASE) + sprintf(src2, "-0x%x", + insn->vop3b.src2 - GFX9_VOP3B_SRC_INTEGER_MINUS_1); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_VCCZ) + sprintf(src2, "%s", "vcc"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_EXECZ) + sprintf(src2, "%s", "exec"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_SCC) + sprintf(src2, "%s", "scc"); + else if (insn->vop3b.src2 =3D=3D GFX9_VOP3B_SRC_LITERAL_CONST) + sprintf(src2, "0x%x", insn->vop3b.literal); + else if (insn->vop3b.src2 >=3D GFX9_VOP3B_SRC_VGPR_BASE) + sprintf(src2, "v%d", + insn->vop3b.src2 - GFX9_VOP3B_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3B][insn->vop3b.op], + insn->vop3b.vdst, sdst, src0, src1, src2); +} + +static inline void gfx9_debugfs_vop1(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vop1.src0 < GFX9_VOP1_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop1.src0); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop1.src0 < GFX9_VOP1_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop1.src0 - GFX9_VOP1_SRC_INTEGER_0); + else if (insn->vop1.src0 < GFX9_VOP1_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop1.src0 - GFX9_VOP1_SRC_INTEGER_MINUS_1); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop1.src0 =3D=3D GFX9_VOP1_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop1.literal); + else if (insn->vop1.src0 >=3D GFX9_VOP1_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vop1.src0 - GFX9_VOP1_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP1][insn->vop1.op], + insn->vop1.vdst, src0); +} + +static inline void gfx9_debugfs_vopc(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vopc.src0 < GFX9_VOPC_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vopc.src0); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vopc.src0 < GFX9_VOPC_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vopc.src0 - GFX9_VOPC_SRC_INTEGER_0); + else if (insn->vopc.src0 < GFX9_VOPC_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vopc.src0 - GFX9_VOPC_SRC_INTEGER_MINUS_1); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vopc.src0 =3D=3D GFX9_VOPC_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vopc.literal); + else if (insn->vopc.src0 >=3D GFX9_VOPC_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vopc.src0 - GFX9_VOPC_SRC_VGPR_BASE); + + seq_printf(m, "%s %s, v%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOPC][insn->vopc.op], + src0, + insn->vopc.vsrc1); +} + +static inline void gfx9_debugfs_vop2(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20]; + + if (insn->vop2.src0 < GFX9_VOP2_SRC_VCC_LO) + sprintf(src0, "s%d", insn->vop2.src0); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCC_LO) + sprintf(src0, "%s", "vcc_lo"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCC_HI) + sprintf(src0, "%s", "vcc_hi"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_NULL) + sprintf(src0, "%s", "null"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXEC_LO) + sprintf(src0, "%s", "exec_lo"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXEC_HI) + sprintf(src0, "%s", "exec_hi"); + else if (insn->vop2.src0 < GFX9_VOP2_SRC_INTEGER_MINUS_1) + sprintf(src0, "0x%x", + insn->vop2.src0 - GFX9_VOP2_SRC_INTEGER_0); + else if (insn->vop2.src0 < GFX9_VOP2_SRC_SHARED_BASE) + sprintf(src0, "-0x%x", + insn->vop2.src0 - GFX9_VOP2_SRC_INTEGER_MINUS_1); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_VCCZ) + sprintf(src0, "%s", "vcc"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_EXECZ) + sprintf(src0, "%s", "exec"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_SCC) + sprintf(src0, "%s", "scc"); + else if (insn->vop2.src0 =3D=3D GFX9_VOP2_SRC_LITERAL_CONST) + sprintf(src0, "0x%x", insn->vop2.literal); + else if (insn->vop2.src0 >=3D GFX9_VOP2_SRC_VGPR_BASE) + sprintf(src0, "v%d", insn->vop2.src0 - GFX9_VOP2_SRC_VGPR_BASE); + + seq_printf(m, "%s v%d, %s v%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP2][insn->vop2.op], + insn->vop2.vdst, + src0, + insn->vop2.vsrc1); +} + +static inline void gfx9_debugfs_sop1(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char ssrc0[20]; + + if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "s%d", insn->sop1.ssrc0); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCC_LO) + sprintf(ssrc0, "%s", "vcc_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCC_HI) + sprintf(ssrc0, "%s", "vcc_hi"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_NULL) + sprintf(ssrc0, "%s", "null"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXEC_LO) + sprintf(ssrc0, "%s", "exec_lo"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXEC_HI) + sprintf(ssrc0, "%s", "exec_hi"); + else if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_INTEGER_MINUS_1) + sprintf(ssrc0, "0x%x", + insn->sop1.ssrc0 - GFX9_SOP1_SSRC_INTEGER_0); + else if (insn->sop1.ssrc0 < GFX9_SOP1_SSRC_SHARED_BASE) + sprintf(ssrc0, "-0x%x", + insn->sop1.ssrc0 - GFX9_SOP1_SSRC_INTEGER_MINUS_1); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_VCCZ) + sprintf(ssrc0, "%s", "vcc"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_EXECZ) + sprintf(ssrc0, "%s", "exec"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_SCC) + sprintf(ssrc0, "%s", "scc"); + else if (insn->sop1.ssrc0 =3D=3D GFX9_SOP1_SSRC_LITERAL_CONST) + sprintf(ssrc0, "0x%x", insn->sop1.literal); + + seq_printf(m, "%s s%d, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOP1][insn->sop1.op], + insn->sop1.sdst, + ssrc0); +} + +static inline void gfx9_debugfs_sopp(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPP][insn->sopp.op], + insn->sopp.simm16); +} + +static inline void gfx9_debugfs_smem(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s s[%d:%d], s[%d:%d], 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SMEM][insn->smem.op], + insn->smem.sdata, + insn->smem.sdata + 1, + (insn->smem.sbase * 2), + (insn->smem.sbase * 2) + 1, + insn->smem.offset); +} + +static inline void gfx9_debugfs_mubuf(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, s[%d:%d], 0x%x offen offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_MUBUF][insn->mubuf.op], + insn->mubuf.vdata, + insn->mubuf.vaddr, + insn->mubuf.srsrc, + insn->mubuf.srsrc + 3, + insn->mubuf.soffset, + insn->mubuf.offset); +} + +static inline void gfx9_debugfs_global(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_BYTE) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_SHORT) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORD) { + seq_printf(m, "%s v[%d:%d], v%d, off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.offset); + } else if (insn->flat.op =3D=3D GFX9_GLOBAL_STORE_DWORDX2) { + seq_printf(m, "%s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.data, + insn->flat.data + 1, + insn->flat.offset); + } else { + seq_printf(m, "%s v[%d:%d], v[%d:%d], off offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_FLAT][insn->flat.op], + insn->flat.vdst, + insn->flat.vdst + 1, + insn->flat.addr, + insn->flat.addr + 1, + insn->flat.offset); + } +} + +static inline void gfx9_debugfs_sop2(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sop2.ssrc0, insn->sop2.literal, ssrc0); + decode_ssrc8(insn->sop2.ssrc1, insn->sop2.literal, ssrc1); + seq_printf(m, "%s s%d, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOP2][insn->sop2.op], + insn->sop2.sdst, ssrc0, ssrc1); +} + +static inline void gfx9_debugfs_sopk(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s s%d, 0x%x\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPK][insn->sopk.op], + insn->sopk.sdst, insn->sopk.simm16); +} + +static inline void gfx9_debugfs_sopc(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char ssrc0[20], ssrc1[20]; + + decode_ssrc8(insn->sopc.ssrc0, insn->sopc.literal, ssrc0); + decode_ssrc8(insn->sopc.ssrc1, insn->sopc.literal, ssrc1); + seq_printf(m, "%s %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_SOPC][insn->sopc.op], + ssrc0, ssrc1); +} + +static inline void gfx9_debugfs_vop3p(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + char src0[20], src1[20], src2[20]; + + decode_vsrc9(insn->vop3p.src0, insn->vop3p.literal, src0); + decode_vsrc9(insn->vop3p.src1, insn->vop3p.literal, src1); + decode_vsrc9(insn->vop3p.src2, insn->vop3p.literal, src2); + seq_printf(m, "%s v%d, %s, %s, %s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_VOP3P][insn->vop3p.op], + (int)insn->vop3p.vdst, src0, src1, src2); +} + +static inline void gfx9_debugfs_sdwa(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "sdwa src0:%d dst_sel:%d src0_sel:%d src1_sel:%d\n", + insn->sdwa.src0, insn->sdwa.dst_sel, + insn->sdwa.src0_sel, insn->sdwa.src1_sel); +} + +static inline void gfx9_debugfs_sdwab(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "sdwab src0:%d sdst:s%d src0_sel:%d src1_sel:%d\n", + insn->sdwab.src0, insn->sdwab.sdst, + insn->sdwab.src0_sel, insn->sdwab.src1_sel); +} + +static inline void gfx9_debugfs_dpp16(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "dpp16 (not decoded)\n"); +} + +static inline void gfx9_debugfs_dpp8(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "dpp8 (not decoded)\n"); +} + +static inline void gfx9_debugfs_vintrp(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "vintrp (not decoded)\n"); +} + +static inline void gfx9_debugfs_ds(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, v%d, v%d offset0:%d offset1:%d%s\n", + opnames_gfx9[AMDGCN_INSN_TYPE_DS][insn->ds.op], + (int)insn->ds.vdst, (int)insn->ds.addr, + (int)insn->ds.data0, (int)insn->ds.data1, + (int)insn->ds.offset0, (int)insn->ds.offset1, + insn->ds.gds ? " gds" : ""); +} + +static inline void gfx9_debugfs_mtbuf(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_printf(m, "%s v%d, v%d, s[%d:%d], s%d dfmt:%d nfmt:%d offset:%d\n", + opnames_gfx9[AMDGCN_INSN_TYPE_MTBUF][insn->mtbuf.op], + (int)insn->mtbuf.vdata, (int)insn->mtbuf.vaddr, + (int)insn->mtbuf.srsrc * 4, (int)insn->mtbuf.srsrc * 4 + 3, + (int)insn->mtbuf.soffset, (int)insn->mtbuf.dfmt, + (int)insn->mtbuf.nfmt, (int)insn->mtbuf.offset); +} + +static inline void gfx9_debugfs_mimg(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "mimg (not decoded)\n"); +} + +static inline void gfx9_debugfs_exp(union amdgcn_gfx9_insn *insn, + struct seq_file *m) +{ + seq_puts(m, "exp (not decoded)\n"); +} + +static inline void gfx9_debugfs_insn(struct amdgcn_insn *insn, + struct seq_file *m) +{ + u32 *ptr =3D (u32 *)&insn->gfx9; + + if (insn->size =3D=3D 4) + seq_printf(m, "%.8X\t\t\t", ptr[0]); + else if (insn->size =3D=3D 8) + seq_printf(m, "%.8X %.8X\t\t", ptr[0], ptr[1]); + else if (insn->size =3D=3D 12) + seq_printf(m, "%.8X %.8X %.8X\t", ptr[0], ptr[1], ptr[2]); + else + WARN_ON_ONCE(1); + + switch (insn->type) { + case AMDGCN_INSN_TYPE_SOP2: + gfx9_debugfs_sop2(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SOPK: + gfx9_debugfs_sopk(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SOP1: + gfx9_debugfs_sop1(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SOPC: + gfx9_debugfs_sopc(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SOPP: + gfx9_debugfs_sopp(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SMEM: + gfx9_debugfs_smem(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOP2: + gfx9_debugfs_vop2(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOP1: + gfx9_debugfs_vop1(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOPC: + gfx9_debugfs_vopc(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOP3A: + gfx9_debugfs_vop3a(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOP3B: + gfx9_debugfs_vop3b(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VOP3P: + gfx9_debugfs_vop3p(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SDWA: + gfx9_debugfs_sdwa(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_SDWAB: + gfx9_debugfs_sdwab(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_DPP16: + gfx9_debugfs_dpp16(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_DPP8: + gfx9_debugfs_dpp8(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_VINTRP: + gfx9_debugfs_vintrp(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_DS: + gfx9_debugfs_ds(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_MTBUF: + gfx9_debugfs_mtbuf(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_MUBUF: + gfx9_debugfs_mubuf(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_MIMG: + gfx9_debugfs_mimg(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_FLAT: + gfx9_debugfs_global(&insn->gfx9, m); + break; + case AMDGCN_INSN_TYPE_EXP: + gfx9_debugfs_exp(&insn->gfx9, m); + break; + default: + WARN_ON_ONCE(1); + break; + } +} + +static inline void debugfs_insn(int version, struct amdgcn_insn *insn, + struct seq_file *m) +{ + if (version =3D=3D 10) + gfx10_debugfs_insn(insn, m); + else if (version =3D=3D 9) + gfx9_debugfs_insn(insn, m); + else + WARN_ON_ONCE(1); +} + +/* + * Recover instruction type + size from a raw machine-code word stream int= o a + * struct amdgcn_insn, so code holding only the emitted u32 blob (the feat= ure + * shaders emit via the low-level emit_gfxN_* helpers straight into the GPU + * code buffer, discarding type/size) can be fed to the shared disassembler + * debugfs_insn() instead of a per-feature hand-rolled hex re-parser. + * + * This mirrors the proven encoding-detection that used to live in the + * per-feature disassemblers. The checks are ordered widest + * fixed-prefix first: SOPP/SOP1/SOPC (9-bit enc) before SOPK (4-bit) befo= re + * SOP2 (2-bit), and every bit31=3D=3D1 class (VOP1/VOPC/SMEM/DS/FLAT/MUBU= F/VOP3) + * before the VOP2 (bit31=3D=3D0) catch-all. Version-specific encodings: S= MEM + * (gfx9 0x30 / gfx10 0x3d), SOPC literal (gfx10 only) and VOP3 (gfx9 0x34= / + * gfx10 0x35). size is in bytes (4/8/12) to match what emit_* stamps in. + */ +static inline void amdgcn_classify(int version, const u32 *code, + struct amdgcn_insn *out) +{ + enum amdgcn_insn_type type; + u32 w0 =3D code[0]; + bool is_b; + u32 dwords; + u32 enc; + u32 op; + + enc =3D (w0 >> 23) & 0x1ff; + + if (enc =3D=3D 0x17f) { /* SOPP */ + type =3D AMDGCN_INSN_TYPE_SOPP; + dwords =3D 1; + } else if (enc =3D=3D 0x17d) { /* SOP1 */ + type =3D AMDGCN_INSN_TYPE_SOP1; + dwords =3D ((w0 & 0xff) =3D=3D 0xff) ? 2 : 1; + } else if (enc =3D=3D 0x17e) { /* SOPC */ + type =3D AMDGCN_INSN_TYPE_SOPC; + dwords =3D (version =3D=3D 10 && + ((w0 & 0xff) =3D=3D 0xff || ((w0 >> 8) & 0xff) =3D=3D 0xff)) + ? 2 : 1; + } else if (((w0 >> 28) & 0xf) =3D=3D 0xb) { /* SOPK */ + type =3D AMDGCN_INSN_TYPE_SOPK; + dwords =3D 1; + } else if (((w0 >> 30) & 0x3) =3D=3D 0x2) { /* SOP2 */ + type =3D AMDGCN_INSN_TYPE_SOP2; + dwords =3D ((w0 & 0xff) =3D=3D 0xff || + ((w0 >> 8) & 0xff) =3D=3D 0xff) ? 2 : 1; + } else if (((w0 >> 26) & 0x3f) =3D=3D + (version =3D=3D 10 ? 0x3d : 0x30)) { /* SMEM */ + type =3D AMDGCN_INSN_TYPE_SMEM; + dwords =3D 2; + } else if (((w0 >> 25) & 0x7f) =3D=3D 0x3f) { /* VOP1 */ + type =3D AMDGCN_INSN_TYPE_VOP1; + dwords =3D ((w0 & 0xff) =3D=3D 0xff) ? 2 : 1; + } else if (((w0 >> 25) & 0x7f) =3D=3D 0x3e) { /* VOPC */ + type =3D AMDGCN_INSN_TYPE_VOPC; + dwords =3D 1; + } else if (((w0 >> 26) & 0x3f) =3D=3D 0x36) { /* DS */ + type =3D AMDGCN_INSN_TYPE_DS; + dwords =3D 2; + /* FLAT/GLOBAL/SCRATCH */ + } else if (((w0 >> 26) & 0x3f) =3D=3D 0x37) { + type =3D AMDGCN_INSN_TYPE_FLAT; + dwords =3D 2; + } else if (((w0 >> 26) & 0x3f) =3D=3D 0x38) { /* MUBUF */ + type =3D AMDGCN_INSN_TYPE_MUBUF; + dwords =3D 2; + } else if (((w0 >> 26) & 0x3f) =3D=3D + (version =3D=3D 10 ? 0x35 : 0x34)) { /* VOP3A / VOP3B */ + op =3D (w0 >> 16) & 0x3ff; + if (version =3D=3D 10) + is_b =3D (op =3D=3D GFX10_V_ADD_CO_U32 || + op =3D=3D GFX10_V_SUB_CO_U32 || + op =3D=3D GFX10_V_SUBREV_CO_U32 || + op =3D=3D GFX10_V_MAD_U64_U32 || + op =3D=3D GFX10_V_MAD_I64_I32 || + op =3D=3D GFX10_V_DIV_SCALE_F32 || + op =3D=3D GFX10_V_DIV_SCALE_F64); + else + is_b =3D (op =3D=3D GFX9_V_MAD_U64_U32 || + op =3D=3D GFX9_V_MAD_I64_I32 || + op =3D=3D GFX9_V_DIV_SCALE_F64); + type =3D is_b ? AMDGCN_INSN_TYPE_VOP3B : AMDGCN_INSN_TYPE_VOP3A; + dwords =3D 2; + } else if (((w0 >> 31) & 0x1) =3D=3D 0) { /* VOP2 */ + type =3D AMDGCN_INSN_TYPE_VOP2; + dwords =3D ((w0 & 0xff) =3D=3D 0xff) ? 2 : 1; + } else { /* unknown: 1 dword */ + type =3D AMDGCN_INSN_TYPE_SOPP; + dwords =3D 1; + } + + memset(out, 0, sizeof(*out)); + out->type =3D type; + out->size =3D dwords * 4; + memcpy(&out->gfx10, code, dwords * 4); +} + +/* + * Drop-in replacement for the old per-feature hex re-parsers: classify + * one raw instruction at @code, print it through the + * shared (complete-opnames) disassembler, and return the number of dwords + * consumed so the caller can advance. @max_dwords guards against reading a + * second dword past the end of the buffer. + */ +static inline int amdgcn_disasm_raw(int version, const u32 *code, + int max_dwords, struct seq_file *m) +{ + struct amdgcn_insn insn; + + if (max_dwords <=3D 0) + return 1; + amdgcn_classify(version, code, &insn); + if ((int)(insn.size / 4) > max_dwords) { + seq_printf(m, "%.8X\t\t\t(truncated)\n", code[0]); + return 1; + } + debugfs_insn(version, &insn, m); + return insn.size / 4; +} + + +#endif diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h b/drivers/gp= u/drm/amd/amdkfd/knod/knod_gfx10_insn.h new file mode 100644 index 000000000000..f2699adb6c7d --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h @@ -0,0 +1,3978 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_AMDGPU_GFX10_INSN_H_INCLUDED +#define KFD_AMDGPU_GFX10_INSN_H_INCLUDED + +#include "knod_amdgpu.h" + +/* See knod_gfx9_insn.h for rationale on not including knod_amdgpu_insn.h = */ + +#define GFX10_SRC_SGPR_BASE 0 +#define GFX10_SRC_VCC_LO 106 +#define GFX10_SRC_VCC_HI 107 +#define GFX10_SRC_TTPM_BASE 108 +#define GFX10_SRC_M0 124 +#define GFX10_SRC_NULL 125 +#define GFX10_SRC_EXEC_LO 126 +#define GFX10_SRC_EXEC_HI 127 +#define GFX10_SRC_INTEGER_0 128 +#define GFX10_SRC_INTEGER_MINUS_1 193 +#define GFX10_SRC_SHARED_BASE 235 +#define GFX10_SRC_SHARED_LIMIT 236 +#define GFX10_SRC_PRIVATE_BASE 237 +#define GFX10_SRC_PRIVATE_LIMIT 238 +#define GFX10_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_SRC_SDWA 249 +#define GFX10_SRC_DDP16 250 +#define GFX10_SRC_VCCZ 251 +#define GFX10_SRC_EXECZ 252 +#define GFX10_SRC_SCC 253 +#define GFX10_SRC_LITERAL_CONST 255 +#define GFX10_SRC_VGPR_BASE 256 + +static int gfx10_param_base[__AMDGCN_PARAM_TYPE_MAX] =3D { + GFX10_SRC_SGPR_BASE, + GFX10_SRC_VCC_LO, + GFX10_SRC_VCC_HI, + GFX10_SRC_TTPM_BASE, + GFX10_SRC_M0, + GFX10_SRC_NULL, + GFX10_SRC_EXEC_LO, + GFX10_SRC_EXEC_HI, + GFX10_SRC_INTEGER_0, + GFX10_SRC_INTEGER_MINUS_1, + GFX10_SRC_SHARED_BASE, + GFX10_SRC_SHARED_LIMIT, + GFX10_SRC_PRIVATE_BASE, + GFX10_SRC_PRIVATE_LIMIT, + GFX10_SRC_POPS_EXITING_WAVE_ID, + GFX10_SRC_SDWA, + GFX10_SRC_DDP16, + GFX10_SRC_VCCZ, + GFX10_SRC_EXECZ, + GFX10_SRC_SCC, + GFX10_SRC_LITERAL_CONST, + GFX10_SRC_VGPR_BASE, +}; + +/* Scalar ALU and Control Format */ +struct amdgcn_gfx10_sop2 { + u32 ssrc0:8; + u32 ssrc1:8; + u32 sdst:7; + u32 op:7; + u32 encoding:2; + u32 literal; +}; + +enum amdgcn_gfx10_sop2_opcode { + GFX10_S_ADD_U32 =3D 0, + GFX10_S_SUB_U32 =3D 1, + GFX10_S_ADD_I32 =3D 2, + GFX10_S_SUB_I32 =3D 3, + GFX10_S_ADDC_U32 =3D 4, + GFX10_S_SUBB_U32 =3D 5, + GFX10_S_MIN_I32 =3D 6, + GFX10_S_MIN_U32 =3D 7, + GFX10_S_MAX_I32 =3D 8, + GFX10_S_MAX_U32 =3D 9, + GFX10_S_CSELECT_B32 =3D 10, + GFX10_S_CELECT_B64 =3D 11, + /* 12-13: reserved */ + GFX10_S_AND_B32 =3D 14, + GFX10_S_AND_B64 =3D 15, + GFX10_S_OR_B32 =3D 16, + GFX10_S_OR_B64 =3D 17, + GFX10_S_XOR_B32 =3D 18, + GFX10_S_XOR_B64 =3D 19, + GFX10_S_ANDN2_B32 =3D 20, + GFX10_S_ANDN2_B64 =3D 21, + GFX10_S_ORN2_B32 =3D 22, + GFX10_S_ORN2_B64 =3D 23, + GFX10_S_NAND_B32 =3D 24, + GFX10_S_NAND_B64 =3D 25, + GFX10_S_NOR_B32 =3D 26, + GFX10_S_NOR_B64 =3D 27, + GFX10_S_XNOR_B32 =3D 28, + GFX10_S_XNOR_B64 =3D 29, + GFX10_S_LSHL_B32 =3D 30, + GFX10_S_LSHL_B64 =3D 31, + GFX10_S_LSHR_B32 =3D 32, + GFX10_S_LSHR_B64 =3D 33, + GFX10_S_ASHR_I32 =3D 34, + GFX10_S_ASHR_I64 =3D 35, + GFX10_S_BFM_B32 =3D 36, + GFX10_S_BFM_B64 =3D 37, + GFX10_S_MUL_I32 =3D 38, + GFX10_S_MUL_I64 =3D 39, + GFX10_S_BFE_U32 =3D 40, + GFX10_S_BFE_I32 =3D 41, + GFX10_S_BFE_U64 =3D 42, + GFX10_S_ABSDIFF_I32 =3D 44, + GFX10_S_LSHL1_ADD_U32 =3D 46, + GFX10_S_LSHL2_ADD_U32 =3D 47, + GFX10_S_LSHL3_ADD_U32 =3D 48, + GFX10_S_LSHL4_ADD_U32 =3D 49, + GFX10_S_PACK_LL_B32_B16 =3D 50, + GFX10_S_PACK_LH_B32_B16 =3D 51, + GFX10_S_PACK_HH_B32_B16 =3D 52, + GFX10_S_MUL_HI_U32 =3D 53, + GFX10_S_MUL_HI_I32 =3D 54, +}; + +#define GFX10_SOP2_ENCODING 0x2 +#define GFX10_SOP2_SSRC_SGPR_BASE 0 +#define GFX10_SOP2_SSRC_VCC_LO 106 +#define GFX10_SOP2_SSRC_VCC_HI 107 +#define GFX10_SOP2_SSRC_TTPM_BASE 108 +#define GFX10_SOP2_SSRC_M0 124 +#define GFX10_SOP2_SSRC_NULL 125 +#define GFX10_SOP2_SSRC_EXEC_LO 126 +#define GFX10_SOP2_SSRC_EXEC_HI 127 +#define GFX10_SOP2_SSRC_INTEGER_0 128 +#define GFX10_SOP2_SSRC_INTEGER_MINUS_1 193 +#define GFX10_SOP2_SSRC_SHARED_BASE 235 +#define GFX10_SOP2_SSRC_SHARED_LIMIT 236 +#define GFX10_SOP2_SSRC_PRIVATE_BASE 237 +#define GFX10_SOP2_SSRC_PRIVATE_LIMIT 238 +#define GFX10_SOP2_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_SOP2_SSRC_VCCZ 251 +#define GFX10_SOP2_SSRC_EXECZ 252 +#define GFX10_SOP2_SSRC_SCC 253 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx10_sopk { + u32 simm16:16; + u32 sdst:7; + u32 op:5; + u32 encoding:4; +}; + +enum amdgcn_gfx10_sopk_opcode { + GFX10_S_MOVK_I32 =3D 0, + GFX10_S_VERSION =3D 1, + GFX10_S_CMOVK_I32 =3D 2, + GFX10_S_CMPK_EQ_I32 =3D 3, + GFX10_S_CMPK_LG_I32 =3D 4, + GFX10_S_CMPK_GT_I32 =3D 5, + GFX10_S_CMPK_GE_I32 =3D 6, + GFX10_S_CMPK_LT_I32 =3D 7, + GFX10_S_CMPK_LE_I32 =3D 8, + GFX10_S_CMPK_EQ_U32 =3D 9, + GFX10_S_CMPK_LG_U32 =3D 10, + GFX10_S_CMPK_GT_U32 =3D 11, + GFX10_S_CMPK_GE_U32 =3D 12, + GFX10_S_CMPK_LT_U32 =3D 13, + GFX10_S_CMPK_LE_U32 =3D 14, + GFX10_S_ADDK_I32 =3D 15, + GFX10_S_MULK_I32 =3D 16, + /* 17: reserved */ + GFX10_S_GETREG_B32 =3D 18, + GFX10_S_SETREG_B32 =3D 19, + /* 20: reserved */ + GFX10_S_SETREG_IMM32_B32 =3D 21, + GFX10_S_CALL_B64 =3D 22, + GFX10_S_WAITCNT_VSCNT =3D 23, + GFX10_S_WAITCNT_VMCNT =3D 24, + GFX10_S_WAITCNT_EXPCNT =3D 25, + GFX10_S_WAITCNT_LGKMCNT =3D 26, + GFX10_S_SUBVECTOR_LOOP_BEGIN =3D 27, + GFX10_S_SUBVECTOR_LOOP_END =3D 28, +}; + +#define GFX10_SOPK_ENCODING 0xb +#define GFX10_SOPK_SDST_SGPR0_BASE 0 +#define GFX10_SOPK_SDST_VCC_LO 106 +#define GFX10_SOPK_SDST_VCC_HI 107 +#define GFX10_SOPK_SDST_TTPM_BASE 108 +#define GFX10_SOPK_SDST_M0 124 +#define GFX10_SOPK_SDST_NULL 125 +#define GFX10_SOPK_SDST_EXEC_LO 126 +#define GFX10_SOPK_SDST_EXEC_HI 127 + +struct amdgcn_gfx10_sop1 { + u32 ssrc0:8; + u32 op:8; + u32 sdst:7; + u32 encoding:9; + u32 literal; +}; + +enum amdgcn_gfx10_sop1_opcode { + GFX10_S_MOV_B32 =3D 3, + GFX10_S_MOV_B64 =3D 4, + GFX10_S_CMOV_B32 =3D 5, + GFX10_S_CMOV_B64 =3D 6, + GFX10_S_NOT_B32 =3D 7, + GFX10_S_NOT_B64 =3D 8, + GFX10_S_WQM_B32 =3D 9, + GFX10_S_WQM_B64 =3D 10, + GFX10_S_BREV_B32 =3D 11, + GFX10_S_BREV_B64 =3D 12, + GFX10_S_BCNT0_I32_B32 =3D 13, + GFX10_S_BCNT0_I32_B64 =3D 14, + GFX10_S_BCNT1_I32_B32 =3D 15, + GFX10_S_BCNT1_I32_B64 =3D 16, + GFX10_S_FF0_I32_B32 =3D 17, + GFX10_S_FF0_I32_B64 =3D 18, + GFX10_S_FF1_I32_B32 =3D 19, + GFX10_S_FF1_I32_B64 =3D 20, + GFX10_S_FLBIT_I32_B32 =3D 21, + GFX10_S_FLBIT_I32_B64 =3D 22, + GFX10_S_FLBIT_I32 =3D 23, + GFX10_S_FLBIT_I32_I64 =3D 24, + GFX10_S_SEXT_I32_I8 =3D 25, + GFX10_S_SEXT_I32_I16 =3D 26, + GFX10_S_BITSET0_B32 =3D 27, + GFX10_S_BITSET0_B64 =3D 28, + GFX10_S_BITSET1_B32 =3D 29, + GFX10_S_BITSET1_B64 =3D 30, + GFX10_S_GETPC_B64 =3D 31, + GFX10_S_SETPC_B64 =3D 32, + GFX10_S_SWAPPC_B64 =3D 33, + GFX10_S_RFE_B64 =3D 34, + /* 35: reserved */ + GFX10_S_AND_SAVEEXEC_B64 =3D 36, + /* 37-42: OR/XOR/ANDN2/ORN2/NAND/NOR_SAVEEXEC_B64 (unused) */ + GFX10_S_XNOR_SAVEEXEC_B64 =3D 43, + GFX10_S_QUADMASK_B32 =3D 44, + GFX10_S_QUADMASK_B64 =3D 45, + GFX10_S_MOVRELS_B32 =3D 46, + GFX10_S_MOVRELS_B64 =3D 47, + GFX10_S_MOVRELD_B32 =3D 48, + GFX10_S_MOVRELD_B64 =3D 49, + /* 50-51: reserved */ + GFX10_S_ABS_I32 =3D 52, + /* 53-54: reserved */ + GFX10_S_ANDN1_SAVEEXEC_B64 =3D 55, + GFX10_S_ORN1_SAVEEXEC_B64 =3D 56, + GFX10_S_ANDN1_WREXEC_B64 =3D 57, + GFX10_S_ANDN2_WREXEC_B64 =3D 58, + GFX10_S_BITREPLICATE_B64_B32 =3D 59, + GFX10_S_AND_SAVEEXEC_B32 =3D 60, + GFX10_S_OR_SAVEEXEC_B32 =3D 61, + GFX10_S_XOR_SAVEEXEC_B32 =3D 62, + GFX10_S_ANDN2_SAVEEXEC_B32 =3D 63, + GFX10_S_ORN2_SAVEEXEC_B32 =3D 64, + GFX10_S_NAND_SAVEEXEC_B32 =3D 65, + GFX10_S_NOR_SAVEEXEC_B32 =3D 66, + GFX10_S_XNOR_SAVEEXEC_B32 =3D 67, + GFX10_S_ANDN1_SAVEEXEC_B32 =3D 68, + GFX10_S_ORN1_SAVEEXEC_B32 =3D 69, + GFX10_S_ANDN1_WREXEC_B32 =3D 70, + GFX10_S_ANDN2_WREXEC_B32 =3D 71, + /* 72: reserved */ + GFX10_S_MOVRELSD_2_B32 =3D 73, +}; + +#define GFX10_SOP1_ENCODING 0x17d +#define GFX10_SOP1_SSRC_SGPR_BASE 0 +#define GFX10_SOP1_SSRC_VCC_LO 106 +#define GFX10_SOP1_SSRC_VCC_HI 107 +#define GFX10_SOP1_SSRC_TTPM_BASE 108 +#define GFX10_SOP1_SSRC_M0 124 +#define GFX10_SOP1_SSRC_NULL 125 +#define GFX10_SOP1_SSRC_EXEC_LO 126 +#define GFX10_SOP1_SSRC_EXEC_HI 127 +#define GFX10_SOP1_SSRC_INTEGER_0 128 +#define GFX10_SOP1_SSRC_INTEGER_MINUS_1 193 +#define GFX10_SOP1_SSRC_SHARED_BASE 235 +#define GFX10_SOP1_SSRC_SHARED_LIMIT 236 +#define GFX10_SOP1_SSRC_PRIVATE_BASE 237 +#define GFX10_SOP1_SSRC_PRIVATE_LIMIT 238 +#define GFX10_SOP1_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_SOP1_SSRC_VCCZ 251 +#define GFX10_SOP1_SSRC_EXECZ 252 +#define GFX10_SOP1_SSRC_SCC 253 +#define GFX10_SOP1_SSRC_LITERAL_CONST 255 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx10_sopc { + u32 ssrc0:8; + u32 ssrc1:8; + u32 op:7; + u32 encoding:9; + u32 literal; +}; + +enum amdgcn_gfx10_sopc_opcode { + GFX10_S_CMP_EQ_I32 =3D 0, + GFX10_S_CMP_LG_I32 =3D 1, + GFX10_S_CMP_GT_I32 =3D 2, + GFX10_S_CMP_GE_I32 =3D 3, + GFX10_S_CMP_LT_I32 =3D 4, + GFX10_S_CMP_LE_I32 =3D 5, + GFX10_S_CMP_EQ_U32 =3D 6, + GFX10_S_CMP_LG_U32 =3D 7, + GFX10_S_CMP_GT_U32 =3D 8, + GFX10_S_CMP_GE_U32 =3D 9, + GFX10_S_CMP_LT_U32 =3D 10, + GFX10_S_CMP_LE_U32 =3D 11, + GFX10_S_BITCMP0_B32 =3D 12, + GFX10_S_BITCMP1_B32 =3D 13, + GFX10_S_BITCMP0_B64 =3D 14, + GFX10_S_BITCMP1_B64 =3D 15, + /* 16-17: reserved */ + GFX10_S_CMP_EQ_U64 =3D 18, + GFX10_S_CMP_LG_U64 =3D 19, +}; + +#define GFX10_SOPC_ENCODING 0x17e +#define GFX10_SOPC_SSRC_SGPR_BASE 0 +#define GFX10_SOPC_SSRC_VCC_LO 106 +#define GFX10_SOPC_SSRC_VCC_HI 107 +#define GFX10_SOPC_SSRC_TTPM_BASE 108 +#define GFX10_SOPC_SSRC_M0 124 +#define GFX10_SOPC_SSRC_NULL 125 +#define GFX10_SOPC_SSRC_EXEC_LO 126 +#define GFX10_SOPC_SSRC_EXEC_HI 127 +#define GFX10_SOPC_SSRC_INTEGER_0 128 +#define GFX10_SOPC_SSRC_INTEGER_MINUS_1 193 +#define GFX10_SOPC_SSRC_SHARED_BASE 235 +#define GFX10_SOPC_SSRC_SHARED_LIMIT 236 +#define GFX10_SOPC_SSRC_PRIVATE_BASE 237 +#define GFX10_SOPC_SSRC_PRIVATE_LIMIT 238 +#define GFX10_SOPC_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_SOPC_SSRC_VCCZ 251 +#define GFX10_SOPC_SSRC_EXECZ 252 +#define GFX10_SOPC_SSRC_SCC 253 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx10_sopp_vmcnt { + u16 vmcnt1:4; + u16 expcnt:3; + u16 dummy:1; + u16 lgkmcnt:6; + u16 vmcnt2:2; +}; + +struct amdgcn_gfx10_sopp { + u32 simm16:16; + u32 op:7; + u32 encoding:9; +}; + +enum amdgcn_gfx10_sopp_opcode { + GFX10_S_NOP =3D 0, + GFX10_S_ENDPGM =3D 1, + GFX10_S_BRANCH =3D 2, + GFX10_S_WAKEUP =3D 3, + GFX10_S_CBRANCH_SCC0 =3D 4, + GFX10_S_CBRANCH_SCC1 =3D 5, + GFX10_S_CBRANCH_VCCZ =3D 6, + GFX10_S_CBRANCH_VCCNZ =3D 7, + GFX10_S_CBRANCH_EXECZ =3D 8, + GFX10_S_CBRANCH_EXECNZ =3D 9, + GFX10_S_BARRIER =3D 10, + GFX10_S_SETKILL =3D 11, + GFX10_S_WAITCNT =3D 12, + GFX10_S_SETHALT =3D 13, + GFX10_S_SLEEP =3D 14, + GFX10_S_SETPRIO =3D 15, + GFX10_S_SENDMSG =3D 16, + GFX10_S_SENDMSGHALT =3D 17, + GFX10_S_TRAP =3D 18, + GFX10_S_ICACHE_INV =3D 19, + GFX10_S_INCPERFLEVEL =3D 20, + GFX10_S_DECPERFLEVEL =3D 21, + GFX10_S_TTRACEDATA =3D 22, + GFX10_S_CBRANCH_CDBGSYS =3D 23, + GFX10_S_CBRANCH_CDBGUSER =3D 24, + GFX10_S_CBRANCH_CDBGSYS_OR_USER =3D 25, + GFX10_S_CBRANCH_CDBGSYS_AND_USER =3D 26, + GFX10_S_ENDPGM_SAVED =3D 27, + /* 28-29: reserved */ + GFX10_S_ENDPGM_ORDERED_PS_DONE =3D 30, + GFX10_S_CODE_END =3D 31, + GFX10_S_INST_PREFETCH =3D 32, + GFX10_S_CLAUSE =3D 33, + /* 34: reserved */ + GFX10_S_WAITCNT_DEPCTR =3D 35, + GFX10_S_ROUND_MODE =3D 36, + GFX10_S_DENORM_MODE =3D 37, + /* 38-39: reserved */ + GFX10_S_TTRACEDATA_IMM =3D 40, +}; + +#define GFX10_SOPP_ENCODING 0x17f + +/* Scalar Memory Format */ +struct amdgcn_gfx10_smem { + u64 sbase:6; + u64 sdata:7; + u64 dummy1:1; + u64 dlc:1; + u64 dummy2:1; + u64 glc:1; + u64 dummy3:1; + u64 op:8; + u64 encoding:6; + u64 offset:21; + u64 dummy4:4; + u64 soffset:7; +}; + +enum amdgcn_gfx10_smem_opcode { + GFX10_S_LOAD_DWORD =3D 0, + GFX10_S_LOAD_DWORDX2 =3D 1, + GFX10_S_LOAD_DWORDX4 =3D 2, + GFX10_S_LOAD_DWORDX8 =3D 3, + GFX10_S_LOAD_DWORDX16 =3D 4, + /* 5-7: reserved */ + GFX10_S_BUFFER_LOAD_DWORD =3D 8, + GFX10_S_BUFFER_LOAD_DWORDX2 =3D 9, + GFX10_S_BUFFER_LOAD_DWORDX4 =3D 10, + GFX10_S_BUFFER_LOAD_DWORDX8 =3D 11, + GFX10_S_BUFFER_LOAD_DWORDX16 =3D 12, + /* 13-30: reserved */ + GFX10_S_GL1_INV =3D 31, + GFX10_S_DCACHE_INV =3D 32, + /* 33-35: reserved */ + GFX10_S_MEMTIME =3D 36, + GFX10_S_MEMREALTIME =3D 37, + GFX10_S_ATC_PROBE =3D 38, + GFX10_S_ATC_PROBE_BUFFER =3D 39, +}; + +#define GFX10_SMEM_ENCODING 0x3d +#define GFX10_SMEM_SOFFSET_NULL 125 + +/* Vector ALU Format */ +struct amdgcn_gfx10_vop2 { + u32 src0:9; + u32 vsrc1:8; + u32 vdst:8; + u32 op:6; + u32 encoding:1; + u32 literal; +}; + +enum amdgcn_gfx10_vop2_opcode { + /* 0: reserved */ + GFX10_V_CNDMASK_B32 =3D 1, + GFX10_V_DOT2C_F32_F16 =3D 2, + GFX10_V_ADD_F32 =3D 3, + GFX10_V_SUB_F32 =3D 4, + GFX10_V_SUBREV_F32 =3D 5, + GFX10_V_FMAC_LEGACY_F32 =3D 6, + GFX10_V_MUL_LEGACY_F32 =3D 7, + GFX10_V_MUL_F32 =3D 8, + GFX10_V_MUL_I32_I24 =3D 9, + GFX10_V_MUL_HI_I32_I24 =3D 10, + GFX10_V_MUL_U32_U24 =3D 11, + GFX10_V_MUL_HI_U32_U24 =3D 12, + GFX10_V_DOT4C_I32_I8 =3D 13, + /* 14: reserved */ + GFX10_V_MIN_F32 =3D 15, + GFX10_V_MAX_F32 =3D 16, + GFX10_V_MIN_I32 =3D 17, + GFX10_V_MAX_I32 =3D 18, + GFX10_V_MIN_U32 =3D 19, + GFX10_V_MAX_U32 =3D 20, + /* 21: reserved */ + GFX10_V_LSHRREV_B32 =3D 22, + /* 23: reserved */ + GFX10_V_ASHRREV_I32 =3D 24, + /* 25: reserved */ + GFX10_V_LSHLREV_B32 =3D 26, + GFX10_V_AND_B32 =3D 27, + GFX10_V_OR_B32 =3D 28, + GFX10_V_XOR_B32 =3D 29, + GFX10_V_XNOR_B32 =3D 30, + /* 31-36: reserved */ + GFX10_V_ADD_NC_U32 =3D 37, + GFX10_V_SUB_NC_U32 =3D 38, + GFX10_V_SUBREV_NC_U32 =3D 39, + GFX10_V_ADD_CO_CI_U32 =3D 40, + GFX10_V_SUB_CO_CI_U32 =3D 41, + GFX10_V_SUBREV_CO_CI_U32 =3D 42, + GFX10_V_FMAC_F32 =3D 43, + GFX10_V_FMAMK_F32 =3D 44, + GFX10_V_FMAAK_F32 =3D 45, + /* 46: reserved */ + GFX10_V_CVT_PKRTZ_F16_F32 =3D 47, + /* 48-49: reserved */ + GFX10_V_ADD_F16 =3D 50, + GFX10_V_SUB_F16 =3D 51, + GFX10_V_SUBREV_F16 =3D 52, + GFX10_V_MUL_F16 =3D 53, + GFX10_V_FMAC_F16 =3D 54, + GFX10_V_FMAMK_F16 =3D 55, + GFX10_V_FMAAK_F16 =3D 56, + GFX10_V_MAX_F16 =3D 57, + GFX10_V_MIN_F16 =3D 58, + GFX10_V_LDEXP_F16 =3D 59, + GFX10_V_PK_FMAC_F16 =3D 60, +}; + +#define GFX10_VOP2_ENCODING 0x0 +#define GFX10_VOP2_SRC_SGPR_BASE 0 +#define GFX10_VOP2_SRC_VCC_LO 106 +#define GFX10_VOP2_SRC_VCC_HI 107 +#define GFX10_VOP2_SRC_TTPM_BASE 108 +#define GFX10_VOP2_SRC_M0 124 +#define GFX10_VOP2_SRC_NULL 125 +#define GFX10_VOP2_SRC_EXEC_LO 126 +#define GFX10_VOP2_SRC_EXEC_HI 127 +#define GFX10_VOP2_SRC_INTEGER_0 128 +#define GFX10_VOP2_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOP2_SRC_SHARED_BASE 235 +#define GFX10_VOP2_SRC_SHARED_LIMIT 236 +#define GFX10_VOP2_SRC_PRIVATE_BASE 237 +#define GFX10_VOP2_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOP2_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOP2_SRC_SDWA 249 +#define GFX10_VOP2_SRC_DDP16 250 +#define GFX10_VOP2_SRC_VCCZ 251 +#define GFX10_VOP2_SRC_EXECZ 252 +#define GFX10_VOP2_SRC_SCC 253 +#define GFX10_VOP2_SRC_LITERAL_CONST 255 +#define GFX10_VOP2_SRC_VGPR_BASE 256 + +struct amdgcn_gfx10_vop1 { + u32 src0:9; + u32 op:8; + u32 vdst:8; + u32 encoding:7; + u32 literal; +}; + +enum amdgcn_gfx10_vop1_opcode { + GFX10_V_NOP =3D 0, + GFX10_V_MOV_B32 =3D 1, + GFX10_V_READFIRSTLANE_B32 =3D 2, + GFX10_V_CVT_I32_F64 =3D 3, + GFX10_V_CVT_F64_I32 =3D 4, + GFX10_V_CVT_F32_I32 =3D 5, + GFX10_V_CVT_F32_U32 =3D 6, + GFX10_V_CVT_U32_F32 =3D 7, + GFX10_V_CVT_I32_F32 =3D 8, + /* 9: reserved */ + GFX10_V_CVT_F16_F32 =3D 10, + GFX10_V_CVT_F32_F16 =3D 11, + GFX10_V_CVT_RPI_I32_F32 =3D 12, + GFX10_V_CVT_FLR_I32_F32 =3D 13, + GFX10_V_CVT_OFF_F32_I4 =3D 14, + GFX10_V_CVT_F32_F64 =3D 15, + GFX10_V_CVT_F64_F32 =3D 16, + GFX10_V_CVT_F32_UBYTE0 =3D 17, + GFX10_V_CVT_F32_UBYTE1 =3D 18, + GFX10_V_CVT_F32_UBYTE2 =3D 19, + GFX10_V_CVT_F32_UBYTE3 =3D 20, + GFX10_V_CVT_U32_F64 =3D 21, + GFX10_V_CVT_F64_U32 =3D 22, + GFX10_V_TRUNC_F64 =3D 23, + GFX10_V_CEIL_F64 =3D 24, + GFX10_V_RNDNE_F64 =3D 25, + GFX10_V_FLOOR_F64 =3D 26, + GFX10_V_PIPEFLUSH =3D 27, + /* 28-31: reserved */ + GFX10_V_FRACT_F32 =3D 32, + GFX10_V_TRUNC_F32 =3D 33, + GFX10_V_CEIL_F32 =3D 34, + GFX10_V_RNDNE_F32 =3D 35, + GFX10_V_FLOOR_F32 =3D 36, + GFX10_V_EXP_F32 =3D 37, + /* 38: reserved */ + GFX10_V_LOG_F32 =3D 39, + /* 40-41: reserved */ + GFX10_V_RCP_F32 =3D 42, + GFX10_V_RCP_IFLAG_F32 =3D 43, + /* 44-45: reserved */ + GFX10_V_RSQ_F32 =3D 46, + GFX10_V_RCP_F64 =3D 47, + /* 48: reserved */ + GFX10_V_RSQ_F64 =3D 49, + /* 50: reserved */ + GFX10_V_SQRT_F32 =3D 51, + GFX10_V_SQRT_F64 =3D 52, + GFX10_V_SIN_F32 =3D 53, + GFX10_V_COS_F32 =3D 54, + GFX10_V_NOT_B32 =3D 55, + GFX10_V_BFREV_B32 =3D 56, + GFX10_V_FFBH_U32 =3D 57, + GFX10_V_FFBL_B32 =3D 58, + GFX10_V_FFBH_I32 =3D 59, + GFX10_V_FREXP_EXP_I32_F64 =3D 60, + GFX10_V_FREXP_MANT_F64 =3D 61, + GFX10_V_FRACT_F64 =3D 62, + GFX10_V_FREXP_EXP_I32_F32 =3D 63, + GFX10_V_FREXP_MANT_F32 =3D 64, + GFX10_V_CLREXCP =3D 65, + GFX10_V_MOVRELD_B32 =3D 66, + GFX10_V_MOVRELS_B32 =3D 67, + GFX10_V_MOVRELSD_B32 =3D 68, + /* 69-71: reserved */ + GFX10_V_MOVRELSD_2_B32 =3D 72, + /* 73-79: reserved */ + GFX10_V_CVT_F16_U16 =3D 80, + GFX10_V_CVT_F16_I16 =3D 81, + GFX10_V_CVT_U16_F16 =3D 82, + GFX10_V_CVT_I16_F16 =3D 83, + GFX10_V_RCP_F16 =3D 84, + GFX10_V_SQRT_F16 =3D 85, + GFX10_V_RSQ_F16 =3D 86, + GFX10_V_LOG_F16 =3D 87, + GFX10_V_EXP_F16 =3D 88, + GFX10_V_FREXP_MANT_F16 =3D 89, + GFX10_V_FREXP_EXP_I16_F16 =3D 90, + GFX10_V_FLOOR_F16 =3D 91, + GFX10_V_CEIL_F16 =3D 92, + GFX10_V_TRUNC_F16 =3D 93, + GFX10_V_RNDNE_F16 =3D 94, + GFX10_V_FRACT_F16 =3D 95, + GFX10_V_SIN_F16 =3D 96, + GFX10_V_COS_F16 =3D 97, + GFX10_V_SAT_PK_U8_I16 =3D 98, + GFX10_V_CVT_NORM_I16_F16 =3D 99, + GFX10_V_CVT_NORM_U16_F16 =3D 100, + GFX10_V_SWAP_B32 =3D 101, + /* 102-103: reserved */ + GFX10_V_SWAPREL_B32 =3D 104, +}; + +#define GFX10_VOP1_ENCODING 0x3f +#define GFX10_VOP1_SRC_SGPR_BASE 0 +#define GFX10_VOP1_SRC_VCC_LO 106 +#define GFX10_VOP1_SRC_VCC_HI 107 +#define GFX10_VOP1_SRC_TTPM_BASE 108 +#define GFX10_VOP1_SRC_M0 124 +#define GFX10_VOP1_SRC_NULL 125 +#define GFX10_VOP1_SRC_EXEC_LO 126 +#define GFX10_VOP1_SRC_EXEC_HI 127 +#define GFX10_VOP1_SRC_INTEGER_0 128 +#define GFX10_VOP1_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOP1_SRC_SHARED_BASE 235 +#define GFX10_VOP1_SRC_SHARED_LIMIT 236 +#define GFX10_VOP1_SRC_PRIVATE_BASE 237 +#define GFX10_VOP1_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOP1_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOP1_SRC_SDWA 249 +#define GFX10_VOP1_SRC_DDP16 250 +#define GFX10_VOP1_SRC_VCCZ 251 +#define GFX10_VOP1_SRC_EXECZ 252 +#define GFX10_VOP1_SRC_SCC 253 +#define GFX10_VOP1_SRC_LITERAL_CONST 255 +#define GFX10_VOP1_SRC_VGPR_BASE 256 + +struct amdgcn_gfx10_vopc { + u32 src0:9; + u32 vsrc1:8; + u32 op:8; + u32 encoding:7; + u32 literal; +}; + +enum amdgcn_gfx10_vopc_compare_offset16 { + GFX10_VOPC16_F =3D 0, + GFX10_VOPC16_LT =3D 1, + GFX10_VOPC16_EQ =3D 2, + GFX10_VOPC16_LE =3D 3, + GFX10_VOPC16_GT =3D 4, + GFX10_VOPC16_LG =3D 5, + GFX10_VOPC16_GE =3D 6, + GFX10_VOPC16_O =3D 7, + GFX10_VOPC16_U =3D 8, + GFX10_VOPC16_NGE =3D 9, + GFX10_VOPC16_NLG =3D 10, + GFX10_VOPC16_NGT =3D 11, + GFX10_VOPC16_NLE =3D 12, + GFX10_VOPC16_NEQ =3D 13, + GFX10_VOPC16_NLT =3D 14, + GFX10_VOPC16_TRU =3D 15, +}; + +enum amdgcn_gfx10_vopc_compare_offset8 { + GFX10_VOPC8_F =3D 0, + GFX10_VOPC8_LT =3D 1, + GFX10_VOPC8_EQ =3D 2, + GFX10_VOPC8_LE =3D 3, + GFX10_VOPC8_GT =3D 4, + GFX10_VOPC8_LG =3D 5, + GFX10_VOPC8_GE =3D 6, + GFX10_VOPC8_TRU =3D 7, +}; + +enum amdgcn_gfx10_vopc_opcode { + GFX10_V_CMP_F_F32 =3D 0, + GFX10_V_CMP_LT_F32 =3D 1, + GFX10_V_CMP_EQ_F32 =3D 2, + GFX10_V_CMP_LE_F32 =3D 3, + GFX10_V_CMP_GT_F32 =3D 4, + GFX10_V_CMP_LG_F32 =3D 5, + GFX10_V_CMP_GE_F32 =3D 6, + GFX10_V_CMP_O_F32 =3D 7, + GFX10_V_CMP_U_F32 =3D 8, + GFX10_V_CMP_NGE_F32 =3D 9, + GFX10_V_CMP_NLG_F32 =3D 10, + GFX10_V_CMP_NGT_F32 =3D 11, + GFX10_V_CMP_NLE_F32 =3D 12, + GFX10_V_CMP_NEQ_F32 =3D 13, + GFX10_V_CMP_NLT_F32 =3D 14, + GFX10_V_CMP_TRU_F32 =3D 15, + GFX10_V_CMPX_F_F32 =3D 16, + GFX10_V_CMPX_LT_F32 =3D 17, + GFX10_V_CMPX_EQ_F32 =3D 18, + GFX10_V_CMPX_LE_F32 =3D 19, + GFX10_V_CMPX_GT_F32 =3D 20, + GFX10_V_CMPX_LG_F32 =3D 21, + GFX10_V_CMPX_GE_F32 =3D 22, + GFX10_V_CMPX_O_F32 =3D 23, + GFX10_V_CMPX_U_F32 =3D 24, + GFX10_V_CMPX_NGE_F32 =3D 25, + GFX10_V_CMPX_NLG_F32 =3D 26, + GFX10_V_CMPX_NGT_F32 =3D 27, + GFX10_V_CMPX_NLE_F32 =3D 28, + GFX10_V_CMPX_NEQ_F32 =3D 29, + GFX10_V_CMPX_NLT_F32 =3D 30, + GFX10_V_CMPX_TRU_F32 =3D 31, + GFX10_V_CMP_F_F64 =3D 32, + GFX10_V_CMP_LT_F64 =3D 33, + GFX10_V_CMP_EQ_F64 =3D 34, + GFX10_V_CMP_LE_F64 =3D 35, + GFX10_V_CMP_GT_F64 =3D 36, + GFX10_V_CMP_LG_F64 =3D 37, + GFX10_V_CMP_GE_F64 =3D 38, + GFX10_V_CMP_O_F64 =3D 39, + GFX10_V_CMP_U_F64 =3D 40, + GFX10_V_CMP_NGE_F64 =3D 41, + GFX10_V_CMP_NLG_F64 =3D 42, + GFX10_V_CMP_NGT_F64 =3D 43, + GFX10_V_CMP_NLE_F64 =3D 44, + GFX10_V_CMP_NEQ_F64 =3D 45, + GFX10_V_CMP_NLT_F64 =3D 46, + GFX10_V_CMP_TRU_F64 =3D 47, + GFX10_V_CMPX_F_F64 =3D 48, + GFX10_V_CMPX_LT_F64 =3D 49, + GFX10_V_CMPX_EQ_F64 =3D 50, + GFX10_V_CMPX_LE_F64 =3D 51, + GFX10_V_CMPX_GT_F64 =3D 52, + GFX10_V_CMPX_LG_F64 =3D 53, + GFX10_V_CMPX_GE_F64 =3D 54, + GFX10_V_CMPX_O_F64 =3D 55, + GFX10_V_CMPX_U_F64 =3D 56, + GFX10_V_CMPX_NGE_F64 =3D 57, + GFX10_V_CMPX_NLG_F64 =3D 58, + GFX10_V_CMPX_NGT_F64 =3D 59, + GFX10_V_CMPX_NLE_F64 =3D 60, + GFX10_V_CMPX_NEQ_F64 =3D 61, + GFX10_V_CMPX_NLT_F64 =3D 62, + GFX10_V_CMPX_TRU_F64 =3D 63, + /* 64-127: reserved */ + GFX10_V_CMP_F_I32 =3D 128, + GFX10_V_CMP_LT_I32 =3D 129, + GFX10_V_CMP_EQ_I32 =3D 130, + GFX10_V_CMP_LE_I32 =3D 131, + GFX10_V_CMP_GT_I32 =3D 132, + GFX10_V_CMP_NE_I32 =3D 133, + GFX10_V_CMP_GE_I32 =3D 134, + GFX10_V_CMP_T_I32 =3D 135, + GFX10_V_CMP_CLASS_F32 =3D 136, + GFX10_V_CMP_LT_I16 =3D 137, + GFX10_V_CMP_EQ_I16 =3D 138, + GFX10_V_CMP_LE_I16 =3D 139, + GFX10_V_CMP_GT_I16 =3D 140, + GFX10_V_CMP_NE_I16 =3D 141, + GFX10_V_CMP_GE_I16 =3D 142, + GFX10_V_CMP_CLASS_F16 =3D 143, + GFX10_V_CMPX_F_I32 =3D 144, + GFX10_V_CMPX_LT_I32 =3D 145, + GFX10_V_CMPX_EQ_I32 =3D 146, + GFX10_V_CMPX_LE_I32 =3D 147, + GFX10_V_CMPX_GT_I32 =3D 148, + GFX10_V_CMPX_NE_I32 =3D 149, + GFX10_V_CMPX_GE_I32 =3D 150, + GFX10_V_CMPX_T_I32 =3D 151, + GFX10_V_CMPX_CLASS_F32 =3D 152, + GFX10_V_CMPX_LT_I16 =3D 153, + GFX10_V_CMPX_EQ_I16 =3D 154, + GFX10_V_CMPX_LE_I16 =3D 155, + GFX10_V_CMPX_GT_I16 =3D 156, + GFX10_V_CMPX_NE_I16 =3D 157, + GFX10_V_CMPX_GE_I16 =3D 158, + GFX10_V_CMPX_CLASS_F16 =3D 159, + GFX10_V_CMP_F_I64 =3D 160, + GFX10_V_CMP_LT_I64 =3D 161, + GFX10_V_CMP_EQ_I64 =3D 162, + GFX10_V_CMP_LE_I64 =3D 163, + GFX10_V_CMP_GT_I64 =3D 164, + GFX10_V_CMP_NE_I64 =3D 165, + GFX10_V_CMP_GE_I64 =3D 166, + GFX10_V_CMP_T_I64 =3D 167, + GFX10_V_CMP_CLASS_F64 =3D 168, + GFX10_V_CMP_LT_U16 =3D 169, + GFX10_V_CMP_EQ_U16 =3D 170, + GFX10_V_CMP_LE_U16 =3D 171, + GFX10_V_CMP_GT_U16 =3D 172, + GFX10_V_CMP_NE_U16 =3D 173, + GFX10_V_CMP_GE_U16 =3D 174, + /* 175: reserved */ + GFX10_V_CMPX_F_I64 =3D 176, + GFX10_V_CMPX_LT_I64 =3D 177, + GFX10_V_CMPX_EQ_I64 =3D 178, + GFX10_V_CMPX_LE_I64 =3D 179, + GFX10_V_CMPX_GT_I64 =3D 180, + GFX10_V_CMPX_NE_I64 =3D 181, + GFX10_V_CMPX_GE_I64 =3D 182, + GFX10_V_CMPX_T_I64 =3D 183, + GFX10_V_CMPX_CLASS_F64 =3D 184, + GFX10_V_CMPX_LT_U16 =3D 185, + GFX10_V_CMPX_EQ_U16 =3D 186, + GFX10_V_CMPX_LE_U16 =3D 187, + GFX10_V_CMPX_GT_U16 =3D 188, + GFX10_V_CMPX_NE_U16 =3D 189, + GFX10_V_CMPX_GE_U16 =3D 190, + /* 191: reserved */ + GFX10_V_CMP_F_U32 =3D 192, + GFX10_V_CMP_LT_U32 =3D 193, + GFX10_V_CMP_EQ_U32 =3D 194, + GFX10_V_CMP_LE_U32 =3D 195, + GFX10_V_CMP_GT_U32 =3D 196, + GFX10_V_CMP_NE_U32 =3D 197, + GFX10_V_CMP_GE_U32 =3D 198, + GFX10_V_CMP_T_U32 =3D 199, + GFX10_V_CMP_F_F16 =3D 200, + GFX10_V_CMP_LT_F16 =3D 201, + GFX10_V_CMP_EQ_F16 =3D 202, + GFX10_V_CMP_LE_F16 =3D 203, + GFX10_V_CMP_GT_F16 =3D 204, + GFX10_V_CMP_LG_F16 =3D 205, + GFX10_V_CMP_GE_F16 =3D 206, + GFX10_V_CMP_O_F16 =3D 207, + GFX10_V_CMPX_F_U32 =3D 208, + GFX10_V_CMPX_LT_U32 =3D 209, + GFX10_V_CMPX_EQ_U32 =3D 210, + GFX10_V_CMPX_LE_U32 =3D 211, + GFX10_V_CMPX_GT_U32 =3D 212, + GFX10_V_CMPX_NE_U32 =3D 213, + GFX10_V_CMPX_GE_U32 =3D 214, + GFX10_V_CMPX_T_U32 =3D 215, + GFX10_V_CMPX_F_F16 =3D 216, + GFX10_V_CMPX_LT_F16 =3D 217, + GFX10_V_CMPX_EQ_F16 =3D 218, + GFX10_V_CMPX_LE_F16 =3D 219, + GFX10_V_CMPX_GT_F16 =3D 220, + GFX10_V_CMPX_LG_F16 =3D 221, + GFX10_V_CMPX_GE_F16 =3D 222, + GFX10_V_CMPX_O_F16 =3D 223, + GFX10_V_CMP_F_U64 =3D 224, + GFX10_V_CMP_LT_U64 =3D 225, + GFX10_V_CMP_EQ_U64 =3D 226, + GFX10_V_CMP_LE_U64 =3D 227, + GFX10_V_CMP_GT_U64 =3D 228, + GFX10_V_CMP_NE_U64 =3D 229, + GFX10_V_CMP_GE_U64 =3D 230, + GFX10_V_CMP_T_U64 =3D 231, + GFX10_V_CMP_U_F16 =3D 232, + GFX10_V_CMP_NGE_F16 =3D 233, + GFX10_V_CMP_NLG_F16 =3D 234, + GFX10_V_CMP_NGT_F16 =3D 235, + GFX10_V_CMP_NLE_F16 =3D 236, + GFX10_V_CMP_NEQ_F16 =3D 237, + GFX10_V_CMP_NLT_F16 =3D 238, + GFX10_V_CMP_TRU_F16 =3D 239, + GFX10_V_CMPX_F_U64 =3D 240, + GFX10_V_CMPX_LT_U64 =3D 241, + GFX10_V_CMPX_EQ_U64 =3D 242, + GFX10_V_CMPX_LE_U64 =3D 243, + GFX10_V_CMPX_GT_U64 =3D 244, + GFX10_V_CMPX_NE_U64 =3D 245, + GFX10_V_CMPX_GE_U64 =3D 246, + GFX10_V_CMPX_T_U64 =3D 247, + GFX10_V_CMPX_U_F16 =3D 248, + GFX10_V_CMPX_NGE_F16 =3D 249, + GFX10_V_CMPX_NLG_F16 =3D 250, + GFX10_V_CMPX_NGT_F16 =3D 251, + GFX10_V_CMPX_NLE_F16 =3D 252, + GFX10_V_CMPX_NEQ_F16 =3D 253, + GFX10_V_CMPX_NLT_F16 =3D 254, + GFX10_V_CMPX_TRU_F16 =3D 255, +}; + +#define GFX10_VOPC_ENCODING 0x3e +#define GFX10_VOPC_SRC_SGPR_BASE 0 +#define GFX10_VOPC_SRC_VCC_LO 106 +#define GFX10_VOPC_SRC_VCC_HI 107 +#define GFX10_VOPC_SRC_TTPM_BASE 108 +#define GFX10_VOPC_SRC_M0 124 +#define GFX10_VOPC_SRC_NULL 125 +#define GFX10_VOPC_SRC_EXEC_LO 126 +#define GFX10_VOPC_SRC_EXEC_HI 127 +#define GFX10_VOPC_SRC_INTEGER_0 128 +#define GFX10_VOPC_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOPC_SRC_SHARED_BASE 235 +#define GFX10_VOPC_SRC_SHARED_LIMIT 236 +#define GFX10_VOPC_SRC_PRIVATE_BASE 237 +#define GFX10_VOPC_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOPC_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOPC_SRC_SDWA 249 +#define GFX10_VOPC_SRC_DDP16 250 +#define GFX10_VOPC_SRC_VCCZ 251 +#define GFX10_VOPC_SRC_EXECZ 252 +#define GFX10_VOPC_SRC_SCC 253 +#define GFX10_VOPC_SRC_LITERAL_CONST 255 +#define GFX10_VOPC_SRC_VGPR_BASE 256 + +struct amdgcn_gfx10_vop3a { + u64 vdst:8; + u64 abs:3; + u64 op_sel:4; + u64 clmp:1; + u64 op:10; + u64 encoding:6; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 omod:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx10_vop3a_opcode { + GFX10_V_FMA_LEGACY_F32 =3D 320, + /* 321: reserved */ + GFX10_V_MAD_I32_I24 =3D 322, + GFX10_V_MAD_U32_U24 =3D 323, + GFX10_V_CUBEID_F32 =3D 324, + GFX10_V_CUBESC_F32 =3D 325, + GFX10_V_CUBETC_F32 =3D 326, + GFX10_V_CUBEMA_F32 =3D 327, + GFX10_V_BFE_U32 =3D 328, + GFX10_V_BFE_I32 =3D 329, + GFX10_V_BFI_B32 =3D 330, + GFX10_V_FMA_F32 =3D 331, + GFX10_V_FMA_F64 =3D 332, + GFX10_V_LERP_U8 =3D 333, + GFX10_V_ALIGNBIT_B32 =3D 334, + GFX10_V_ALIGNBYTE_B32 =3D 335, + GFX10_V_MULLIT_F32 =3D 336, + GFX10_V_MIN3_F32 =3D 337, + GFX10_V_MIN3_I32 =3D 338, + GFX10_V_MIN3_U32 =3D 339, + GFX10_V_MAX3_F32 =3D 340, + GFX10_V_MAX3_I32 =3D 341, + GFX10_V_MAX3_U32 =3D 342, + GFX10_V_MED3_F32 =3D 343, + GFX10_V_MED3_I32 =3D 344, + GFX10_V_MED3_U32 =3D 345, + GFX10_V_SAD_U8 =3D 346, + GFX10_V_SAD_HI_U8 =3D 347, + GFX10_V_SAD_U16 =3D 348, + GFX10_V_SAD_U32 =3D 349, + GFX10_V_CVT_PK_U8_F32 =3D 350, + GFX10_V_DIV_FIXUP_F32 =3D 351, + GFX10_V_DIV_FIXUP_F64 =3D 352, + /* 353-355: reserved */ + GFX10_V_ADD_F64 =3D 356, + GFX10_V_MUL_F64 =3D 357, + GFX10_V_MIN_F64 =3D 358, + GFX10_V_MAX_F64 =3D 359, + GFX10_V_LDEXP_F64 =3D 360, + GFX10_V_MUL_LO_U32 =3D 361, + GFX10_V_MUL_HI_U32 =3D 362, + /* 363: reserved */ + GFX10_V_MUL_HI_I32 =3D 364, + /* 365-366: VOP3B (V_DIV_SCALE_F32/F64) */ + GFX10_V_DIV_FMAS_F32 =3D 367, + GFX10_V_DIV_FMAS_F64 =3D 368, + GFX10_V_MSAD_U8 =3D 369, + GFX10_V_QSAD_PK_U16_U8 =3D 370, + GFX10_V_MQSAD_PK_U16_U8 =3D 371, + GFX10_V_TRIG_PREOP_F64 =3D 372, + GFX10_V_MQSAD_U32_U8 =3D 373, + /* 374-375: VOP3B (V_MAD_U64_U32/I64_I32) */ + GFX10_V_XOR3_B32 =3D 376, + /* 377-766: reserved */ + GFX10_V_LSHLREV_B64 =3D 767, + GFX10_V_LSHRREV_B64 =3D 768, + GFX10_V_ASHRREV_I64 =3D 769, + /* 770: reserved */ + GFX10_V_ADD_NC_U16 =3D 771, + GFX10_V_SUB_NC_U16 =3D 772, + GFX10_V_MUL_LO_U16 =3D 773, + /* 774: reserved */ + GFX10_V_LSHRREV_B16 =3D 775, + GFX10_V_ASHRREV_I16 =3D 776, + GFX10_V_MAX_U16 =3D 777, + GFX10_V_MAX_I16 =3D 778, + GFX10_V_MIN_U16 =3D 779, + GFX10_V_MIN_I16 =3D 780, + GFX10_V_ADD_NC_I16 =3D 781, + GFX10_V_SUB_NC_I16 =3D 782, + /* 783-784: VOP3B (V_ADD_CO_U32/V_SUB_CO_U32) */ + GFX10_V_PACK_B32_F16 =3D 785, + GFX10_V_CVT_PKNORM_I16_F16 =3D 786, + GFX10_V_CVT_PKNORM_U16_F16 =3D 787, + GFX10_V_LSHLREV_B16 =3D 788, + /* 789-792: reserved; 793: VOP3B (V_SUBREV_CO_U32) */ + GFX10_V_MAD_U16 =3D 832, + /* 833: reserved */ + GFX10_V_INTERP_P1LL_F16 =3D 834, + GFX10_V_INTERP_P1LV_F16 =3D 835, + GFX10_V_PERM_B32 =3D 836, + GFX10_V_XAD_U32 =3D 837, + GFX10_V_LSHL_ADD_U32 =3D 838, + GFX10_V_ADD_LSHL_U32 =3D 839, + /* 840-842: reserved */ + GFX10_V_FMA_F16 =3D 843, + /* 844-848: reserved */ + GFX10_V_MIN3_F16 =3D 849, + GFX10_V_MIN3_I16 =3D 850, + GFX10_V_MIN3_U16 =3D 851, + GFX10_V_MAX3_F16 =3D 852, + GFX10_V_MAX3_I16 =3D 853, + GFX10_V_MAX3_U16 =3D 854, + GFX10_V_MED3_F16 =3D 855, + GFX10_V_MED3_I16 =3D 856, + GFX10_V_MED3_U16 =3D 857, + GFX10_V_INTERP_P2_F16 =3D 858, + /* 859-861: reserved */ + GFX10_V_MAD_I16 =3D 862, + GFX10_V_DIV_FIXUP_F16 =3D 863, + GFX10_V_READLANE_B32 =3D 864, + GFX10_V_WRITELANE_B32 =3D 865, + GFX10_V_LDEXP_F32 =3D 866, + GFX10_V_BFM_B32 =3D 867, + GFX10_V_BCNT_U32_B32 =3D 868, + GFX10_V_MBCNT_LO_U32_B32 =3D 869, + GFX10_V_MBCNT_HI_U32_B32 =3D 870, + /* 871: reserved */ + GFX10_V_CVT_PKNORM_I16_F32 =3D 872, + GFX10_V_CVT_PKNORM_U16_F32 =3D 873, + GFX10_V_CVT_PK_U16_U32 =3D 874, + GFX10_V_CVT_PK_I16_I32 =3D 875, + /* 876: reserved */ + GFX10_V_ADD3_U32 =3D 877, + /* 878: reserved */ + GFX10_V_LSHL_OR_B32 =3D 879, + /* 880: reserved */ + GFX10_V_AND_OR_B32 =3D 881, + GFX10_V_OR3_B32 =3D 882, + GFX10_V_MAD_U32_U16 =3D 883, + /* 884: reserved */ + GFX10_V_MAD_I32_I16 =3D 885, + GFX10_V_SUB_NC_I32 =3D 886, + GFX10_V_PERMLANE16_B32 =3D 887, + GFX10_V_PERMLANEX16_B32 =3D 888, + /* 889-894: reserved */ + GFX10_V_ADD_NC_I32 =3D 895, +}; + +enum amdgcn_gfx10_vop3a_abs { + GFX10_VOP3A_ABS_SRC0, + GFX10_VOP3A_ABS_SRC1, + GFX10_VOP3A_ABS_SRC2, +}; + +#define GFX10_VOP3A_ENCODING 0x35 +#define GFX10_VOP3A_SRC_SGPR_BASE 0 +#define GFX10_VOP3A_SRC_VCC_LO 106 +#define GFX10_VOP3A_SRC_VCC_HI 107 +#define GFX10_VOP3A_SRC_TTPM_BASE 108 +#define GFX10_VOP3A_SRC_M0 124 +#define GFX10_VOP3A_SRC_NULL 125 +#define GFX10_VOP3A_SRC_EXEC_LO 126 +#define GFX10_VOP3A_SRC_EXEC_HI 127 +#define GFX10_VOP3A_SRC_INTEGER_0 128 +#define GFX10_VOP3A_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOP3A_SRC_SHARED_BASE 235 +#define GFX10_VOP3A_SRC_SHARED_LIMIT 236 +#define GFX10_VOP3A_SRC_PRIVATE_BASE 237 +#define GFX10_VOP3A_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOP3A_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOP3A_SRC_SDWA 249 +#define GFX10_VOP3A_SRC_DDP16 250 +#define GFX10_VOP3A_SRC_VCCZ 251 +#define GFX10_VOP3A_SRC_EXECZ 252 +#define GFX10_VOP3A_SRC_SCC 253 +#define GFX10_VOP3A_SRC_LITERAL_CONST 255 +#define GFX10_VOP3A_SRC_VGPR_BASE 256 + +/* + * Two-source VOP3 forms have reserved third-source bits on gfx10. Encoding + * an inline integer zero there executes, but LLVM/RGP reject the instruct= ion. + */ +#define GFX10_VOP3_UNUSED_SRC 0 + +struct amdgcn_gfx10_vop3b { + u64 vdst:8; + u64 sdst:7; + u64 clmp:1; + u64 op:10; + u64 encoding:6; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 omod:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx10_vop3b_opcode { + GFX10_V_DIV_SCALE_F32 =3D 365, + GFX10_V_DIV_SCALE_F64 =3D 366, + GFX10_V_MAD_U64_U32 =3D 374, + GFX10_V_MAD_I64_I32 =3D 375, + GFX10_V_ADD_CO_U32 =3D 783, + GFX10_V_SUB_CO_U32 =3D 784, + GFX10_V_SUBREV_CO_U32 =3D 793, +}; + +#define GFX10_VOP3B_ENCODING 0x35 +#define GFX10_VOP3B_SRC_SGPR_BASE 0 +#define GFX10_VOP3B_SRC_VCC_LO 106 +#define GFX10_VOP3B_SRC_VCC_HI 107 +#define GFX10_VOP3B_SRC_TTPM_BASE 108 +#define GFX10_VOP3B_SRC_M0 124 +#define GFX10_VOP3B_SRC_NULL 125 +#define GFX10_VOP3B_SRC_EXEC_LO 126 +#define GFX10_VOP3B_SRC_EXEC_HI 127 +#define GFX10_VOP3B_SRC_INTEGER_0 128 +#define GFX10_VOP3B_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOP3B_SRC_SHARED_BASE 235 +#define GFX10_VOP3B_SRC_SHARED_LIMIT 236 +#define GFX10_VOP3B_SRC_PRIVATE_BASE 237 +#define GFX10_VOP3B_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOP3B_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOP3B_SRC_SDWA 249 +#define GFX10_VOP3B_SRC_DDP16 250 +#define GFX10_VOP3B_SRC_VCCZ 251 +#define GFX10_VOP3B_SRC_EXECZ 252 +#define GFX10_VOP3B_SRC_SCC 253 +#define GFX10_VOP3B_SRC_LITERAL_CONST 255 +#define GFX10_VOP3B_SRC_VGPR_BASE 256 + +struct amdgcn_gfx10_vop3p { + u64 vdst:8; + u64 neg_hi:3; + u64 op_sel:3; + u64 op_sel_hi2:1; + u64 clmp:1; + u64 op:7; + u64 dummy:3; + u64 encoding:6; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 op_sel_hi:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx10_vop3p_opcode { + GFX10_V_PK_MAD_I16 =3D 0, + GFX10_V_PK_MUL_LO_U16 =3D 1, + GFX10_V_PK_ADD_I16 =3D 2, + GFX10_V_PK_SUB_I16 =3D 3, + GFX10_V_PK_LSHLREV_B16 =3D 4, + GFX10_V_PK_LSHRREV_B16 =3D 5, + GFX10_V_PK_ASHRREV_I16 =3D 6, + GFX10_V_PK_MAX_I16 =3D 7, + GFX10_V_PK_MIN_I16 =3D 8, + GFX10_V_PK_MAD_U16 =3D 9, + GFX10_V_PK_ADD_U16 =3D 10, + GFX10_V_PK_SUB_U16 =3D 11, + GFX10_V_PK_MAX_U16 =3D 12, + GFX10_V_PK_MIN_U16 =3D 13, + GFX10_V_PK_FMA_F16 =3D 14, + GFX10_V_PK_ADD_F16 =3D 15, + GFX10_V_PK_MUL_F16 =3D 16, + GFX10_V_PK_MIN_F16 =3D 17, + GFX10_V_PK_MAX_F16 =3D 18, + GFX10_V_DOT2_F32_F16 =3D 19, + GFX10_V_DOT2_I32_I16 =3D 20, + GFX10_V_DOT2_U32_U16 =3D 21, + GFX10_V_DOT4_I32_I8 =3D 22, + GFX10_V_DOT4_U32_U8 =3D 23, + GFX10_V_DOT8_I32_I4 =3D 24, + GFX10_V_DOT8_U32_U4 =3D 25, + /* 26-31: reserved */ + GFX10_V_FMA_MIX_F32 =3D 32, + GFX10_V_FMA_MIXLO_F16 =3D 33, + GFX10_V_FMA_MIXHI_F16 =3D 34, +}; + +#define GFX10_VOP3P_ENCODING 0x33 +#define GFX10_VOP3P_SRC_SGPR_BASE 0 +#define GFX10_VOP3P_SRC_VCC_LO 106 +#define GFX10_VOP3P_SRC_VCC_HI 107 +#define GFX10_VOP3P_SRC_TTPM_BASE 108 +#define GFX10_VOP3P_SRC_M0 124 +#define GFX10_VOP3P_SRC_NULL 125 +#define GFX10_VOP3P_SRC_EXEC_LO 126 +#define GFX10_VOP3P_SRC_EXEC_HI 127 +#define GFX10_VOP3P_SRC_INTEGER_0 128 +#define GFX10_VOP3P_SRC_INTEGER_MINUS_1 193 +#define GFX10_VOP3P_SRC_SHARED_BASE 235 +#define GFX10_VOP3P_SRC_SHARED_LIMIT 236 +#define GFX10_VOP3P_SRC_PRIVATE_BASE 237 +#define GFX10_VOP3P_SRC_PRIVATE_LIMIT 238 +#define GFX10_VOP3P_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX10_VOP3P_SRC_SDWA 249 +#define GFX10_VOP3P_SRC_DDP16 250 +#define GFX10_VOP3P_SRC_VCCZ 251 +#define GFX10_VOP3P_SRC_EXECZ 252 +#define GFX10_VOP3P_SRC_SCC 253 +#define GFX10_VOP3P_SRC_VGPR_BASE 256 + +struct amdgcn_gfx10_sdwa { + u32 src0:8; + u32 dst_sel:3; + u32 dst_u:2; + u32 clmp:1; + u32 omod:2; + u32 src0_sel:3; + u32 src0_sext:1; + u32 src0_neg:1; + u32 src0_abs:1; + u32 dummy1:1; + u32 s0:1; + u32 src1_sel:3; + u32 src1_sext:1; + u32 src1_neg:1; + u32 src1_abs:1; + u32 dummy2:1; + u32 s1:1; +}; + +struct amdgcn_gfx10_sdwab { + u32 src0:8; + u32 sdst:7; + u32 sd:1; + u32 src0_sel:3; + u32 src0_sext:1; + u32 src0_neg:1; + u32 src0_abs:1; + u32 dummy1:1; + u32 s0:1; + u32 src1_sel:3; + u32 src1_sext:1; + u32 src1_neg:1; + u32 src1_abs:1; + u32 dummy2:1; + u32 s1:1; +}; + +struct amdgcn_gfx10_dpp16 { +}; + +struct amdgcn_gfx10_dpp8 { +}; + +/* Vector Parameter Interpolation Format */ +struct amdgcn_gfx10_vintrp { +}; + +/* LDS and GDS Format */ +struct amdgcn_gfx10_ds { + u64 offset0:8; + u64 offset1:8; + u64 dummy:1; + u64 gds:1; + u64 op:8; + u64 encoding:6; + u64 addr:8; + u64 data0:8; + u64 data1:8; + u64 vdst:8; +}; + +enum amdgcn_gfx10_ds_opcode { + GFX10_DS_ADD_U32 =3D 0, + GFX10_DS_SUB_U32 =3D 1, + GFX10_DS_RSUB_U32 =3D 2, + GFX10_DS_INC_U32 =3D 3, + GFX10_DS_DEC_U32 =3D 4, + GFX10_DS_MIN_I32 =3D 5, + GFX10_DS_MAX_I32 =3D 6, + GFX10_DS_MIN_U32 =3D 7, + GFX10_DS_MAX_U32 =3D 8, + GFX10_DS_AND_B32 =3D 9, + GFX10_DS_OR_B32 =3D 10, + GFX10_DS_XOR_B32 =3D 11, + GFX10_DS_MSKOR_B32 =3D 12, + GFX10_DS_WRITE_B32 =3D 13, + GFX10_DS_WRITE2_B32 =3D 14, + GFX10_DS_WRITE2ST64_B32 =3D 15, + GFX10_DS_CMPST_B32 =3D 16, + GFX10_DS_CMPST_F32 =3D 17, + GFX10_DS_MIN_F32 =3D 18, + GFX10_DS_MAX_F32 =3D 19, + GFX10_DS_NOP =3D 20, + GFX10_DS_ADD_F32 =3D 21, + /* 22-23: reserved */ + GFX10_DS_GWS_SEMA_RELEASE_ALL =3D 24, + GFX10_DS_GWS_INIT =3D 25, + GFX10_DS_GWS_SEMA_V =3D 26, + GFX10_DS_GWS_SEMA_BR =3D 27, + GFX10_DS_GWS_SEMA_P =3D 28, + GFX10_DS_GWS_BARRIER =3D 29, + GFX10_DS_WRITE_B8 =3D 30, + GFX10_DS_WRITE_B16 =3D 31, + GFX10_DS_ADD_RTN_U32 =3D 32, + GFX10_DS_SUB_RTN_U32 =3D 33, + GFX10_DS_RSUB_RTN_U32 =3D 34, + GFX10_DS_INC_RTN_U32 =3D 35, + GFX10_DS_DEC_RTN_U32 =3D 36, + GFX10_DS_MIN_RTN_I32 =3D 37, + GFX10_DS_MAX_RTN_I32 =3D 38, + GFX10_DS_MIN_RTN_U32 =3D 39, + GFX10_DS_MAX_RTN_U32 =3D 40, + GFX10_DS_AND_RTN_B32 =3D 41, + GFX10_DS_OR_RTN_B32 =3D 42, + GFX10_DS_XOR_RTN_B32 =3D 43, + GFX10_DS_MSKOR_RTN_B32 =3D 44, + GFX10_DS_WRXCHG_RTN_B32 =3D 45, + GFX10_DS_WRXCHG2_RTN_B32 =3D 46, + GFX10_DS_WRXCHG2ST64_RTN_B32 =3D 47, + GFX10_DS_CMPST_RTN_B32 =3D 48, + GFX10_DS_CMPST_RTN_F32 =3D 49, + GFX10_DS_MIN_RTN_F32 =3D 50, + GFX10_DS_MAX_RTN_F32 =3D 51, + GFX10_DS_WRAP_RTN_B32 =3D 52, + GFX10_DS_SWIZZLE_B32 =3D 53, + GFX10_DS_READ_B32 =3D 54, + GFX10_DS_READ2_B32 =3D 55, + GFX10_DS_READ2ST64_B32 =3D 56, + GFX10_DS_READ_I8 =3D 57, + GFX10_DS_READ_U8 =3D 58, + GFX10_DS_READ_I16 =3D 59, + GFX10_DS_READ_U16 =3D 60, + GFX10_DS_CONSUME =3D 61, + GFX10_DS_APPEND =3D 62, + GFX10_DS_ORDERED_COUNT =3D 63, + GFX10_DS_ADD_U64 =3D 64, + GFX10_DS_SUB_U64 =3D 65, + GFX10_DS_RSUB_U64 =3D 66, + GFX10_DS_INC_U64 =3D 67, + GFX10_DS_DEC_U64 =3D 68, + GFX10_DS_MIN_I64 =3D 69, + GFX10_DS_MAX_I64 =3D 70, + GFX10_DS_MIN_U64 =3D 71, + GFX10_DS_MAX_U64 =3D 72, + GFX10_DS_AND_B64 =3D 73, + GFX10_DS_OR_B64 =3D 74, + GFX10_DS_XOR_B64 =3D 75, + GFX10_DS_MSKOR_B64 =3D 76, + GFX10_DS_WRITE_B64 =3D 77, + GFX10_DS_WRITE2_B64 =3D 78, + GFX10_DS_WRITE2ST64_B64 =3D 79, + GFX10_DS_CMPST_B64 =3D 80, + GFX10_DS_CMPST_F64 =3D 81, + GFX10_DS_MIN_F64 =3D 82, + GFX10_DS_MAX_F64 =3D 83, + /* 84: reserved */ + GFX10_DS_ADD_RTN_F32 =3D 85, + /* 86-95: reserved */ + GFX10_DS_ADD_RTN_U64 =3D 96, + GFX10_DS_SUB_RTN_U64 =3D 97, + GFX10_DS_RSUB_RTN_U64 =3D 98, + GFX10_DS_INC_RTN_U64 =3D 99, + GFX10_DS_DEC_RTN_U64 =3D 100, + GFX10_DS_MIN_RTN_I64 =3D 101, + GFX10_DS_MAX_RTN_I64 =3D 102, + GFX10_DS_MIN_RTN_U64 =3D 103, + GFX10_DS_MAX_RTN_U64 =3D 104, + GFX10_DS_AND_RTN_B64 =3D 105, + GFX10_DS_OR_RTN_B64 =3D 106, + GFX10_DS_XOR_RTN_B64 =3D 107, + GFX10_DS_MSKOR_RTN_B64 =3D 108, + GFX10_DS_WRXCHG_RTN_B64 =3D 109, + GFX10_DS_WRXCHG2_RTN_B64 =3D 110, + GFX10_DS_WRXCHG2ST64_RTN_B64 =3D 111, + GFX10_DS_CMPST_RTN_B64 =3D 112, + GFX10_DS_CMPST_RTN_F64 =3D 113, + GFX10_DS_MIN_RTN_F64 =3D 114, + GFX10_DS_MAX_RTN_F64 =3D 115, + /* 116-117: reserved */ + GFX10_DS_READ_B64 =3D 118, + GFX10_DS_READ2_B64 =3D 119, + GFX10_DS_READ2ST64_B64 =3D 120, + /* 121-125: reserved */ + GFX10_DS_CONDXCHG32_RTN_B64 =3D 126, + /* 127-159: reserved */ + GFX10_DS_WRITE_B8_D16_HI =3D 160, + GFX10_DS_WRITE_B16_D16_HI =3D 161, + GFX10_DS_READ_U8_D16 =3D 162, + GFX10_DS_READ_U8_D16_HI =3D 163, + GFX10_DS_READ_I8_D16 =3D 164, + GFX10_DS_READ_I8_D16_HI =3D 165, + GFX10_DS_READ_U16_D16 =3D 166, + GFX10_DS_READ_U16_D16_HI =3D 167, + /* 168-175: reserved */ + GFX10_DS_WRITE_ADDTID_B32 =3D 176, + GFX10_DS_READ_ADDTID_B32 =3D 177, + GFX10_DS_PERMUTE_B32 =3D 178, + GFX10_DS_BPERMUTE_B32 =3D 179, + /* 180-221: reserved */ + GFX10_DS_WRITE_B96 =3D 222, + GFX10_DS_WRITE_B128 =3D 223, + /* 224-253: reserved */ + GFX10_DS_READ_B96 =3D 254, + GFX10_DS_READ_B128 =3D 255, +}; + +#define GFX10_DS_ENCODING 0x36 +#define GFX10_DS_GDS 1 +#define GFX10_DS_LDS 0 + +/* Vector Memory Buffer Formats */ +struct amdgcn_gfx10_mtbuf { + u64 offset:12; + u64 offen:1; + u64 idxen:1; + u64 glc:1; + u64 dlc:1; + u64 op:3; + u64 foamat:7; + u64 encoding:6; + u64 vaddr:8; + u64 vdata:8; + u64 srsrc:5; + u64 opm:1; + u64 slc:1; + u64 tfe:1; + u64 soffset:8; +}; + +enum amdgcn_gfx10_mtbuf_opcode { + GFX10_TBUFFER_LOAD_FORMAT_X =3D 0, + GFX10_TBUFFER_LOAD_FORMAT_XY =3D 1, + GFX10_TBUFFER_LOAD_FORMAT_XYZ =3D 2, + GFX10_TBUFFER_LOAD_FORMAT_XYZW =3D 3, + GFX10_TBUFFER_STORE_FORMAT_X =3D 4, + GFX10_TBUFFER_STORE_FORMAT_XY =3D 5, + GFX10_TBUFFER_STORE_FORMAT_XYZ =3D 6, + GFX10_TBUFFER_STORE_FORMAT_XYZW =3D 7, + GFX10_TBUFFER_LOAD_FORMAT_D16_X =3D 8, + GFX10_TBUFFER_LOAD_FORMAT_D16_XY =3D 9, + GFX10_TBUFFER_LOAD_FORMAT_D16_XYZ =3D 10, + GFX10_TBUFFER_LOAD_FORMAT_D16_XYZW =3D 11, + GFX10_TBUFFER_STORE_FORMAT_D16_X =3D 12, + GFX10_TBUFFER_STORE_FORMAT_D16_XY =3D 13, + GFX10_TBUFFER_STORE_FORMAT_D16_XYZ =3D 14, + GFX10_TBUFFER_STORE_FORMAT_D16_XYZW =3D 15, +}; + +#define GFX10_MUBUF_ENCODING 0x38 +#define GFX10_MUBUF_SOFFSET_SGPR_BASE 0 +#define GFX10_MUBUF_SOFFSET_VCC_LO 106 +#define GFX10_MUBUF_SOFFSET_VCC_HI 107 +#define GFX10_MUBUF_SOFFSET_TTPM_BASE 108 +#define GFX10_MUBUF_SOFFSET_M0 124 +#define GFX10_MUBUF_SOFFSET_NULL 125 +#define GFX10_MUBUF_SOFFSET_EXEC_LO 126 +#define GFX10_MUBUF_SOFFSET_EXEC_HI 127 +#define GFX10_MUBUF_SOFFSET_INTEGER_0 128 +#define GFX10_MUBUF_SOFFSET_INTEGER_MINUS_1 193 +#define GFX10_MUBUF_SOFFSET_SHARED_BASE 235 +#define GFX10_MUBUF_SOFFSET_SHARED_LIMIT 236 +#define GFX10_MUBUF_SOFFSET_PRIVATE_BASE 237 +#define GFX10_MUBUF_SOFFSET_PRIVATE_LIMIT 238 +#define GFX10_MUBUF_SOFFSET_POPS_EXITING_WAVE_ID 239 +#define GFX10_MUBUF_SOFFSET_VCCZ 251 +#define GFX10_MUBUF_SOFFSET_EXECZ 252 +#define GFX10_MUBUF_SOFFSET_SCC 253 + +struct amdgcn_gfx10_mubuf { + u64 offset:12; + u64 offen:1; + u64 idxen:1; + u64 glc:1; + u64 dlc:1; + u64 lds:1; + u64 dummy1:1; + u64 op:7; + u64 opm:1; + u64 encoding:6; + u64 vaddr:8; + u64 vdata:8; + u64 srsrc:5; + u64 dummy2:1; + u64 slc:1; + u64 tfe:1; + u64 soffset:8; +}; + +enum amdgcn_gfx10_mubuf_opcode { + GFX10_BUFFER_LOAD_FORMAT_X =3D 0, + GFX10_BUFFER_LOAD_FORMAT_XY =3D 1, + GFX10_BUFFER_LOAD_FORMAT_XYZ =3D 2, + GFX10_BUFFER_LOAD_FORMAT_XYZW =3D 3, + GFX10_BUFFER_STORE_FORMAT_X =3D 4, + GFX10_BUFFER_STORE_FORMAT_XY =3D 5, + GFX10_BUFFER_STORE_FORMAT_XYZ =3D 6, + GFX10_BUFFER_STORE_FORMAT_XYZW =3D 7, + GFX10_BUFFER_LOAD_UBYTE =3D 8, + GFX10_BUFFER_LOAD_SBYTE =3D 9, + GFX10_BUFFER_LOAD_USHORT =3D 10, + GFX10_BUFFER_LOAD_SSHORT =3D 11, + GFX10_BUFFER_LOAD_DWORD =3D 12, + GFX10_BUFFER_LOAD_DWORDX2 =3D 13, + GFX10_BUFFER_LOAD_DWORDX4 =3D 14, + GFX10_BUFFER_LOAD_DWORDX3 =3D 15, + /* 16-23: reserved */ + GFX10_BUFFER_STORE_BYTE =3D 24, + GFX10_BUFFER_STORE_BYTE_D16_HI =3D 25, + GFX10_BUFFER_STORE_SHORT =3D 26, + GFX10_BUFFER_STORE_SHORT_D16_HI =3D 27, + GFX10_BUFFER_STORE_DWORD =3D 28, + GFX10_BUFFER_STORE_DWORDX2 =3D 29, + GFX10_BUFFER_STORE_DWORDX4 =3D 30, + GFX10_BUFFER_STORE_DWORDX3 =3D 31, + GFX10_BUFFER_LOAD_UBYTE_D16 =3D 32, + GFX10_BUFFER_LOAD_UBYTE_D16_HI =3D 33, + GFX10_BUFFER_LOAD_SBYTE_D16 =3D 34, + GFX10_BUFFER_LOAD_SBYTE_D16_HI =3D 35, + GFX10_BUFFER_LOAD_SHORT_D16 =3D 36, + GFX10_BUFFER_LOAD_SHORT_D16_HI =3D 37, + GFX10_BUFFER_LOAD_FORMAT_D16_HI_X =3D 38, + GFX10_BUFFER_STORE_FORMAT_D16_HI_X =3D 39, + /* 40-47: reserved */ + GFX10_BUFFER_ATOMIC_SWAP =3D 48, + GFX10_BUFFER_ATOMIC_CMPSWAP =3D 49, + GFX10_BUFFER_ATOMIC_ADD =3D 50, + GFX10_BUFFER_ATOMIC_SUB =3D 51, + GFX10_BUFFER_ATOMIC_CSUB =3D 52, + GFX10_BUFFER_ATOMIC_SMIN =3D 53, + GFX10_BUFFER_ATOMIC_UMIN =3D 54, /* was 58 - BUG FIX (ISA p.212) */ + GFX10_BUFFER_ATOMIC_SMAX =3D 55, + GFX10_BUFFER_ATOMIC_UMAX =3D 56, + GFX10_BUFFER_ATOMIC_AND =3D 57, + GFX10_BUFFER_ATOMIC_OR =3D 58, + GFX10_BUFFER_ATOMIC_XOR =3D 59, + GFX10_BUFFER_ATOMIC_INC =3D 60, + GFX10_BUFFER_ATOMIC_DEC =3D 61, + GFX10_BUFFER_ATOMIC_FCMPSWAP =3D 62, + GFX10_BUFFER_ATOMIC_FMIN =3D 63, + GFX10_BUFFER_ATOMIC_FMAX =3D 64, + /* 65-79: reserved */ + GFX10_BUFFER_ATOMIC_SWAP_X2 =3D 80, + GFX10_BUFFER_ATOMIC_CMPSWAP_X2 =3D 81, + GFX10_BUFFER_ATOMIC_ADD_X2 =3D 82, + GFX10_BUFFER_ATOMIC_SUB_X2 =3D 83, + /* 84: reserved */ + GFX10_BUFFER_ATOMIC_SMIN_X2 =3D 85, + GFX10_BUFFER_ATOMIC_UMIN_X2 =3D 86, + GFX10_BUFFER_ATOMIC_SMAX_X2 =3D 87, + GFX10_BUFFER_ATOMIC_UMAX_X2 =3D 88, + GFX10_BUFFER_ATOMIC_AND_X2 =3D 89, + GFX10_BUFFER_ATOMIC_OR_X2 =3D 90, + GFX10_BUFFER_ATOMIC_XOR_X2 =3D 91, + GFX10_BUFFER_ATOMIC_INC_X2 =3D 92, + GFX10_BUFFER_ATOMIC_DEC_X2 =3D 93, + GFX10_BUFFER_ATOMIC_FCMPSWAP_X2 =3D 94, + GFX10_BUFFER_ATOMIC_FMIN_X2 =3D 95, + GFX10_BUFFER_ATOMIC_FMAX_X2 =3D 96, + /* 97-112: reserved */ + GFX10_BUFFER_GL0_INV =3D 113, + GFX10_BUFFER_GL1_INV =3D 114, + /* 115-127: reserved */ + GFX10_BUFFER_LOAD_FORMAT_D16_X =3D 128, + GFX10_BUFFER_LOAD_FORMAT_D16_XY =3D 129, + GFX10_BUFFER_LOAD_FORMAT_D16_XYZ =3D 130, + GFX10_BUFFER_LOAD_FORMAT_D16_XYZW =3D 131, + GFX10_BUFFER_STORE_FORMAT_D16_X =3D 132, + GFX10_BUFFER_STORE_FORMAT_D16_XY =3D 133, + GFX10_BUFFER_STORE_FORMAT_D16_XYZ =3D 134, + GFX10_BUFFER_STORE_FORMAT_D16_XYZW =3D 135, +}; + +/* Vector Memory Image Format */ +struct amdgcn_gfx10_mimg { +}; + +#define GFX10_FLAT_ENCODING 0x37 +#define GFX10_FLAT_SADDR_SGPR_BASE 0 +#define GFX10_FLAT_SADDR_VCC_LO 106 +#define GFX10_FLAT_SADDR_VCC_HI 107 +#define GFX10_FLAT_SADDR_TTPM_BASE 108 +#define GFX10_FLAT_SADDR_M0 124 +#define GFX10_FLAT_SADDR_NULL 125 +#define GFX10_FLAT_SADDR_EXEC_LO 126 +#define GFX10_FLAT_SADDR_EXEC_HI 127 +#define GFX10_FLAT_SADDR_INTEGER_0 128 +#define GFX10_FLAT_SADDR_INTEGER_MINUS_1 193 +#define GFX10_FLAT_SADDR_SHARED_BASE 235 +#define GFX10_FLAT_SADDR_SHARED_LIMIT 236 +#define GFX10_FLAT_SADDR_PRIVATE_BASE 237 +#define GFX10_FLAT_SADDR_PRIVATE_LIMIT 238 +#define GFX10_FLAT_SADDR_POPS_EXITING_WAVE_ID 239 +#define GFX10_FLAT_SADDR_VCCZ 251 +#define GFX10_FLAT_SADDR_EXECZ 252 +#define GFX10_FLAT_SADDR_SCC 253 + +/* + * Scalar SGPR that provides an offset address. Use NULL for disabled glob= al + * addressing; the 0x7f encoding is not accepted by LLVM/RGP for gfx10 glo= bal + * memory instructions. + * Meaning of this field is different for Scratch and Global: + * Flat: Unused. + * Scratch: Use an SGPR (instead of VGPR) for the address. + * Global: Use the SGPR to provide a base address; the VGPR provides a 32-= bit + * offset per lane. + */ +#define GFX10_FLAT_SADDR_DISABLE GFX10_FLAT_SADDR_NULL +#define GFX10_FLAT_SEG_FLAT 0 +#define GFX10_FLAT_SEG_SCRATCH 1 +#define GFX10_FLAT_SEG_GLOBAL 2 + +/* Flat Formats */ +struct amdgcn_gfx10_flat { + u64 offset:12; + u64 dlc:1; + u64 lds:1; + u64 seg:2; + u64 glc:1; + u64 slc:1; + u64 op:7; + u64 dummy1:1; + u64 encoding:6; + u64 addr:8; + u64 data:8; + u64 saddr:7; + u64 dummy2:1; + u64 vdst:8; +}; + +enum amdgcn_gfx10_flat_opcode { + /* 0-7: reserved */ + GFX10_FLAT_LOAD_UBYTE =3D 8, + GFX10_FLAT_LOAD_SBYTE =3D 9, + GFX10_FLAT_LOAD_USHORT =3D 10, + GFX10_FLAT_LOAD_SSHORT =3D 11, + GFX10_FLAT_LOAD_DWORD =3D 12, + GFX10_FLAT_LOAD_DWORDX2 =3D 13, + GFX10_FLAT_LOAD_DWORDX4 =3D 14, + GFX10_FLAT_LOAD_DWORDX3 =3D 15, + /* 16-23: reserved */ + GFX10_FLAT_STORE_BYTE =3D 24, + GFX10_FLAT_STORE_BYTE_D16_HI =3D 25, + GFX10_FLAT_STORE_SHORT =3D 26, + GFX10_FLAT_STORE_SHORT_D16_HI =3D 27, + GFX10_FLAT_STORE_DWORD =3D 28, + GFX10_FLAT_STORE_DWORDX2 =3D 29, + GFX10_FLAT_STORE_DWORDX4 =3D 30, + GFX10_FLAT_STORE_DWORDX3 =3D 31, + GFX10_FLAT_LOAD_UBYTE_D16 =3D 32, + GFX10_FLAT_LOAD_UBYTE_D16_HI =3D 33, + GFX10_FLAT_LOAD_SBYTE_D16 =3D 34, + GFX10_FLAT_LOAD_SBYTE_D16_HI =3D 35, + GFX10_FLAT_LOAD_SHORT_D16 =3D 36, + GFX10_FLAT_LOAD_SHORT_D16_HI =3D 37, + /* 38-47: reserved */ + GFX10_FLAT_ATOMIC_SWAP =3D 48, + GFX10_FLAT_ATOMIC_CMPSWAP =3D 49, + GFX10_FLAT_ATOMIC_ADD =3D 50, + GFX10_FLAT_ATOMIC_SUB =3D 51, + /* 52: reserved (CSUB is GLOBAL/MUBUF only) */ + GFX10_FLAT_ATOMIC_SMIN =3D 53, + GFX10_FLAT_ATOMIC_UMIN =3D 54, + GFX10_FLAT_ATOMIC_SMAX =3D 55, + GFX10_FLAT_ATOMIC_UMAX =3D 56, + GFX10_FLAT_ATOMIC_AND =3D 57, + GFX10_FLAT_ATOMIC_OR =3D 58, + GFX10_FLAT_ATOMIC_XOR =3D 59, + GFX10_FLAT_ATOMIC_INC =3D 60, + GFX10_FLAT_ATOMIC_DEC =3D 61, + GFX10_FLAT_ATOMIC_FCMPSWAP =3D 62, + GFX10_FLAT_ATOMIC_FMIN =3D 63, + GFX10_FLAT_ATOMIC_FMAX =3D 64, + /* 65-79: reserved */ + GFX10_FLAT_ATOMIC_SWAP_X2 =3D 80, + GFX10_FLAT_ATOMIC_CMPSWAP_X2 =3D 81, + GFX10_FLAT_ATOMIC_ADD_X2 =3D 82, + GFX10_FLAT_ATOMIC_SUB_X2 =3D 83, + /* 84: reserved */ + GFX10_FLAT_ATOMIC_SMIN_X2 =3D 85, + GFX10_FLAT_ATOMIC_UMIN_X2 =3D 86, + GFX10_FLAT_ATOMIC_SMAX_X2 =3D 87, + GFX10_FLAT_ATOMIC_UMAX_X2 =3D 88, + GFX10_FLAT_ATOMIC_AND_X2 =3D 89, + GFX10_FLAT_ATOMIC_OR_X2 =3D 90, + GFX10_FLAT_ATOMIC_XOR_X2 =3D 91, + GFX10_FLAT_ATOMIC_INC_X2 =3D 92, + GFX10_FLAT_ATOMIC_DEC_X2 =3D 93, + GFX10_FLAT_ATOMIC_FCMPSWAP_X2 =3D 94, + GFX10_FLAT_ATOMIC_FMIN_X2 =3D 95, + GFX10_FLAT_ATOMIC_FMAX_X2 =3D 96, +}; + +enum amdgcn_gfx10_global_opcode { + /* 0-7: reserved */ + GFX10_GLOBAL_LOAD_UBYTE =3D 8, + GFX10_GLOBAL_LOAD_SBYTE =3D 9, + GFX10_GLOBAL_LOAD_USHORT =3D 10, + GFX10_GLOBAL_LOAD_SSHORT =3D 11, + GFX10_GLOBAL_LOAD_DWORD =3D 12, + GFX10_GLOBAL_LOAD_DWORDX2 =3D 13, + GFX10_GLOBAL_LOAD_DWORDX4 =3D 14, + GFX10_GLOBAL_LOAD_DWORDX3 =3D 15, + /* 16-21: reserved */ + GFX10_GLOBAL_LOAD_DWORD_ADDTID =3D 22, + GFX10_GLOBAL_STORE_DWORD_ADDTID =3D 23, + GFX10_GLOBAL_STORE_BYTE =3D 24, + GFX10_GLOBAL_STORE_BYTE_D16_HI =3D 25, + GFX10_GLOBAL_STORE_SHORT =3D 26, + GFX10_GLOBAL_STORE_SHORT_D16_HI =3D 27, + GFX10_GLOBAL_STORE_DWORD =3D 28, + GFX10_GLOBAL_STORE_DWORDX2 =3D 29, + GFX10_GLOBAL_STORE_DWORDX4 =3D 30, + GFX10_GLOBAL_STORE_DWORDX3 =3D 31, + GFX10_GLOBAL_LOAD_UBYTE_D16 =3D 32, + GFX10_GLOBAL_LOAD_UBYTE_D16_HI =3D 33, + GFX10_GLOBAL_LOAD_SBYTE_D16 =3D 34, + GFX10_GLOBAL_LOAD_SBYTE_D16_HI =3D 35, + GFX10_GLOBAL_LOAD_SHORT_D16 =3D 36, + GFX10_GLOBAL_LOAD_SHORT_D16_HI =3D 37, + /* 38-47: reserved */ + GFX10_GLOBAL_ATOMIC_SWAP =3D 48, + GFX10_GLOBAL_ATOMIC_CMPSWAP =3D 49, + GFX10_GLOBAL_ATOMIC_ADD =3D 50, + GFX10_GLOBAL_ATOMIC_SUB =3D 51, + GFX10_GLOBAL_ATOMIC_CSUB =3D 52, + GFX10_GLOBAL_ATOMIC_SMIN =3D 53, + GFX10_GLOBAL_ATOMIC_UMIN =3D 54, + GFX10_GLOBAL_ATOMIC_SMAX =3D 55, + GFX10_GLOBAL_ATOMIC_UMAX =3D 56, + GFX10_GLOBAL_ATOMIC_AND =3D 57, + GFX10_GLOBAL_ATOMIC_OR =3D 58, + GFX10_GLOBAL_ATOMIC_XOR =3D 59, + GFX10_GLOBAL_ATOMIC_INC =3D 60, + GFX10_GLOBAL_ATOMIC_DEC =3D 61, + GFX10_GLOBAL_ATOMIC_FCMPSWAP =3D 62, + GFX10_GLOBAL_ATOMIC_FMIN =3D 63, + GFX10_GLOBAL_ATOMIC_FMAX =3D 64, + /* 65-79: reserved */ + GFX10_GLOBAL_ATOMIC_SWAP_X2 =3D 80, + GFX10_GLOBAL_ATOMIC_CMPSWAP_X2 =3D 81, + GFX10_GLOBAL_ATOMIC_ADD_X2 =3D 82, + GFX10_GLOBAL_ATOMIC_SUB_X2 =3D 83, + /* 84: reserved */ + GFX10_GLOBAL_ATOMIC_SMIN_X2 =3D 85, + GFX10_GLOBAL_ATOMIC_UMIN_X2 =3D 86, + GFX10_GLOBAL_ATOMIC_SMAX_X2 =3D 87, + GFX10_GLOBAL_ATOMIC_UMAX_X2 =3D 88, + GFX10_GLOBAL_ATOMIC_AND_X2 =3D 89, + GFX10_GLOBAL_ATOMIC_OR_X2 =3D 90, + GFX10_GLOBAL_ATOMIC_XOR_X2 =3D 91, + GFX10_GLOBAL_ATOMIC_INC_X2 =3D 92, + GFX10_GLOBAL_ATOMIC_DEC_X2 =3D 93, + GFX10_GLOBAL_ATOMIC_FCMPSWAP_X2 =3D 94, + GFX10_GLOBAL_ATOMIC_FMIN_X2 =3D 95, + GFX10_GLOBAL_ATOMIC_FMAX_X2 =3D 96, +}; + +enum amdgcn_gfx10_scratch_opcode { + /* 0-7: reserved */ + GFX10_SCRATCH_LOAD_UBYTE =3D 8, + GFX10_SCRATCH_LOAD_SBYTE =3D 9, + GFX10_SCRATCH_LOAD_USHORT =3D 10, + GFX10_SCRATCH_LOAD_SSHORT =3D 11, + GFX10_SCRATCH_LOAD_DWORD =3D 12, + GFX10_SCRATCH_LOAD_DWORDX2 =3D 13, + GFX10_SCRATCH_LOAD_DWORDX4 =3D 14, + GFX10_SCRATCH_LOAD_DWORDX3 =3D 15, + /* 16-23: reserved */ + GFX10_SCRATCH_STORE_BYTE =3D 24, + GFX10_SCRATCH_STORE_BYTE_D16_HI =3D 25, + GFX10_SCRATCH_STORE_SHORT =3D 26, + GFX10_SCRATCH_STORE_SHORT_D16_HI =3D 27, + GFX10_SCRATCH_STORE_DWORD =3D 28, + GFX10_SCRATCH_STORE_DWORDX2 =3D 29, + GFX10_SCRATCH_STORE_DWORDX4 =3D 30, + GFX10_SCRATCH_STORE_DWORDX3 =3D 31, + GFX10_SCRATCH_LOAD_UBYTE_D16 =3D 32, + GFX10_SCRATCH_LOAD_UBYTE_D16_HI =3D 33, + GFX10_SCRATCH_LOAD_SBYTE_D16 =3D 34, + GFX10_SCRATCH_LOAD_SBYTE_D16_HI =3D 35, + GFX10_SCRATCH_LOAD_SHORT_D16 =3D 36, + GFX10_SCRATCH_LOAD_SHORT_D16_HI =3D 37, +}; + +/* Export Format */ +struct amdgcn_gfx10_exp { +}; + +union amdgcn_gfx10_insn { + struct amdgcn_gfx10_sop2 sop2; + struct amdgcn_gfx10_sopk sopk; + struct amdgcn_gfx10_sop1 sop1; + struct amdgcn_gfx10_sopc sopc; + struct amdgcn_gfx10_sopp sopp; + struct amdgcn_gfx10_smem smem; + struct amdgcn_gfx10_vop2 vop2; + struct amdgcn_gfx10_vop1 vop1; + struct amdgcn_gfx10_vopc vopc; + struct amdgcn_gfx10_vop3a vop3a; + struct amdgcn_gfx10_vop3b vop3b; + struct amdgcn_gfx10_vop3p vop3p; + struct amdgcn_gfx10_sdwa sdwa; + struct amdgcn_gfx10_sdwab sdwab; + struct amdgcn_gfx10_dpp16 dpp16; + struct amdgcn_gfx10_dpp8 dpp8; + struct amdgcn_gfx10_vintrp vintrp; + struct amdgcn_gfx10_ds ds; + struct amdgcn_gfx10_mtbuf mtbuf; + struct amdgcn_gfx10_mubuf mubuf; + struct amdgcn_gfx10_mimg mimg; + struct amdgcn_gfx10_flat flat; + struct amdgcn_gfx10_exp exp; +}; + +/* Cast macro - convert u32 *buf position to GFX10 insn union pointer. */ +#define I10(buf, n) ((union amdgcn_gfx10_insn *)&(buf)[(n)]) + +inline u32 gfx10_get_param_base(struct amdgcn_param32 param) +{ + return gfx10_param_base[param.type]; +} + +inline u32 emit_gfx10_s_load_dwordx2(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int offset) +{ + /* s_load_dwordx2 , , + * sdata: register, load to. + * sbase: sgpr-pair, load from. + * offset: offset of kernarg_address in the hsa_kernel_dispatch_packet. + * + * sdata =3D *(sbase + offset); + * param =3D (__global struct _knod_bpf_param *)pkt.kernarg_address; + */ + + insn->smem.sbase =3D KNOD_AMDGPU_REG_PAIR(src.v); + insn->smem.sdata =3D dst.v; + insn->smem.dummy1 =3D 0; + insn->smem.dlc =3D 0; + insn->smem.dummy2 =3D 0; + insn->smem.glc =3D 0; + insn->smem.dummy3 =3D 0; + insn->smem.op =3D GFX10_S_LOAD_DWORDX2; + insn->smem.encoding =3D GFX10_SMEM_ENCODING; + insn->smem.offset =3D offset; + insn->smem.dummy4 =3D 0; + insn->smem.soffset =3D GFX10_SMEM_SOFFSET_NULL; /* no SGPR offset */ + + return 8; +} + +inline u32 emit_gfx10_v_bfe_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_BFE_I32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_bfe_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_BFE_U32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_bfi_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_BFI_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_lshl_add_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* v_lshl_add_u32 , , , + * + * =3D ( << ) + ; + */ + + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_LSHL_ADD_U32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_lshl_or_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* v_lshl_or_b32 , , , + * + * D.u =3D (S0.u << S1.u[4:0]) | S2.u + */ + + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_LSHL_OR_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_mad_u64_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 vdst, + struct amdgcn_param32 sdst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param64 src2) +{ + /* v_mad_u64_u32 , , , , + * : destination vgpr. + * : destination sgpr. + * : source vgpr + * : source vgpr + * : source vgpr + * + * {vcc_out,D.u64} =3D S0.u32 * S1.u32 + S2.u64. + * ctx =3D ¶m->sub[idx].ctx; + */ + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.hi.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3b.vdst =3D vdst.lo.v; + insn->vop3b.sdst =3D sdst.v; + insn->vop3b.clmp =3D 0; + insn->vop3b.op =3D GFX10_V_MAD_U64_U32; + insn->vop3b.encoding =3D GFX10_VOP3B_ENCODING; + insn->vop3b.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3b.src2 =3D gfx10_get_param_base(src2.lo) + src2.lo.v; + insn->vop3b.omod =3D 0; + insn->vop3b.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3b.src0 =3D gfx10_get_param_base(src0); + insn->vop3b.literal =3D src0.v; + return 12; + } + insn->vop3b.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_s_mov_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, struct amdgcn_param32 src) +{ + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->sop1.ssrc0 =3D gfx10_get_param_base(src); + insn->sop1.literal =3D src.v; + } else { + insn->sop1.ssrc0 =3D gfx10_get_param_base(src) + src.v; + } + insn->sop1.op =3D GFX10_S_MOV_B32; + insn->sop1.sdst =3D gfx10_get_param_base(dst) + dst.v; + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) + return 8; + return 4; +} + +inline u32 emit_gfx10_v_mov_b32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + if (dst.type !=3D AMDGCN_PARAM_TYPE_VGPR) + WARN_ON_ONCE(1); + insn->vop1.encoding =3D GFX10_VOP1_ENCODING; + insn->vop1.vdst =3D dst.v; + insn->vop1.op =3D GFX10_V_MOV_B32; + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop1.src0 =3D GFX10_VOP1_SRC_LITERAL_CONST; + insn->vop1.literal =3D src.v; + + return 8; + } + insn->vop1.src0 =3D gfx10_get_param_base(src) + src.v; + + return 4; +} + +inline u32 emit_gfx10_v_add_co_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3b.vdst =3D dst.v; + insn->vop3b.sdst =3D GFX10_VOP3B_SRC_VCC_LO; + insn->vop3b.clmp =3D 0; + insn->vop3b.op =3D GFX10_V_ADD_CO_U32; + insn->vop3b.encoding =3D GFX10_VOP3B_ENCODING; + insn->vop3b.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3b.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3b.omod =3D 0; + insn->vop3b.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3b.src0 =3D gfx10_get_param_base(src0); + insn->vop3b.literal =3D src0.v; + return 12; + } + insn->vop3b.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_add_co_ci_u32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_ADD_CO_CI_U32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +inline u32 emit_gfx10_v_xor_b32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_XOR_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_or_b32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_OR_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_cndmask_b32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_CNDMASK_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_and_b32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_AND_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_sub_co_ci_u32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* vdst =3D src0 - vsrc1 - vcc */ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_SUB_CO_CI_U32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +inline u32 emit_gfx10_v_subrev_co_ci_u32_e32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* vdst =3D src0 - vsrc1 - vcc */ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_SUBREV_CO_CI_U32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +inline u32 emit_gfx10_v_sub_co_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D src0 - src1 */ + insn->vop3b.vdst =3D dst.v; + insn->vop3b.sdst =3D GFX10_VOP3B_SRC_VCC_LO; + insn->vop3b.clmp =3D 0; + insn->vop3b.op =3D GFX10_V_SUB_CO_U32; + insn->vop3b.encoding =3D GFX10_VOP3B_ENCODING; + insn->vop3b.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3b.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3b.omod =3D 0; + insn->vop3b.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3b.src0 =3D gfx10_get_param_base(src0); + insn->vop3b.literal =3D src0.v; + return 12; + } + insn->vop3b.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_subrev_co_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D src0 - src1 */ + insn->vop3b.vdst =3D dst.v; + insn->vop3b.sdst =3D GFX10_VOP3B_SRC_VCC_LO; + insn->vop3b.clmp =3D 0; + insn->vop3b.op =3D GFX10_V_SUBREV_CO_U32; + insn->vop3b.encoding =3D GFX10_VOP3B_ENCODING; + insn->vop3b.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3b.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3b.omod =3D 0; + insn->vop3b.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3b.src0 =3D gfx10_get_param_base(src0); + insn->vop3b.literal =3D src0.v; + return 12; + } + insn->vop3b.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_mul_lo_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_MUL_LO_U32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +/* v_mbcnt_lo_u32_b32 vdst, src0, vsrc1 */ +inline u32 emit_gfx10_v_mbcnt_lo_u32_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_MBCNT_LO_U32_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +/* v_mbcnt_hi_u32_b32 vdst, src0, vsrc1 */ +inline u32 emit_gfx10_v_mbcnt_hi_u32_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_MBCNT_HI_U32_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +inline u32 emit_gfx10_v_mul_hi_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_MUL_HI_U32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_lshlrev_b64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 << s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_LSHLREV_B64; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1.lo) + src1.lo.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx10_v_lshrrev_b64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 >> s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_LSHRREV_B64; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1.lo) + src1.lo.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx10_v_lshlrev_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 << s0 */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_LSHLREV_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_lshrrev_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 << s0 */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_LSHRREV_B32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_add_nc_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_ADD_NC_U32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +inline u32 emit_gfx10_v_sub_nc_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_SUB_NC_U32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +inline u32 emit_gfx10_v_ashrrev_i64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 >> s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_ASHRREV_I64; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1.lo) + src1.lo.v; + insn->vop3a.src2 =3D GFX10_VOP3_UNUSED_SRC; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx10_v_ashrrev_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 >> s0 (arithmetic) */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_ASHRREV_I32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx10_v_alignbit_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* dst =3D s1 >> s0 */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_ALIGNBIT_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx10_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx10_v_cmp_eq_u64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 =3D=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_EQ_U64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_eq_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 =3D=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_EQ_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_gt_u64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 > S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_GT_U64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_gt_i64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 > S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_GT_I64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_ge_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 >=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_GE_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_gt_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 > S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_GT_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_lt_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 < S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_LT_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_le_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 <=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_LE_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_gt_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 > S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_GT_I32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_ge_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 >=3D S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_GE_I32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_lt_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 < S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_LT_I32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_le_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 <=3D S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMP_LE_I32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmpx_lt_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* EXEC &=3D (S0 < S1) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX10_V_CMPX_LT_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_ge_u64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 >=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_GE_U64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_ge_i64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 >=3D S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_GE_I64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_lt_u64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 < S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_LT_U64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_lt_i64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 < S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_LT_I64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_le_u64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 <=3D S1 */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_LE_U64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_v_cmp_le_i64(union amdgcn_gfx10_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 <=3D S1 (signed) */ + insn->vopc.src0 =3D gfx10_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX10_V_CMP_LE_I64; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_buffer_load_ubyte(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_ubyte , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dlc =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.op =3D GFX10_BUFFER_LOAD_UBYTE; + insn->mubuf.opm =3D 0; + insn->mubuf.encoding =3D GFX10_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy2 =3D 0; + insn->mubuf.slc =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX10_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx10_buffer_load_ushort(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_ushort , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dlc =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.op =3D GFX10_BUFFER_LOAD_USHORT; + insn->mubuf.opm =3D 0; + insn->mubuf.encoding =3D GFX10_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy2 =3D 0; + insn->mubuf.slc =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX10_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx10_buffer_load_dword(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dword , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dlc =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.op =3D GFX10_BUFFER_LOAD_DWORD; + insn->mubuf.opm =3D 0; + insn->mubuf.encoding =3D GFX10_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy2 =3D 0; + insn->mubuf.slc =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX10_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx10_buffer_load_dwordx2(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dwordx2 , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dlc =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.op =3D GFX10_BUFFER_LOAD_DWORDX2; + insn->mubuf.opm =3D 0; + insn->mubuf.encoding =3D GFX10_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy2 =3D 0; + insn->mubuf.slc =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX10_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx10_buffer_load_dwordx4(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dwordx4 , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dlc =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.op =3D GFX10_BUFFER_LOAD_DWORDX4; + insn->mubuf.opm =3D 0; + insn->mubuf.encoding =3D GFX10_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy2 =3D 0; + insn->mubuf.slc =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX10_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx10_global_load_ubyte(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_ubyte , , off offset: */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_UBYTE; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx10_global_load_ushort(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_ushort , , off offset: */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_USHORT; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx10_global_load_dword(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dword , , off offset: */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_DWORD; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx10_global_load_dwordx2(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dwordx2 , , off offset: */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_DWORDX2; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx10_global_load_dwordx4(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dwordx4 , , off offset: */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_DWORDX4; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx10_global_store_byte(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_byte , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_BYTE; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx10_global_store_short(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_short , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_SHORT; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx10_global_store_dword(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dword , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_DWORD; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +/* global_atomic_add vdst, addr, data, off (GLC=3D1: return old value) + * old_val =3D *(u32 *)(addr + off); *(u32 *)(addr + off) +=3D data; vdst = =3D old_val + */ +/* GFX10 global atomic: opcode only differs, all else identical */ +#define DEFINE_GFX10_GLOBAL_ATOMIC(name, opcode) \ +inline u32 emit_gfx10_global_atomic_##name(union amdgcn_gfx10_insn *insn,\ + struct amdgcn_param32 vdst, \ + struct amdgcn_param32 addr, \ + struct amdgcn_param32 data, \ + int off, int glc) \ +{ \ + insn->flat.offset =3D off; \ + insn->flat.dlc =3D 0; \ + insn->flat.lds =3D 0; \ + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; \ + insn->flat.glc =3D glc; \ + insn->flat.slc =3D 0; \ + insn->flat.op =3D (opcode); \ + insn->flat.encoding =3D GFX10_FLAT_ENCODING; \ + insn->flat.addr =3D addr.v; \ + insn->flat.data =3D data.v; \ + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; \ + insn->flat.dummy1 =3D 0; \ + insn->flat.dummy2 =3D 0; \ + insn->flat.vdst =3D vdst.v; \ + return 8; \ +} + +DEFINE_GFX10_GLOBAL_ATOMIC(add, GFX10_GLOBAL_ATOMIC_ADD) +DEFINE_GFX10_GLOBAL_ATOMIC(and, GFX10_GLOBAL_ATOMIC_AND) +DEFINE_GFX10_GLOBAL_ATOMIC(or, GFX10_GLOBAL_ATOMIC_OR) +DEFINE_GFX10_GLOBAL_ATOMIC(xor, GFX10_GLOBAL_ATOMIC_XOR) +DEFINE_GFX10_GLOBAL_ATOMIC(swap, GFX10_GLOBAL_ATOMIC_SWAP) +DEFINE_GFX10_GLOBAL_ATOMIC(cmpswap, GFX10_GLOBAL_ATOMIC_CMPSWAP) +DEFINE_GFX10_GLOBAL_ATOMIC(add_x2, GFX10_GLOBAL_ATOMIC_ADD_X2) +DEFINE_GFX10_GLOBAL_ATOMIC(and_x2, GFX10_GLOBAL_ATOMIC_AND_X2) +DEFINE_GFX10_GLOBAL_ATOMIC(or_x2, GFX10_GLOBAL_ATOMIC_OR_X2) +DEFINE_GFX10_GLOBAL_ATOMIC(xor_x2, GFX10_GLOBAL_ATOMIC_XOR_X2) +DEFINE_GFX10_GLOBAL_ATOMIC(swap_x2, GFX10_GLOBAL_ATOMIC_SWAP_X2) +DEFINE_GFX10_GLOBAL_ATOMIC(cmpswap_x2, GFX10_GLOBAL_ATOMIC_CMPSWAP_X2) + +inline u32 emit_gfx10_global_store_dwordx2(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dwordx2 , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_DWORDX2; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx10_global_store_dwordx4(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dwordx4 , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_DWORDX4; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX10_FLAT_SADDR_DISABLE; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx10_s_branch(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_BRANCH; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_cbranch_vccz(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_VCCZ; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_cbranch_vccnz(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_VCCNZ; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_branch_fixup(union amdgcn_gfx10_insn *insn, + short off) +{ + WARN_ON(insn->sopp.op !=3D GFX10_S_BRANCH && + insn->sopp.op !=3D GFX10_S_CBRANCH_VCCZ && + insn->sopp.op !=3D GFX10_S_CBRANCH_VCCNZ && + insn->sopp.op !=3D GFX10_S_CBRANCH_EXECZ && + insn->sopp.op !=3D GFX10_S_CBRANCH_EXECNZ); + insn->sopp.simm16 =3D off; + + return 4; +} + +inline u32 emit_gfx10_s_waitcnt_lgkmcnt(union amdgcn_gfx10_insn *insn) +{ + struct amdgcn_gfx10_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D -1; + vmcnt.vmcnt2 =3D -1; + vmcnt.expcnt =3D -1; + vmcnt.dummy =3D 0; + vmcnt.lgkmcnt =3D 0; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX10_S_WAITCNT; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_waitcnt_vmcnt(union amdgcn_gfx10_insn *insn) +{ + struct amdgcn_gfx10_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D 0; + vmcnt.vmcnt2 =3D 0; + vmcnt.expcnt =3D -1; + vmcnt.dummy =3D 0; + vmcnt.lgkmcnt =3D -1; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX10_S_WAITCNT; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_waitcnt_vmcnt_lgkmcnt(union amdgcn_gfx10_insn *ins= n) +{ + struct amdgcn_gfx10_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D 0; + vmcnt.vmcnt2 =3D 0; + vmcnt.expcnt =3D -1; + vmcnt.dummy =3D 0; + vmcnt.lgkmcnt =3D 0; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX10_S_WAITCNT; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_nop(union amdgcn_gfx10_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX10_S_NOP; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_endpgm(union amdgcn_gfx10_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX10_S_ENDPGM; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_code_end(union amdgcn_gfx10_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX10_S_CODE_END; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx10_s_icache_inv(union amdgcn_gfx10_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX10_S_ICACHE_INV; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +/* Structurized CFG instructions. + * These use raw SGPR indices for 64-bit pair operations involving + * EXEC (126) and VCC (106) special registers. + */ + +/* s_and_saveexec_b64 s[sdst:sdst+1], s[ssrc:ssrc+1] + * sdst =3D EXEC; EXEC &=3D ssrc; SCC =3D (EXEC !=3D 0) + */ +inline u32 emit_gfx10_s_and_saveexec_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX10_S_AND_SAVEEXEC_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + + return 4; +} + +/* s_bcnt1_i32_b64 sdst, s[ssrc:ssrc+1] + * sdst =3D popcount(ssrc). SCC =3D (sdst !=3D 0) + */ +inline u32 emit_gfx10_s_bcnt1_i32_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX10_S_BCNT1_I32_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + + return 4; +} + +/* s_mov_b64 s[sdst:sdst+1], s[ssrc:ssrc+1] (or special src like 0) */ +inline u32 emit_gfx10_s_mov_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX10_S_MOV_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + + return 4; +} + +/* s_and_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] */ +inline u32 emit_gfx10_s_and_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX10_S_AND_B64; + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; + + return 4; +} + +/* s_or_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] */ +inline u32 emit_gfx10_s_or_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX10_S_OR_B64; + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; + + return 4; +} + +/* s_andn2_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] + * sdst =3D ssrc0 & ~ssrc1 + */ +inline u32 emit_gfx10_s_andn2_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX10_S_ANDN2_B64; + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; + + return 4; +} + +/* s_cbranch_execz off - branch if EXEC =3D=3D 0 */ +inline u32 emit_gfx10_s_cbranch_execz(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_EXECZ; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +/* s_cbranch_execnz off - branch if EXEC !=3D 0 */ +inline u32 emit_gfx10_s_cbranch_execnz(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_EXECNZ; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +/* s_sub_u32 sdst, ssrc0, ssrc1 - sdst =3D ssrc0 - ssrc1; SCC =3D borrow */ +inline u32 emit_gfx10_s_sub_u32(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX10_S_SUB_U32; + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; + + return 4; +} + +/* s_cbranch_scc0 off - branch if SCC =3D=3D 0 (no borrow) */ +inline u32 emit_gfx10_s_cbranch_scc0(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_SCC0; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + + return 4; +} + +static inline void __emit_gfx10_sop2(union amdgcn_gfx10_insn *insn, + int op, int sdst, int ssrc0, int ssrc1) +{ + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; + insn->sop2.op =3D op; + insn->sop2.sdst =3D sdst; + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; +} + +static inline void __emit_gfx10_sop1(union amdgcn_gfx10_insn *insn, + int op, int sdst, int ssrc0) +{ + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + insn->sop1.op =3D op; + insn->sop1.sdst =3D sdst; + insn->sop1.ssrc0 =3D ssrc0; +} + +static inline void __emit_gfx10_sopp(union amdgcn_gfx10_insn *insn, + int op, u16 simm16) +{ + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + insn->sopp.op =3D op; + insn->sopp.simm16 =3D simm16; +} + +static inline void __emit_gfx10_vop2(union amdgcn_gfx10_insn *insn, + int op, int vdst, int vsrc1, int src0) +{ + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + insn->vop2.op =3D op; + insn->vop2.vdst =3D vdst; + insn->vop2.vsrc1 =3D vsrc1; + insn->vop2.src0 =3D src0; +} + +static inline void __emit_gfx10_smem(union amdgcn_gfx10_insn *insn, + int op, int sdata, int sbase_pair, + u32 offset) +{ + insn->smem.encoding =3D GFX10_SMEM_ENCODING; + insn->smem.op =3D op; + insn->smem.sdata =3D sdata; + insn->smem.sbase =3D sbase_pair; + insn->smem.offset =3D offset; + insn->smem.soffset =3D GFX10_SRC_NULL; +} + +static inline void __emit_gfx10_ds(union amdgcn_gfx10_insn *insn, + int op, int addr, int data0, int vdst, + int off0, int off1) +{ + insn->ds.encoding =3D GFX10_DS_ENCODING; + insn->ds.op =3D op; + insn->ds.gds =3D GFX10_DS_LDS; + insn->ds.addr =3D addr; + insn->ds.data0 =3D data0; + insn->ds.vdst =3D vdst; + insn->ds.offset0 =3D off0; + insn->ds.offset1 =3D off1; +} + +static inline void __emit_gfx10_global(union amdgcn_gfx10_insn *insn, + int op, int vdst, int vaddr, + int vdata, int saddr, int offset) +{ + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.op =3D op; + insn->flat.vdst =3D vdst; + insn->flat.addr =3D vaddr; + insn->flat.data =3D vdata; + insn->flat.saddr =3D saddr; + insn->flat.offset =3D offset; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX10 Param-Aware Emit Functions + * + * Paired with GFX9 equivalents in knod_gfx9_insn.h. Used by shader + * emitters (aesgcm_shader.h, ipsec_fused_gfx10.h, ...) that construct + * operands via P_S/P_V/P_I/P_L helpers. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static inline int __p2e10(struct amdgcn_param32 p) +{ + if (p.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) + return gfx10_get_param_base(p); + return gfx10_get_param_base(p) + p.v; +} + +/* --- GFX10 SOP2 (param32) --- */ + +#define DEFINE_GFX10_SOP2_P(name, opcode) \ +inline u32 emit_gfx10_##name(union amdgcn_gfx10_insn *insn, \ + struct amdgcn_param32 dst, \ + struct amdgcn_param32 src0, \ + struct amdgcn_param32 src1) \ +{ \ + insn->sop2.sdst =3D __p2e10(dst); \ + insn->sop2.ssrc0 =3D __p2e10(src0); \ + insn->sop2.ssrc1 =3D __p2e10(src1); \ + insn->sop2.op =3D opcode; \ + insn->sop2.encoding =3D GFX10_SOP2_ENCODING; \ + if (knod_param_is_literal(src0)) { \ + insn->sop2.literal =3D src0.v; \ + return 8; \ + } \ + if (knod_param_is_literal(src1)) { \ + insn->sop2.literal =3D src1.v; \ + return 8; \ + } \ + return 4; \ +} + +DEFINE_GFX10_SOP2_P(s_add_u32, GFX10_S_ADD_U32) +DEFINE_GFX10_SOP2_P(s_sub_u32_p, GFX10_S_SUB_U32) +DEFINE_GFX10_SOP2_P(s_addc_u32, GFX10_S_ADDC_U32) +DEFINE_GFX10_SOP2_P(s_subb_u32, GFX10_S_SUBB_U32) +DEFINE_GFX10_SOP2_P(s_and_b32_p, GFX10_S_AND_B32) +DEFINE_GFX10_SOP2_P(s_lshl_b32, GFX10_S_LSHL_B32) +DEFINE_GFX10_SOP2_P(s_lshr_b32, GFX10_S_LSHR_B32) +DEFINE_GFX10_SOP2_P(s_mul_i32, GFX10_S_MUL_I32) +DEFINE_GFX10_SOP2_P(s_xor_b32, GFX10_S_XOR_B32) + +#undef DEFINE_GFX10_SOP2_P + +/* --- GFX10 SOPC (param32) --- */ + +#define DEFINE_GFX10_SOPC_P(name, opcode) \ +inline u32 emit_gfx10_##name(union amdgcn_gfx10_insn *insn, \ + struct amdgcn_param32 src0, \ + struct amdgcn_param32 src1) \ +{ \ + insn->sopc.ssrc0 =3D __p2e10(src0); \ + insn->sopc.ssrc1 =3D __p2e10(src1); \ + insn->sopc.op =3D opcode; \ + insn->sopc.encoding =3D GFX10_SOPC_ENCODING; \ + if (knod_param_is_literal(src0)) { \ + insn->sopc.literal =3D src0.v; \ + return 8; \ + } \ + if (knod_param_is_literal(src1)) { \ + insn->sopc.literal =3D src1.v; \ + return 8; \ + } \ + return 4; \ +} + +DEFINE_GFX10_SOPC_P(s_cmp_eq_u32, GFX10_S_CMP_EQ_U32) +DEFINE_GFX10_SOPC_P(s_cmp_lg_u32, GFX10_S_CMP_LG_U32) +DEFINE_GFX10_SOPC_P(s_cmp_ge_u32, GFX10_S_CMP_GE_U32) +DEFINE_GFX10_SOPC_P(s_cmp_lt_u32, GFX10_S_CMP_LT_U32) + +#undef DEFINE_GFX10_SOPC_P + +/* --- GFX10 SOPP --- */ + +inline u32 emit_gfx10_s_barrier(union amdgcn_gfx10_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX10_S_BARRIER; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + return 4; +} + +inline u32 emit_gfx10_s_cbranch_scc1(union amdgcn_gfx10_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX10_S_CBRANCH_SCC1; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + return 4; +} + +/* + * GFX10 s_waitcnt encoding: + * simm16[3:0] =3D vmcnt[3:0] + * simm16[7:4] =3D expcnt (set to 7 =3D unused) + * simm16[13:8] =3D lgkmcnt[5:0] + * simm16[15:14] =3D vmcnt[5:4] + */ +#define GFX10_WAITCNT(vm, lgkm) \ + (((((vm) >> 4) & 0x3) << 14) | (((lgkm) & 0x3F) << 8) | \ + (7 << 4) | ((vm) & 0xF)) + +inline u32 emit_gfx10_s_waitcnt(union amdgcn_gfx10_insn *insn, + int vm, int lgkm) +{ + insn->sopp.simm16 =3D GFX10_WAITCNT(vm, lgkm); + insn->sopp.op =3D GFX10_S_WAITCNT; + insn->sopp.encoding =3D GFX10_SOPP_ENCODING; + return 4; +} + +/* --- GFX10 SOP1 --- */ + +inline u32 emit_gfx10_s_not_b64(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX10_S_NOT_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX10_SOP1_ENCODING; + return 4; +} + +/* --- GFX10 VOPC (v_cmp_ne_u32 param variant) --- */ + +inline u32 emit_gfx10_v_cmp_ne_u32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vopc.vsrc1 =3D src1.v; + insn->vopc.op =3D GFX10_V_CMP_NE_U32; + insn->vopc.encoding =3D GFX10_VOPC_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vopc.src0 =3D gfx10_get_param_base(src0); + insn->vopc.literal =3D src0.v; + return 8; + } + insn->vopc.src0 =3D gfx10_get_param_base(src0) + src0.v; + return 4; +} + +/* --- GFX10 SMEM --- */ + +#define DEFINE_GFX10_SMEM_P(name, opcode) \ +inline u32 emit_gfx10_##name(union amdgcn_gfx10_insn *insn, \ + struct amdgcn_param32 dst, \ + struct amdgcn_param32 src, int offset) \ +{ \ + insn->smem.sdata =3D dst.v; \ + insn->smem.sbase =3D src.v / 2; \ + insn->smem.op =3D opcode; \ + insn->smem.offset =3D offset; \ + insn->smem.soffset =3D GFX10_SRC_NULL; \ + insn->smem.encoding =3D GFX10_SMEM_ENCODING; \ + return 8; \ +} + +DEFINE_GFX10_SMEM_P(s_load_dword, GFX10_S_LOAD_DWORD) +DEFINE_GFX10_SMEM_P(s_load_dwordx4, GFX10_S_LOAD_DWORDX4) + +#undef DEFINE_GFX10_SMEM_P + +/* s_dcache_inv - no operands */ +inline u32 emit_gfx10_s_dcache_inv(union amdgcn_gfx10_insn *insn) +{ + insn->smem.sdata =3D 0; + insn->smem.sbase =3D 0; + insn->smem.op =3D GFX10_S_DCACHE_INV; + insn->smem.offset =3D 0; + insn->smem.soffset =3D GFX10_SRC_NULL; + insn->smem.encoding =3D GFX10_SMEM_ENCODING; + return 8; +} + +/* --- GFX10 VOP1: v_readfirstlane_b32 --- */ + +inline u32 emit_gfx10_v_readfirstlane_b32(union amdgcn_gfx10_insn *insn, + u8 sdst, u8 vsrc) +{ + insn->vop1.encoding =3D GFX10_VOP1_ENCODING; + insn->vop1.vdst =3D sdst; + insn->vop1.op =3D GFX10_V_READFIRSTLANE_B32; + insn->vop1.src0 =3D GFX10_SRC_VGPR_BASE + vsrc; + return 4; +} + +/* --- GFX10 VOP2: v_max_i32 --- */ + +inline u32 emit_gfx10_v_max_i32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX10_V_MAX_I32; + insn->vop2.encoding =3D GFX10_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx10_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx10_get_param_base(src0) + src0.v; + + return 4; +} + +/* --- GFX10 VOP3A: v_perm_b32 --- */ + +inline u32 emit_gfx10_v_perm_b32(union amdgcn_gfx10_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + /* VOP3 does not support literal constants on GFX10 */ + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX10_V_PERM_B32; + insn->vop3a.encoding =3D GFX10_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx10_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx10_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx10_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +/* --- GFX10 DS --- */ + +inline u32 emit_gfx10_ds_write_b32(union amdgcn_gfx10_insn *insn, + int addr, int data0) +{ + __emit_gfx10_ds(insn, GFX10_DS_WRITE_B32, addr, data0, 0, 0, 0); + return 8; +} + +inline u32 emit_gfx10_ds_read_b32(union amdgcn_gfx10_insn *insn, + int vdst, int addr) +{ + __emit_gfx10_ds(insn, GFX10_DS_READ_B32, addr, 0, vdst, 0, 0); + return 8; +} + +inline u32 emit_gfx10_ds_write_b32_off(union amdgcn_gfx10_insn *insn, + int addr, int data0, int offset) +{ + __emit_gfx10_ds(insn, GFX10_DS_WRITE_B32, addr, data0, 0, offset, 0); + return 8; +} + +inline u32 emit_gfx10_ds_read_b32_off(union amdgcn_gfx10_insn *insn, + int vdst, int addr, int offset) +{ + __emit_gfx10_ds(insn, GFX10_DS_READ_B32, addr, 0, vdst, offset, 0); + return 8; +} + +inline u32 emit_gfx10_ds_write_b128(union amdgcn_gfx10_insn *insn, + int addr, int data0) +{ + __emit_gfx10_ds(insn, GFX10_DS_WRITE_B128, addr, data0, 0, 0, 0); + return 8; +} + +inline u32 emit_gfx10_ds_read_b128(union amdgcn_gfx10_insn *insn, + int vdst, int addr) +{ + __emit_gfx10_ds(insn, GFX10_DS_READ_B128, addr, 0, vdst, 0, 0); + return 8; +} + +/* --- GFX10 GLOBAL with saddr mode (scalar base + VGPR offset) --- */ + +inline u32 emit_gfx10_global_load_dword_saddr(union amdgcn_gfx10_insn *ins= n, + int vdst, int vaddr, + int saddr, short off) +{ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_DWORD; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D vaddr; + insn->flat.data =3D 0; + insn->flat.saddr =3D saddr / 2; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D vdst; + return 8; +} + +inline u32 emit_gfx10_global_load_dwordx4_saddr(union amdgcn_gfx10_insn *i= nsn, + int vdst, int vaddr, + int saddr, short off) +{ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX10_GLOBAL_LOAD_DWORDX4; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D vaddr; + insn->flat.data =3D 0; + insn->flat.saddr =3D saddr / 2; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D vdst; + return 8; +} + +inline u32 emit_gfx10_global_store_dwordx4_saddr(union amdgcn_gfx10_insn *= insn, + int vaddr, int vdata, + int saddr, short off) +{ + insn->flat.offset =3D off; + insn->flat.dlc =3D 0; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX10_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX10_GLOBAL_STORE_DWORDX4; + insn->flat.encoding =3D GFX10_FLAT_ENCODING; + insn->flat.addr =3D vaddr; + insn->flat.data =3D vdata; + insn->flat.saddr =3D saddr / 2; + insn->flat.dummy1 =3D 0; + insn->flat.dummy2 =3D 0; + insn->flat.vdst =3D 0; + return 8; +} + +#endif diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h b/drivers/gpu= /drm/amd/amdkfd/knod/knod_gfx9_insn.h new file mode 100644 index 000000000000..ea987188f982 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h @@ -0,0 +1,4068 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_AMDGPU_GFX9_INSN_H_INCLUDED +#define KFD_AMDGPU_GFX9_INSN_H_INCLUDED + +#include "knod_amdgpu.h" + +/* + * knod_amdgpu_insn.h is intentionally NOT included here. + * This file provides ISA definitions (structs, enums, encoding constants) + * and per-version emit functions that are self-contained. + * + * The version-dispatch layer (knod_amdgpu_insn.h) includes this file + * and adds struct amdgcn_insn + wrapper functions on top. + */ + +#define GFX9_SRC_SGPR_BASE 0 +#define GFX9_SRC_VCC_LO 106 +#define GFX9_SRC_VCC_HI 107 +#define GFX9_SRC_TTPM_BASE 108 +#define GFX9_SRC_M0 124 +#define GFX9_SRC_NULL 125 +#define GFX9_SRC_EXEC_LO 126 +#define GFX9_SRC_EXEC_HI 127 +#define GFX9_SRC_INTEGER_0 128 +#define GFX9_SRC_INTEGER_MINUS_1 193 +#define GFX9_SRC_SHARED_BASE 235 +#define GFX9_SRC_SHARED_LIMIT 236 +#define GFX9_SRC_PRIVATE_BASE 237 +#define GFX9_SRC_PRIVATE_LIMIT 238 +#define GFX9_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_SRC_SDWA 249 +#define GFX9_SRC_DDP16 250 +#define GFX9_SRC_VCCZ 251 +#define GFX9_SRC_EXECZ 252 +#define GFX9_SRC_SCC 253 +#define GFX9_SRC_LITERAL_CONST 255 +#define GFX9_SRC_VGPR_BASE 256 + +static int gfx9_param_base[__AMDGCN_PARAM_TYPE_MAX] =3D { + GFX9_SRC_SGPR_BASE, + GFX9_SRC_VCC_LO, + GFX9_SRC_VCC_HI, + GFX9_SRC_TTPM_BASE, + GFX9_SRC_M0, + GFX9_SRC_NULL, + GFX9_SRC_EXEC_LO, + GFX9_SRC_EXEC_HI, + GFX9_SRC_INTEGER_0, + GFX9_SRC_INTEGER_MINUS_1, + GFX9_SRC_SHARED_BASE, + GFX9_SRC_SHARED_LIMIT, + GFX9_SRC_PRIVATE_BASE, + GFX9_SRC_PRIVATE_LIMIT, + GFX9_SRC_POPS_EXITING_WAVE_ID, + GFX9_SRC_SDWA, + GFX9_SRC_DDP16, + GFX9_SRC_VCCZ, + GFX9_SRC_EXECZ, + GFX9_SRC_SCC, + GFX9_SRC_LITERAL_CONST, + GFX9_SRC_VGPR_BASE, +}; + +/* Scalar ALU and Control Format */ +struct amdgcn_gfx9_sop2 { + u32 ssrc0:8; + u32 ssrc1:8; + u32 sdst:7; + u32 op:7; + u32 encoding:2; + u32 literal; +}; + +enum amdgcn_gfx9_sop2_opcode { + GFX9_S_ADD_U32, + GFX9_S_SUB_U32, + GFX9_S_ADD_I32, + GFX9_S_SUB_I32, + GFX9_S_ADDC_U32, + GFX9_S_SUBB_U32, + GFX9_S_MIN_I32, + GFX9_S_MIN_U32, + GFX9_S_MAX_I32, + GFX9_S_MAX_U32, + GFX9_S_CSELECT_B32, + GFX9_S_CSELECT_B64, + GFX9_S_AND_B32, + GFX9_S_AND_B64, + GFX9_S_OR_B32, + GFX9_S_OR_B64, + GFX9_S_XOR_B32, + GFX9_S_XOR_B64, + GFX9_S_ANDN2_B32, + GFX9_S_ANDN2_B64, + GFX9_S_ORN2_B32, + GFX9_S_ORN2_B64, + GFX9_S_NAND_B32, + GFX9_S_NAND_B64, + GFX9_S_NOR_B32, + GFX9_S_NOR_B64, + GFX9_S_XNOR_B32, + GFX9_S_XNOR_B64, + GFX9_S_LSHL_B32, + GFX9_S_LSHL_B64, + GFX9_S_LSHR_B32, + GFX9_S_LSHR_B64, + GFX9_S_ASHR_I32, + GFX9_S_ASHR_I64, + GFX9_S_BFM_B32, + GFX9_S_BFM_B64, + GFX9_S_MUL_I32, + GFX9_S_BFE_U32, + GFX9_S_BFE_I32, + GFX9_S_BFE_U64, + GFX9_S_BFE_I64, + GFX9_S_CBRANCH_G_FORK, + GFX9_S_ABSDIFF_I32, + GFX9_S_RFE_RESTORE_B64, + GFX9_S_MUL_HI_U32, + GFX9_S_MUL_HI_I32, + GFX9_S_LSHL1_ADD_U32, + GFX9_S_LSHL2_ADD_U32, + GFX9_S_LSHL3_ADD_U32, + GFX9_S_LSHL4_ADD_U32, + GFX9_S_PACK_LL_B32_B16, + GFX9_S_PACK_LH_B32_B16, + GFX9_S_PACK_HH_B32_B16, +}; + +#define GFX9_SOP2_ENCODING 0x2 +#define GFX9_SOP2_SSRC_SGPR_BASE 0 +#define GFX9_SOP2_SSRC_VCC_LO 106 +#define GFX9_SOP2_SSRC_VCC_HI 107 +#define GFX9_SOP2_SSRC_TTPM_BASE 108 +#define GFX9_SOP2_SSRC_M0 124 +#define GFX9_SOP2_SSRC_NULL 125 +#define GFX9_SOP2_SSRC_EXEC_LO 126 +#define GFX9_SOP2_SSRC_EXEC_HI 127 +#define GFX9_SOP2_SSRC_INTEGER_0 128 +#define GFX9_SOP2_SSRC_INTEGER_MINUS_1 193 +#define GFX9_SOP2_SSRC_SHARED_BASE 235 +#define GFX9_SOP2_SSRC_SHARED_LIMIT 236 +#define GFX9_SOP2_SSRC_PRIVATE_BASE 237 +#define GFX9_SOP2_SSRC_PRIVATE_LIMIT 238 +#define GFX9_SOP2_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_SOP2_SSRC_VCCZ 251 +#define GFX9_SOP2_SSRC_EXECZ 252 +#define GFX9_SOP2_SSRC_SCC 253 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx9_sopk { + u32 simm16:16; + u32 sdst:7; + u32 op:5; + u32 encoding:4; +}; + +enum amdgcn_gfx9_sopk_opcode { + GFX9_S_MOVK_I32, + GFX9_S_CMOVK_I32, + GFX9_S_CMPK_EQ_I32, + GFX9_S_CMPK_LG_I32, + GFX9_S_CMPK_GT_I32, + GFX9_S_CMPK_GE_I32, + GFX9_S_CMPK_LT_I32, + GFX9_S_CMPK_LE_I32, + GFX9_S_CMPK_EQ_U32, + GFX9_S_CMPK_LG_U32, + GFX9_S_CMPK_GT_U32, + GFX9_S_CMPK_GE_U32, + GFX9_S_CMPK_LT_U32, + GFX9_S_CMPK_LE_U32, + GFX9_S_ADDK_I32, + GFX9_S_MULK_I32, + GFX9_S_CBRANCH_I_FORK, + GFX9_S_GETREG_B32, + GFX9_S_SETREG_B32 =3D 18, + GFX9_S_SETREG_IMM32_B32 =3D 20, + GFX9_S_CALL_B64, +}; + +#define GFX9_SOPK_ENCODING 0xb +#define GFX9_SOPK_SDST_SGPR0_BASE 0 +#define GFX9_SOPK_SDST_VCC_LO 106 +#define GFX9_SOPK_SDST_VCC_HI 107 +#define GFX9_SOPK_SDST_TTPM_BASE 108 +#define GFX9_SOPK_SDST_M0 124 +#define GFX9_SOPK_SDST_NULL 125 +#define GFX9_SOPK_SDST_EXEC_LO 126 +#define GFX9_SOPK_SDST_EXEC_HI 127 + +struct amdgcn_gfx9_sop1 { + u32 ssrc0:8; + u32 op:8; + u32 sdst:7; + u32 encoding:9; + u32 literal; +}; + +enum amdgcn_gfx9_sop1_opcode { + GFX9_S_MOV_B32 =3D 0, + GFX9_S_MOV_B64, + GFX9_S_CMOV_B32, + GFX9_S_CMOV_B64, + GFX9_S_NOT_B32, + GFX9_S_NOT_B64, + GFX9_S_WQM_B32, + GFX9_S_WQM_B64, + GFX9_S_BREV_B32, + GFX9_S_BREV_B64, + GFX9_S_BCNT0_I32_B32, + GFX9_S_BCNT0_I32_B64, + GFX9_S_BCNT1_I32_B32, + GFX9_S_BCNT1_I32_B64, + GFX9_S_FF0_I32_B32, + GFX9_S_FF0_I32_B64, + GFX9_S_FF1_I32_B32, + GFX9_S_FF1_I32_B64, + GFX9_S_FLBIT_I32_B32, + GFX9_S_FLBIT_I32_B64, + GFX9_S_FLBIT_I32, + GFX9_S_FLBIT_I32_I64, + GFX9_S_SEXT_I32_I8, + GFX9_S_SEXT_I32_I16, + GFX9_S_BITSET0_B32, + GFX9_S_BITSET0_B64, + GFX9_S_BITSET1_B32, + GFX9_S_BITSET1_B64, + GFX9_S_GETPC_B64, + GFX9_S_SETPC_B64, + GFX9_S_SWAPPC_B64, + GFX9_S_RFE_B64 =3D 31, + GFX9_S_AND_SAVEEXEC_B64, + GFX9_S_OR_SAVEEXEC_B64, + GFX9_S_XOR_SAVEEXEC_B64, + GFX9_S_ANDN2_SAVEEXEC_B64, + GFX9_S_ORN2_SAVEEXEC_B64, + GFX9_S_NAND_SAVEEXEC_B64, + GFX9_S_NOR_SAVEEXEC_B64, + GFX9_S_XNOR_SAVEEXEC_B64, + GFX9_S_QUADMASK_B32, + GFX9_S_QUADMASK_B64, + GFX9_S_MOVRELS_B32, + GFX9_S_MOVRELS_B64, + GFX9_S_MOVRELD_B32, + GFX9_S_MOVRELD_B64, + GFX9_S_ABS_I32 =3D 48, + GFX9_S_SET_GPR_IDX_IDX =3D 50, + GFX9_S_ANDN1_SAVEEXEC_B64, + GFX9_S_ORN1_SAVEEXEC_B64, + GFX9_S_ANDN1_WREXEC_B64, + GFX9_S_ANDN2_WREXEC_B64, + GFX9_S_BITREPLICATE_B64_B32, +}; + +#define GFX9_SOP1_ENCODING 0x17d +#define GFX9_SOP1_SSRC_SGPR_BASE 0 +#define GFX9_SOP1_SSRC_VCC_LO 106 +#define GFX9_SOP1_SSRC_VCC_HI 107 +#define GFX9_SOP1_SSRC_TTPM_BASE 108 +#define GFX9_SOP1_SSRC_M0 124 +#define GFX9_SOP1_SSRC_NULL 125 +#define GFX9_SOP1_SSRC_EXEC_LO 126 +#define GFX9_SOP1_SSRC_EXEC_HI 127 +#define GFX9_SOP1_SSRC_INTEGER_0 128 +#define GFX9_SOP1_SSRC_INTEGER_MINUS_1 193 +#define GFX9_SOP1_SSRC_SHARED_BASE 235 +#define GFX9_SOP1_SSRC_SHARED_LIMIT 236 +#define GFX9_SOP1_SSRC_PRIVATE_BASE 237 +#define GFX9_SOP1_SSRC_PRIVATE_LIMIT 238 +#define GFX9_SOP1_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_SOP1_SSRC_VCCZ 251 +#define GFX9_SOP1_SSRC_EXECZ 252 +#define GFX9_SOP1_SSRC_SCC 253 +#define GFX9_SOP1_SSRC_LITERAL_CONST 255 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx9_sopc { + u32 ssrc0:8; + u32 ssrc1:8; + u32 op:7; + u32 encoding:9; + u32 literal; +}; + +enum amdgcn_gfx9_sopc_opcode { + GFX9_S_CMP_EQ_I32, + GFX9_S_CMP_LG_I32, + GFX9_S_CMP_GT_I32, + GFX9_S_CMP_GE_I32, + GFX9_S_CMP_LT_I32, + GFX9_S_CMP_LE_I32, + GFX9_S_CMP_EQ_U32, + GFX9_S_CMP_LG_U32, + GFX9_S_CMP_GT_U32, + GFX9_S_CMP_GE_U32, + GFX9_S_CMP_LT_U32, + GFX9_S_CMP_LE_U32, + GFX9_S_BITCMP0_B32, + GFX9_S_BITCMP1_B32, + GFX9_S_BITCMP0_B64, + GFX9_S_BITCMP1_B64, + GFX9_S_SETVSKIP, + GFX9_S_SET_GPR_IDX_ON, + GFX9_S_CMP_EQ_U64, + GFX9_S_CMP_LG_U64, +}; + +#define GFX9_SOPC_ENCODING 0x17e +#define GFX9_SOPC_SSRC_SGPR_BASE 0 +#define GFX9_SOPC_SSRC_VCC_LO 106 +#define GFX9_SOPC_SSRC_VCC_HI 107 +#define GFX9_SOPC_SSRC_TTPM_BASE 108 +#define GFX9_SOPC_SSRC_M0 124 +#define GFX9_SOPC_SSRC_NULL 125 +#define GFX9_SOPC_SSRC_EXEC_LO 126 +#define GFX9_SOPC_SSRC_EXEC_HI 127 +#define GFX9_SOPC_SSRC_INTEGER_0 128 +#define GFX9_SOPC_SSRC_INTEGER_MINUS_1 193 +#define GFX9_SOPC_SSRC_SHARED_BASE 235 +#define GFX9_SOPC_SSRC_SHARED_LIMIT 236 +#define GFX9_SOPC_SSRC_PRIVATE_BASE 237 +#define GFX9_SOPC_SSRC_PRIVATE_LIMIT 238 +#define GFX9_SOPC_SSRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_SOPC_SSRC_VCCZ 251 +#define GFX9_SOPC_SSRC_EXECZ 252 +#define GFX9_SOPC_SSRC_SCC 253 +/* SDST + * Same codes as SSRC0, above except only codes 0-127 are valid. + */ + +struct amdgcn_gfx9_sopp_vmcnt { + u16 vmcnt1:4; + u16 expcnt:3; + u16 dummy1:1; + u16 lgkmcnt:4; /* GFX9: 4-bit counter, max=3D15 */ + u16 dummy2:2; + u16 vmcnt2:2; +}; + +struct amdgcn_gfx9_sopp { + u32 simm16:16; + u32 op:7; + u32 encoding:9; +}; + +enum amdgcn_gfx9_sopp_opcode { + GFX9_S_NOP, + GFX9_S_ENDPGM, + GFX9_S_BRANCH, + GFX9_S_WAKEUP, + GFX9_S_CBRANCH_SCC0, + GFX9_S_CBRANCH_SCC1, + GFX9_S_CBRANCH_VCCZ, + GFX9_S_CBRANCH_VCCNZ, + GFX9_S_CBRANCH_EXECZ, + GFX9_S_CBRANCH_EXECNZ, + GFX9_S_BARRIER, + GFX9_S_SETKILL, + GFX9_S_WAITCNT, + GFX9_S_SETHALT, + GFX9_S_SLEEP, + GFX9_S_SETPRIO, + GFX9_S_SENDMSG, + GFX9_S_SENDMSGHALT, + GFX9_S_TRAP, + GFX9_S_ICACHE_INV, + GFX9_S_INCPERFLEVEL, + GFX9_S_DECPERFLEVEL, + GFX9_S_TTRACEDATA, + GFX9_S_CBRANCH_CDBGSYS, + GFX9_S_CBRANCH_CDBGUSER, + GFX9_S_CBRANCH_CDBGSYS_OR_USER, + GFX9_S_CBRANCH_CDBGSYS_AND_USER, + GFX9_S_ENDPGM_SAVED, + GFX9_S_SET_GPR_IDX_OFF, + GFX9_S_SET_GPR_IDX_MODE, + GFX9_S_ENDPGM_ORDERED_PS_DONE, +}; + +#define GFX9_SOPP_ENCODING 0x17f + +/* Scalar Memory Format */ +struct amdgcn_gfx9_smem { + u64 sbase:6; + u64 sdata:7; + u64 dummy1:1; + u64 soe:1; /* Scalar Offset Enable */ + u64 nv:1; /* Non-Volatile */ + u64 glc:1; /* Globally Memory Coherent */ + u64 imm:1; /* Immediate Enable */ + u64 op:8; + u64 encoding:6; + u64 offset:21; + u64 dummy2:4; + u64 soffset:7; +}; + +enum amdgcn_gfx9_smem_opcode { + GFX9_S_LOAD_DWORD, + GFX9_S_LOAD_DWORDX2, + GFX9_S_LOAD_DWORDX4, + GFX9_S_LOAD_DWORDX8, + GFX9_S_LOAD_DWORDX16, + GFX9_S_SCRATCH_LOAD_DWORD, + GFX9_S_SCRATCH_LOAD_DWORDX2, + GFX9_S_SCRATCH_LOAD_DWORDX4, + GFX9_S_BUFFER_LOAD_DWORD, + GFX9_S_BUFFER_LOAD_DWORDX2, + GFX9_S_BUFFER_LOAD_DWORDX4, + GFX9_S_BUFFER_LOAD_DWORDX8, + GFX9_S_BUFFER_LOAD_DWORDX16 =3D 12, + GFX9_S_STORE_DWORD =3D 16, + GFX9_S_STORE_DWORDX2, + GFX9_S_STORE_DWORDX4 =3D 18, + GFX9_S_SCRATCH_STORE_DWORD =3D 21, + GFX9_S_SCRATCH_STORE_DWORDX2, + GFX9_S_SCRATCH_STORE_DWORDX4, + GFX9_S_BUFFER_STORE_DWORD, + GFX9_S_BUFFER_STORE_DWORDX2, + GFX9_S_BUFFER_STORE_DWORDX4 =3D 26, + GFX9_S_DCACHE_INV =3D 32, + GFX9_S_DCACHE_WB, + GFX9_S_DCACHE_INV_VOL, + GFX9_S_DCACHE_WB_VOL, + GFX9_S_MEMTIME, + GFX9_S_MEMREALTIME, + GFX9_S_ATC_PROBE, + GFX9_S_ATC_PROBE_BUFFER, + GFX9_S_DCACHE_DISCARD, + GFX9_S_DCACHE_DISCARD_X2 =3D 41, + GFX9_S_BUFFER_ATOMIC_SWAP =3D 64, + GFX9_S_BUFFER_ATOMIC_CMPSWAP, + GFX9_S_BUFFER_ATOMIC_ADD, + GFX9_S_BUFFER_ATOMIC_SUB, + GFX9_S_BUFFER_ATOMIC_SMIN, + GFX9_S_BUFFER_ATOMIC_UMIN, + GFX9_S_BUFFER_ATOMIC_SMAX, + GFX9_S_BUFFER_ATOMIC_UMAX, + GFX9_S_BUFFER_ATOMIC_AND, + GFX9_S_BUFFER_ATOMIC_OR, + GFX9_S_BUFFER_ATOMIC_XOR, + GFX9_S_BUFFER_ATOMIC_INC, + GFX9_S_BUFFER_ATOMIC_DEC =3D 76, + GFX9_S_BUFFER_ATOMIC_SWAP_X2 =3D 96, + GFX9_S_BUFFER_ATOMIC_CMPSWAP_X2, + GFX9_S_BUFFER_ATOMIC_ADD_X2, + GFX9_S_BUFFER_ATOMIC_SUB_X2, + GFX9_S_BUFFER_ATOMIC_SMIN_X2, + GFX9_S_BUFFER_ATOMIC_UMIN_X2, + GFX9_S_BUFFER_ATOMIC_SMAX_X2, + GFX9_S_BUFFER_ATOMIC_UMAX_X2, + GFX9_S_BUFFER_ATOMIC_AND_X2, + GFX9_S_BUFFER_ATOMIC_OR_X2, + GFX9_S_BUFFER_ATOMIC_XOR_X2, + GFX9_S_BUFFER_ATOMIC_INC_X2, + GFX9_S_BUFFER_ATOMIC_DEC_X2 =3D 108, + GFX9_S_ATOMIC_SWAP =3D 128, + GFX9_S_ATOMIC_CMPSWAP, + GFX9_S_ATOMIC_ADD, + GFX9_S_ATOMIC_SUB, + GFX9_S_ATOMIC_SMIN, + GFX9_S_ATOMIC_UMIN, + GFX9_S_ATOMIC_SMAX, + GFX9_S_ATOMIC_UMAX, + GFX9_S_ATOMIC_AND, + GFX9_S_ATOMIC_OR, + GFX9_S_ATOMIC_XOR, + GFX9_S_ATOMIC_INC, + GFX9_S_ATOMIC_DEC =3D 140, + GFX9_S_ATOMIC_SWAP_X2 =3D 160, + GFX9_S_ATOMIC_CMPSWAP_X2, + GFX9_S_ATOMIC_ADD_X2, + GFX9_S_ATOMIC_SUB_X2, + GFX9_S_ATOMIC_SMIN_X2, + GFX9_S_ATOMIC_UMIN_X2, + GFX9_S_ATOMIC_SMAX_X2, + GFX9_S_ATOMIC_UMAX_X2, + GFX9_S_ATOMIC_AND_X2, + GFX9_S_ATOMIC_OR_X2, + GFX9_S_ATOMIC_XOR_X2, + GFX9_S_ATOMIC_INC_X2, + GFX9_S_ATOMIC_DEC_X2, +}; + +#define GFX9_SMEM_ENCODING 0x30 +#define GFX9_SMEM_SOFFSET_NULL 125 + +/* Vector ALU Format */ +struct amdgcn_gfx9_vop2 { + u32 src0:9; + u32 vsrc1:8; + u32 vdst:8; + u32 op:6; + u32 encoding:1; + u32 literal; +}; + +enum amdgcn_gfx9_vop2_opcode { + GFX9_V_CNDMASK_B32, + GFX9_V_ADD_F32, + GFX9_V_SUB_F32, + GFX9_V_SUBREV_F32, + GFX9_V_MUL_LEGACY_F32, + GFX9_V_MUL_F32, + GFX9_V_MUL_I32_I24, + GFX9_V_MUL_HI_I32_I24, + GFX9_V_MUL_U32_U24, + GFX9_V_MUL_HI_U32_U24, + GFX9_V_MIN_F32, + GFX9_V_MAX_F32, + GFX9_V_MIN_I32, + GFX9_V_MAX_I32, + GFX9_V_MIN_U32, + GFX9_V_MAX_U32, + GFX9_V_LSHRREV_B32, + GFX9_V_ASHRREV_I32, + GFX9_V_LSHLREV_B32, + GFX9_V_AND_B32, + GFX9_V_OR_B32, + GFX9_V_XOR_B32, + GFX9_V_MAC_F32, + GFX9_V_MADMK_F32, + GFX9_V_MADAK_F32, + GFX9_V_ADD_CO_U32, + GFX9_V_SUB_CO_U32, + GFX9_V_SUBREV_CO_U32, + GFX9_V_ADDC_CO_U32, + GFX9_V_SUBB_CO_U32, + GFX9_V_SUBBREV_CO_U32, + GFX9_V_ADD_F16, + GFX9_V_SUB_F16, + GFX9_V_SUBREV_F16, + GFX9_V_MUL_F16, + GFX9_V_MAC_F16, + GFX9_V_MADMK_F16, + GFX9_V_MADAK_F16, + GFX9_V_ADD_U16, + GFX9_V_SUB_U16, + GFX9_V_SUBREV_U16, + GFX9_V_MUL_LO_U16, + GFX9_V_LSHLREV_B16, + GFX9_V_LSHRREV_B16, + GFX9_V_ASHRREV_I16, + GFX9_V_MAX_F16, + GFX9_V_MIN_F16, + GFX9_V_MAX_U16, + GFX9_V_MAX_I16, + GFX9_V_MIN_U16, + GFX9_V_MIN_I16, + GFX9_V_LDEXP_F16, + GFX9_V_ADD_U32, + GFX9_V_SUB_U32, + GFX9_V_SUBREV_U32, +}; + +#define GFX9_VOP2_ENCODING 0x0 +#define GFX9_VOP2_SRC_SGPR_BASE 0 +#define GFX9_VOP2_SRC_VCC_LO 106 +#define GFX9_VOP2_SRC_VCC_HI 107 +#define GFX9_VOP2_SRC_TTPM_BASE 108 +#define GFX9_VOP2_SRC_M0 124 +#define GFX9_VOP2_SRC_NULL 125 +#define GFX9_VOP2_SRC_EXEC_LO 126 +#define GFX9_VOP2_SRC_EXEC_HI 127 +#define GFX9_VOP2_SRC_INTEGER_0 128 +#define GFX9_VOP2_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOP2_SRC_SHARED_BASE 235 +#define GFX9_VOP2_SRC_SHARED_LIMIT 236 +#define GFX9_VOP2_SRC_PRIVATE_BASE 237 +#define GFX9_VOP2_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOP2_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOP2_SRC_SDWA 249 +#define GFX9_VOP2_SRC_DDP16 250 +#define GFX9_VOP2_SRC_VCCZ 251 +#define GFX9_VOP2_SRC_EXECZ 252 +#define GFX9_VOP2_SRC_SCC 253 +#define GFX9_VOP2_SRC_LITERAL_CONST 255 +#define GFX9_VOP2_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_vop1 { + u32 src0:9; + u32 op:8; + u32 vdst:8; + u32 encoding:7; + u32 literal; +}; + +enum amdgcn_gfx9_vop1_opcode { + GFX9_V_NOP, + GFX9_V_MOV_B32, + GFX9_V_READFIRSTLANE_B32, + GFX9_V_CVT_I32_F64, + GFX9_V_CVT_F64_I32, + GFX9_V_CVT_F32_I32, + GFX9_V_CVT_F32_U32, + GFX9_V_CVT_U32_F32, + GFX9_V_CVT_I32_F32 =3D 8, + GFX9_V_CVT_F16_F32 =3D 10, + GFX9_V_CVT_F32_F16, + GFX9_V_CVT_RPI_I32_F32, + GFX9_V_CVT_FLR_I32_F32, + GFX9_V_CVT_OFF_F32_I4, + GFX9_V_CVT_F32_F64, + GFX9_V_CVT_F64_F32, + GFX9_V_CVT_F32_UBYTE0, + GFX9_V_CVT_F32_UBYTE1, + GFX9_V_CVT_F32_UBYTE2, + GFX9_V_CVT_F32_UBYTE3, + GFX9_V_CVT_U32_F64, + GFX9_V_CVT_F64_U32, + GFX9_V_TRUNC_F64, + GFX9_V_CEIL_F64, + GFX9_V_RNDNE_F64, + GFX9_V_FLOOR_F64, + GFX9_V_FRACT_F32, + GFX9_V_TRUNC_F32, + GFX9_V_CEIL_F32, + GFX9_V_RNDNE_F32, + GFX9_V_FLOOR_F32, + GFX9_V_EXP_F32, + GFX9_V_LOG_F32, + GFX9_V_RCP_F32, + GFX9_V_RCP_IFLAG_F32, + GFX9_V_RSQ_F32, + GFX9_V_RCP_F64, + GFX9_V_RSQ_F64, + GFX9_V_SQRT_F32, + GFX9_V_SQRT_F64, + GFX9_V_SIN_F32, + GFX9_V_COS_F32, + GFX9_V_NOT_B32, + GFX9_V_BFREV_B32, + GFX9_V_FFBH_U32, + GFX9_V_FFBL_B32, + GFX9_V_FFBH_I32, + GFX9_V_FREXP_EXP_I32_F64, + GFX9_V_FREXP_MANT_F64, + GFX9_V_FRACT_F64, + GFX9_V_FREXP_EXP_I32_F32, + GFX9_V_FREXP_MANT_F32, + GFX9_V_CLREXCP, + /* ISA opcodes 54 and 56 are not defined (reserved gaps). */ + GFX9_V_SCREEN_PARTITION_4SE_B32 =3D 55, + GFX9_V_CVT_F16_U16 =3D 57, + GFX9_V_CVT_F16_I16, + GFX9_V_CVT_U16_F16, + GFX9_V_CVT_I16_F16, + GFX9_V_RCP_F16, + GFX9_V_SQRT_F16, + GFX9_V_RSQ_F16, + GFX9_V_LOG_F16, + GFX9_V_EXP_F16, + GFX9_V_FREXP_MANT_F16, + GFX9_V_FREXP_EXP_I16_F16, + GFX9_V_FLOOR_F16, + GFX9_V_CEIL_F16, + GFX9_V_TRUNC_F16, + GFX9_V_RNDNE_F16, + GFX9_V_FRACT_F16, + GFX9_V_SIN_F16, + GFX9_V_COS_F16, + GFX9_V_EXP_LEGACY_F32, + GFX9_V_LOG_LEGACY_F32, + GFX9_V_CVT_NORM_I16_F16, + GFX9_V_CVT_NORM_U16_F16, + GFX9_V_SAT_PK_U8_I16 =3D 79, + GFX9_V_SWAP_B32 =3D 81, +}; + +#define GFX9_VOP1_ENCODING 0x3f +#define GFX9_VOP1_SRC_SGPR_BASE 0 +#define GFX9_VOP1_SRC_VCC_LO 106 +#define GFX9_VOP1_SRC_VCC_HI 107 +#define GFX9_VOP1_SRC_TTPM_BASE 108 +#define GFX9_VOP1_SRC_M0 124 +#define GFX9_VOP1_SRC_NULL 125 +#define GFX9_VOP1_SRC_EXEC_LO 126 +#define GFX9_VOP1_SRC_EXEC_HI 127 +#define GFX9_VOP1_SRC_INTEGER_0 128 +#define GFX9_VOP1_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOP1_SRC_SHARED_BASE 235 +#define GFX9_VOP1_SRC_SHARED_LIMIT 236 +#define GFX9_VOP1_SRC_PRIVATE_BASE 237 +#define GFX9_VOP1_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOP1_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOP1_SRC_SDWA 249 +#define GFX9_VOP1_SRC_DDP16 250 +#define GFX9_VOP1_SRC_VCCZ 251 +#define GFX9_VOP1_SRC_EXECZ 252 +#define GFX9_VOP1_SRC_SCC 253 +#define GFX9_VOP1_SRC_LITERAL_CONST 255 +#define GFX9_VOP1_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_vopc { + u32 src0:9; + u32 vsrc1:8; + u32 op:8; + u32 encoding:7; + u32 literal; +}; + +enum amdgcn_gfx9_vopc_compare_offset16 { + GFX9_VOPC16_F =3D 0, + GFX9_VOPC16_LT, + GFX9_VOPC16_EQ, + GFX9_VOPC16_LE, + GFX9_VOPC16_GT, + GFX9_VOPC16_LG, + GFX9_VOPC16_GE, + GFX9_VOPC16_O, + GFX9_VOPC16_U, + GFX9_VOPC16_NGE, + GFX9_VOPC16_NLG, + GFX9_VOPC16_NGT, + GFX9_VOPC16_NLE, + GFX9_VOPC16_NEQ, + GFX9_VOPC16_NLT, + GFX9_VOPC16_TRU, +}; + +enum amdgcn_gfx9_vopc_compare_offset8 { + GFX9_VOPC8_F =3D 0, + GFX9_VOPC8_LT, + GFX9_VOPC8_EQ, + GFX9_VOPC8_LE, + GFX9_VOPC8_GT, + GFX9_VOPC8_LG, + GFX9_VOPC8_GE, + GFX9_VOPC8_TRU, +}; + +enum amdgcn_gfx9_vopc_opcode { + GFX9_V_CMP_CLASS_F32 =3D 16, + GFX9_V_CMPX_CLASS_F32, + GFX9_V_CMP_CLASS_F64, + GFX9_V_CMPX_CLASS_F64, + GFX9_V_CMP_CLASS_F16, + GFX9_V_CMPX_CLASS_F16 =3D 21, + GFX9_V_CMP_F_F16 =3D 32, + GFX9_V_CMP_LT_F16, + GFX9_V_CMP_EQ_F16, + GFX9_V_CMP_LE_F16, + GFX9_V_CMP_GT_F16, + GFX9_V_CMP_LG_F16, + GFX9_V_CMP_GE_F16, + GFX9_V_CMP_O_F16, + GFX9_V_CMP_U_F16, + GFX9_V_CMP_NGE_F16, + GFX9_V_CMP_NLG_F16, + GFX9_V_CMP_NGT_F16, + GFX9_V_CMP_NLE_F16, + GFX9_V_CMP_NEQ_F16, + GFX9_V_CMP_NLT_F16, + GFX9_V_CMP_TRU_F16, + GFX9_V_CMPX_F_F16, + GFX9_V_CMPX_LT_F16, + GFX9_V_CMPX_EQ_F16, + GFX9_V_CMPX_LE_F16, + GFX9_V_CMPX_GT_F16, + GFX9_V_CMPX_LG_F16, + GFX9_V_CMPX_GE_F16, + GFX9_V_CMPX_O_F16, + GFX9_V_CMPX_U_F16, + GFX9_V_CMPX_NGE_F16, + GFX9_V_CMPX_NLG_F16, + GFX9_V_CMPX_NGT_F16, + GFX9_V_CMPX_NLE_F16, + GFX9_V_CMPX_NEQ_F16, + GFX9_V_CMPX_NLT_F16, + GFX9_V_CMPX_TRU_F16, + GFX9_V_CMP_F_F32, + GFX9_V_CMP_LT_F32, + GFX9_V_CMP_EQ_F32, + GFX9_V_CMP_LE_F32, + GFX9_V_CMP_GT_F32, + GFX9_V_CMP_LG_F32, + GFX9_V_CMP_GE_F32, + GFX9_V_CMP_O_F32, + GFX9_V_CMP_U_F32, + GFX9_V_CMP_NGE_F32, + GFX9_V_CMP_NLG_F32, + GFX9_V_CMP_NGT_F32, + GFX9_V_CMP_NLE_F32, + GFX9_V_CMP_NEQ_F32, + GFX9_V_CMP_NLT_F32, + GFX9_V_CMP_TRU_F32, + GFX9_V_CMPX_F_F32, + GFX9_V_CMPX_LT_F32, + GFX9_V_CMPX_EQ_F32, + GFX9_V_CMPX_LE_F32, + GFX9_V_CMPX_GT_F32, + GFX9_V_CMPX_LG_F32, + GFX9_V_CMPX_GE_F32, + GFX9_V_CMPX_O_F32, + GFX9_V_CMPX_U_F32, + GFX9_V_CMPX_NGE_F32, + GFX9_V_CMPX_NLG_F32, + GFX9_V_CMPX_NGT_F32, + GFX9_V_CMPX_NLE_F32, + GFX9_V_CMPX_NEQ_F32, + GFX9_V_CMPX_NLT_F32, + GFX9_V_CMPX_TRU_F32, + GFX9_V_CMP_F_F64, + GFX9_V_CMP_LT_F64, + GFX9_V_CMP_EQ_F64, + GFX9_V_CMP_LE_F64, + GFX9_V_CMP_GT_F64, + GFX9_V_CMP_LG_F64, + GFX9_V_CMP_GE_F64, + GFX9_V_CMP_O_F64, + GFX9_V_CMP_U_F64, + GFX9_V_CMP_NGE_F64, + GFX9_V_CMP_NLG_F64, + GFX9_V_CMP_NGT_F64, + GFX9_V_CMP_NLE_F64, + GFX9_V_CMP_NEQ_F64, + GFX9_V_CMP_NLT_F64, + GFX9_V_CMP_TRU_F64, + GFX9_V_CMPX_F_F64, + GFX9_V_CMPX_LT_F64, + GFX9_V_CMPX_EQ_F64, + GFX9_V_CMPX_LE_F64, + GFX9_V_CMPX_GT_F64, + GFX9_V_CMPX_LG_F64, + GFX9_V_CMPX_GE_F64, + GFX9_V_CMPX_O_F64, + GFX9_V_CMPX_U_F64, + GFX9_V_CMPX_NGE_F64, + GFX9_V_CMPX_NLG_F64, + GFX9_V_CMPX_NGT_F64, + GFX9_V_CMPX_NLE_F64, + GFX9_V_CMPX_NEQ_F64, + GFX9_V_CMPX_NLT_F64, + GFX9_V_CMPX_TRU_F64 =3D 127, + GFX9_V_CMP_F_I16 =3D 160, + GFX9_V_CMP_LT_I16, + GFX9_V_CMP_EQ_I16, + GFX9_V_CMP_LE_I16, + GFX9_V_CMP_GT_I16, + GFX9_V_CMP_NE_I16, + GFX9_V_CMP_GE_I16, + GFX9_V_CMP_T_I16, + GFX9_V_CMP_F_U16, + GFX9_V_CMP_LT_U16, + GFX9_V_CMP_EQ_U16, + GFX9_V_CMP_LE_U16, + GFX9_V_CMP_GT_U16, + GFX9_V_CMP_NE_U16, + GFX9_V_CMP_GE_U16, + GFX9_V_CMP_T_U16, + GFX9_V_CMPX_F_I16, + GFX9_V_CMPX_LT_I16, + GFX9_V_CMPX_EQ_I16, + GFX9_V_CMPX_LE_I16, + GFX9_V_CMPX_GT_I16, + GFX9_V_CMPX_NE_I16, + GFX9_V_CMPX_GE_I16, + GFX9_V_CMPX_T_I16, + GFX9_V_CMPX_F_U16, + GFX9_V_CMPX_LT_U16, + GFX9_V_CMPX_EQ_U16, + GFX9_V_CMPX_LE_U16, + GFX9_V_CMPX_GT_U16, + GFX9_V_CMPX_NE_U16, + GFX9_V_CMPX_GE_U16, + GFX9_V_CMPX_T_U16, + GFX9_V_CMP_F_I32, + GFX9_V_CMP_LT_I32, + GFX9_V_CMP_EQ_I32, + GFX9_V_CMP_LE_I32, + GFX9_V_CMP_GT_I32, + GFX9_V_CMP_NE_I32, + GFX9_V_CMP_GE_I32, + GFX9_V_CMP_T_I32, + GFX9_V_CMP_F_U32, + GFX9_V_CMP_LT_U32, + GFX9_V_CMP_EQ_U32, + GFX9_V_CMP_LE_U32, + GFX9_V_CMP_GT_U32, + GFX9_V_CMP_NE_U32, + GFX9_V_CMP_GE_U32, + GFX9_V_CMP_T_U32, + GFX9_V_CMPX_F_I32, + GFX9_V_CMPX_LT_I32, + GFX9_V_CMPX_EQ_I32, + GFX9_V_CMPX_LE_I32, + GFX9_V_CMPX_GT_I32, + GFX9_V_CMPX_NE_I32, + GFX9_V_CMPX_GE_I32, + GFX9_V_CMPX_T_I32, + GFX9_V_CMPX_F_U32, + GFX9_V_CMPX_LT_U32, + GFX9_V_CMPX_EQ_U32, + GFX9_V_CMPX_LE_U32, + GFX9_V_CMPX_GT_U32, + GFX9_V_CMPX_NE_U32, + GFX9_V_CMPX_GE_U32, + GFX9_V_CMPX_T_U32, + GFX9_V_CMP_F_I64, + GFX9_V_CMP_LT_I64, + GFX9_V_CMP_EQ_I64, + GFX9_V_CMP_LE_I64, + GFX9_V_CMP_GT_I64, + GFX9_V_CMP_NE_I64, + GFX9_V_CMP_GE_I64, + GFX9_V_CMP_T_I64, + GFX9_V_CMP_F_U64, + GFX9_V_CMP_LT_U64, + GFX9_V_CMP_EQ_U64, + GFX9_V_CMP_LE_U64, + GFX9_V_CMP_GT_U64, + GFX9_V_CMP_NE_U64, + GFX9_V_CMP_GE_U64, + GFX9_V_CMP_T_U64, + GFX9_V_CMPX_F_I64, + GFX9_V_CMPX_LT_I64, + GFX9_V_CMPX_EQ_I64, + GFX9_V_CMPX_LE_I64, + GFX9_V_CMPX_GT_I64, + GFX9_V_CMPX_NE_I64, + GFX9_V_CMPX_GE_I64, + GFX9_V_CMPX_T_I64, + GFX9_V_CMPX_F_U64, + GFX9_V_CMPX_LT_U64, + GFX9_V_CMPX_EQ_U64, + GFX9_V_CMPX_LE_U64, + GFX9_V_CMPX_GT_U64, + GFX9_V_CMPX_NE_U64, + GFX9_V_CMPX_GE_U64, + GFX9_V_CMPX_T_U64, +}; + +#define GFX9_VOPC_ENCODING 0x3e +#define GFX9_VOPC_SRC_SGPR_BASE 0 +#define GFX9_VOPC_SRC_VCC_LO 106 +#define GFX9_VOPC_SRC_VCC_HI 107 +#define GFX9_VOPC_SRC_TTPM_BASE 108 +#define GFX9_VOPC_SRC_M0 124 +#define GFX9_VOPC_SRC_NULL 125 +#define GFX9_VOPC_SRC_EXEC_LO 126 +#define GFX9_VOPC_SRC_EXEC_HI 127 +#define GFX9_VOPC_SRC_INTEGER_0 128 +#define GFX9_VOPC_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOPC_SRC_SHARED_BASE 235 +#define GFX9_VOPC_SRC_SHARED_LIMIT 236 +#define GFX9_VOPC_SRC_PRIVATE_BASE 237 +#define GFX9_VOPC_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOPC_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOPC_SRC_SDWA 249 +#define GFX9_VOPC_SRC_DDP16 250 +#define GFX9_VOPC_SRC_VCCZ 251 +#define GFX9_VOPC_SRC_EXECZ 252 +#define GFX9_VOPC_SRC_SCC 253 +#define GFX9_VOPC_SRC_LITERAL_CONST 255 +#define GFX9_VOPC_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_vop3a { + u64 vdst:8; + u64 abs:3; + u64 op_sel:4; + u64 clmp:1; + u64 op:10; + u64 encoding:6; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 omod:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx9_vop3a_opcode { + GFX9_V_MAD_LEGACY_F32 =3D 448, + GFX9_V_MAD_F32, + GFX9_V_MAD_I32_I24, + GFX9_V_MAD_U32_U24, + GFX9_V_CUBEID_F32, + GFX9_V_CUBESC_F32, + GFX9_V_CUBETC_F32, + GFX9_V_CUBEMA_F32, + GFX9_V_BFE_U32, + GFX9_V_BFE_I32, + GFX9_V_BFI_B32, + GFX9_V_FMA_F32, + GFX9_V_FMA_F64, + GFX9_V_LERP_U8, + GFX9_V_ALIGNBIT_B32, + GFX9_V_ALIGNBYTE_B32, + GFX9_V_MIN3_F32, + GFX9_V_MIN3_I32, + GFX9_V_MIN3_U32, + GFX9_V_MAX3_F32, + GFX9_V_MAX3_I32, + GFX9_V_MAX3_U32, + GFX9_V_MED3_F32, + GFX9_V_MED3_I32, + GFX9_V_MED3_U32, + GFX9_V_SAD_U8, + GFX9_V_SAD_HI_U8, + GFX9_V_SAD_U16, + GFX9_V_SAD_U32, + GFX9_V_CVT_PK_U8_F32, + GFX9_V_DIV_FIXUP_F32, + GFX9_V_DIV_FIXUP_F64 =3D 479, + GFX9_V_DIV_FMAS_F32 =3D 482, + GFX9_V_DIV_FMAS_F64, + GFX9_V_MSAD_U8, + GFX9_V_QSAD_PK_U16_U8, + GFX9_V_MQSAD_PK_U16_U8, + GFX9_V_MQSAD_U32_U8 =3D 487, + GFX9_V_MAD_LEGACY_F16 =3D 490, + GFX9_V_MAD_LEGACY_U16, + GFX9_V_MAD_LEGACY_I16, + GFX9_V_PERM_B32, + GFX9_V_FMA_LEGACY_F16, + GFX9_V_DIV_FIXUP_LEGACY_F16, + GFX9_V_CVT_PKACCUM_U8_F32, + GFX9_V_MAD_U32_U16, + GFX9_V_MAD_I32_I16, + GFX9_V_XAD_U32, + GFX9_V_MIN3_F16, + GFX9_V_MIN3_I16, + GFX9_V_MIN3_U16, + GFX9_V_MAX3_F16, + GFX9_V_MAX3_I16, + GFX9_V_MAX3_U16, + GFX9_V_MED3_F16, + GFX9_V_MED3_I16, + GFX9_V_MED3_U16, + GFX9_V_LSHL_ADD_U32, + GFX9_V_ADD_LSHL_U32, + GFX9_V_ADD3_U32, + GFX9_V_LSHL_OR_B32, + GFX9_V_AND_OR_B32, + GFX9_V_OR3_B32, + GFX9_V_MAD_F16, + GFX9_V_MAD_U16, + GFX9_V_MAD_I16, + GFX9_V_FMA_F16, + GFX9_V_DIV_FIXUP_F16 =3D 519, + GFX9_V_INTERP_P1LL_F16 =3D 628, + GFX9_V_INTERP_P1LV_F16, + GFX9_V_INTERP_P2_LEGACY_F16, + GFX9_V_INTERP_P2_F16 =3D 631, + GFX9_V_ADD_F64 =3D 640, + GFX9_V_MUL_F64, + GFX9_V_MIN_F64, + GFX9_V_MAX_F64, + GFX9_V_LDEXP_F64, + GFX9_V_MUL_LO_U32, + GFX9_V_MUL_HI_U32, + GFX9_V_MUL_HI_I32, + GFX9_V_LDEXP_F32, + GFX9_V_READLANE_B32, + GFX9_V_WRITELANE_B32, + GFX9_V_BCNT_U32_B32, + GFX9_V_MBCNT_LO_U32_B32, + GFX9_V_MBCNT_HI_U32_B32 =3D 653, + GFX9_V_LSHLREV_B64 =3D 655, + GFX9_V_LSHRREV_B64, + GFX9_V_ASHRREV_I64, + GFX9_V_TRIG_PREOP_F64, + GFX9_V_BFM_B32, + GFX9_V_CVT_PKNORM_I16_F32, + GFX9_V_CVT_PKNORM_U16_F32, + GFX9_V_CVT_PKRTZ_F16_F32, + GFX9_V_CVT_PK_U16_U32, + GFX9_V_CVT_PK_I16_I32, + GFX9_V_CVT_PKNORM_I16_F16, + GFX9_V_CVT_PKNORM_U16_F16 =3D 666, + GFX9_V_ADD_I32 =3D 668, + GFX9_V_SUB_I32, + GFX9_V_ADD_I16, + GFX9_V_SUB_I16, + GFX9_V_PACK_B32_F16, +}; + +enum amdgcn_gfx9_vop3a_abs { + GFX9_VOP3A_ABS_SRC0, + GFX9_VOP3A_ABS_SRC1, + GFX9_VOP3A_ABS_SRC2, +}; + +#define GFX9_VOP3A_ENCODING 0x34 +#define GFX9_VOP3A_SRC_SGPR_BASE 0 +#define GFX9_VOP3A_SRC_VCC_LO 106 +#define GFX9_VOP3A_SRC_VCC_HI 107 +#define GFX9_VOP3A_SRC_TTPM_BASE 108 +#define GFX9_VOP3A_SRC_M0 124 +#define GFX9_VOP3A_SRC_NULL 125 +#define GFX9_VOP3A_SRC_EXEC_LO 126 +#define GFX9_VOP3A_SRC_EXEC_HI 127 +#define GFX9_VOP3A_SRC_INTEGER_0 128 +#define GFX9_VOP3A_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOP3A_SRC_SHARED_BASE 235 +#define GFX9_VOP3A_SRC_SHARED_LIMIT 236 +#define GFX9_VOP3A_SRC_PRIVATE_BASE 237 +#define GFX9_VOP3A_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOP3A_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOP3A_SRC_SDWA 249 +#define GFX9_VOP3A_SRC_DDP16 250 +#define GFX9_VOP3A_SRC_VCCZ 251 +#define GFX9_VOP3A_SRC_EXECZ 252 +#define GFX9_VOP3A_SRC_SCC 253 +#define GFX9_VOP3A_SRC_LITERAL_CONST 255 +#define GFX9_VOP3A_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_vop3b { + u64 vdst:8; + u64 sdst:7; + u64 clmp:1; + u64 op:10; + u64 encoding:6; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 omod:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx9_vop3b_opcode { + GFX9_V_DIV_SCALE_F64 =3D 481, + GFX9_V_MAD_U64_U32 =3D 488, + GFX9_V_MAD_I64_I32 =3D 489, +}; + +#define GFX9_VOP3B_ENCODING 0x34 +#define GFX9_VOP3B_SRC_SGPR_BASE 0 +#define GFX9_VOP3B_SRC_VCC_LO 106 +#define GFX9_VOP3B_SRC_VCC_HI 107 +#define GFX9_VOP3B_SRC_TTPM_BASE 108 +#define GFX9_VOP3B_SRC_M0 124 +#define GFX9_VOP3B_SRC_NULL 125 +#define GFX9_VOP3B_SRC_EXEC_LO 126 +#define GFX9_VOP3B_SRC_EXEC_HI 127 +#define GFX9_VOP3B_SRC_INTEGER_0 128 +#define GFX9_VOP3B_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOP3B_SRC_SHARED_BASE 235 +#define GFX9_VOP3B_SRC_SHARED_LIMIT 236 +#define GFX9_VOP3B_SRC_PRIVATE_BASE 237 +#define GFX9_VOP3B_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOP3B_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOP3B_SRC_SDWA 249 +#define GFX9_VOP3B_SRC_DDP16 250 +#define GFX9_VOP3B_SRC_VCCZ 251 +#define GFX9_VOP3B_SRC_EXECZ 252 +#define GFX9_VOP3B_SRC_SCC 253 +#define GFX9_VOP3B_SRC_LITERAL_CONST 255 +#define GFX9_VOP3B_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_vop3p { + u64 vdst:8; + u64 neg_hi:3; + u64 op_sel:3; + u64 op_sel_hi2:1; + u64 clmp:1; + u64 op:7; + u64 encoding:9; + u64 src0:9; + u64 src1:9; + u64 src2:9; + u64 op_sel_hi:2; + u64 neg:3; + u32 literal; +}; + +enum amdgcn_gfx9_vop3p_opcode { + GFX9_V_PK_MAD_I16, + GFX9_V_PK_MUL_LO_U16, + GFX9_V_PK_ADD_I16, + GFX9_V_PK_SUB_I16, + GFX9_V_PK_LSHLREV_B16, + GFX9_V_PK_LSHRREV_B16, + GFX9_V_PK_ASHRREV_I16, + GFX9_V_PK_MAX_I16, + GFX9_V_PK_MIN_I16, + GFX9_V_PK_MAD_U16, + GFX9_V_PK_ADD_U16, + GFX9_V_PK_SUB_U16, + GFX9_V_PK_MAX_U16, + GFX9_V_PK_MIN_U16, + GFX9_V_PK_FMA_F16, + GFX9_V_PK_ADD_F16, + GFX9_V_PK_MUL_F16, + GFX9_V_PK_MIN_F16, + GFX9_V_PK_MAX_F16 =3D 18, + GFX9_V_MAD_MIX_F32 =3D 32, + GFX9_V_MAD_MIXLO_F16, + GFX9_V_MAD_MIXHI_F16, +}; + +#define GFX9_VOP3P_ENCODING 0x1a7 +#define GFX9_VOP3P_SRC_SGPR_BASE 0 +#define GFX9_VOP3P_SRC_VCC_LO 106 +#define GFX9_VOP3P_SRC_VCC_HI 107 +#define GFX9_VOP3P_SRC_TTPM_BASE 108 +#define GFX9_VOP3P_SRC_M0 124 +#define GFX9_VOP3P_SRC_NULL 125 +#define GFX9_VOP3P_SRC_EXEC_LO 126 +#define GFX9_VOP3P_SRC_EXEC_HI 127 +#define GFX9_VOP3P_SRC_INTEGER_0 128 +#define GFX9_VOP3P_SRC_INTEGER_MINUS_1 193 +#define GFX9_VOP3P_SRC_SHARED_BASE 235 +#define GFX9_VOP3P_SRC_SHARED_LIMIT 236 +#define GFX9_VOP3P_SRC_PRIVATE_BASE 237 +#define GFX9_VOP3P_SRC_PRIVATE_LIMIT 238 +#define GFX9_VOP3P_SRC_POPS_EXITING_WAVE_ID 239 +#define GFX9_VOP3P_SRC_SDWA 249 +#define GFX9_VOP3P_SRC_DDP16 250 +#define GFX9_VOP3P_SRC_VCCZ 251 +#define GFX9_VOP3P_SRC_EXECZ 252 +#define GFX9_VOP3P_SRC_SCC 253 +#define GFX9_VOP3P_SRC_VGPR_BASE 256 + +struct amdgcn_gfx9_sdwa { + u32 src0:8; + u32 dst_sel:3; + u32 dst_u:2; + u32 clmp:1; + u32 omod:2; + u32 src0_sel:3; + u32 src0_sext:1; + u32 src0_neg:1; + u32 src0_abs:1; + u32 dummy1:1; + u32 s0:1; + u32 src1_sel:3; + u32 src1_sext:1; + u32 src1_neg:1; + u32 src1_abs:1; + u32 dummy2:1; + u32 s1:1; +}; + +struct amdgcn_gfx9_sdwab { + u32 src0:8; + u32 sdst:7; + u32 sd:1; + u32 src0_sel:3; + u32 src0_sext:1; + u32 src0_neg:1; + u32 src0_abs:1; + u32 dummy1:1; + u32 s0:1; + u32 src1_sel:3; + u32 src1_sext:1; + u32 src1_neg:1; + u32 src1_abs:1; + u32 dummy2:1; + u32 s1:1; +}; + +struct amdgcn_gfx9_dpp16 { +}; + +struct amdgcn_gfx9_dpp8 { +}; + +/* Vector Parameter Interpolation Format */ +struct amdgcn_gfx9_vintrp { +}; + +/* LDS and GDS Format */ +struct amdgcn_gfx9_ds { + u64 offset0:8; + u64 offset1:8; + u64 gds:1; + u64 op:9; /* bits [25:17] */ + u64 encoding:6; /* bits [31:26] =3D 0x36 */ + u64 addr:8; + u64 data0:8; + u64 data1:8; + u64 vdst:8; +}; + +enum amdgcn_gfx9_ds_opcode { + GFX9_DS_ADD_U32, + GFX9_DS_SUB_U32, + GFX9_DS_RSUB_U32, + GFX9_DS_INC_U32, + GFX9_DS_DEC_U32, + GFX9_DS_MIN_I32, + GFX9_DS_MAX_I32, + GFX9_DS_MIN_U32, + GFX9_DS_MAX_U32, + GFX9_DS_AND_B32, + GFX9_DS_OR_B32, + GFX9_DS_XOR_B32, + GFX9_DS_MSKOR_B32, + GFX9_DS_WRITE_B32, + GFX9_DS_WRITE2_B32, + GFX9_DS_WRITE2ST64_B32, + GFX9_DS_CMPST_B32, + GFX9_DS_CMPST_F32, + GFX9_DS_MIN_F32, + GFX9_DS_MAX_F32, + GFX9_DS_NOP, + GFX9_DS_ADD_F32 =3D 21, + GFX9_DS_WRITE_ADDTID_B32 =3D 29, + GFX9_DS_WRITE_B8, + GFX9_DS_WRITE_B16, + GFX9_DS_ADD_RTN_U32, + GFX9_DS_SUB_RTN_U32, + GFX9_DS_RSUB_RTN_U32, + GFX9_DS_INC_RTN_U32, + GFX9_DS_DEC_RTN_U32, + GFX9_DS_MIN_RTN_I32, + GFX9_DS_MAX_RTN_I32, + GFX9_DS_MIN_RTN_U32, + GFX9_DS_MAX_RTN_U32, + GFX9_DS_AND_RTN_B32, + GFX9_DS_OR_RTN_B32, + GFX9_DS_XOR_RTN_B32, + GFX9_DS_MSKOR_RTN_B32, + GFX9_DS_WRXCHG_RTN_B32, + GFX9_DS_WRXCHG2_RTN_B32, + GFX9_DS_WRXCHG2ST64_RTN_B32, + GFX9_DS_CMPST_RTN_B32, + GFX9_DS_CMPST_RTN_F32, + GFX9_DS_MIN_RTN_F32, + GFX9_DS_MAX_RTN_F32, + GFX9_DS_WRAP_RTN_B32, + GFX9_DS_ADD_RTN_F32, + GFX9_DS_READ_B32, + GFX9_DS_READ2_B32, + GFX9_DS_READ2ST64_B32, + GFX9_DS_READ_I8, + GFX9_DS_READ_U8, + GFX9_DS_READ_I16, + GFX9_DS_READ_U16, + GFX9_DS_SWIZZLE_B32, + GFX9_DS_PERMUTE_B32, + GFX9_DS_BPERMUTE_B32, + GFX9_DS_ADD_U64, + GFX9_DS_SUB_U64, + GFX9_DS_RSUB_U64, + GFX9_DS_INC_U64, + GFX9_DS_DEC_U64, + GFX9_DS_MIN_I64, + GFX9_DS_MAX_I64, + GFX9_DS_MIN_U64, + GFX9_DS_MAX_U64, + GFX9_DS_AND_B64, + GFX9_DS_OR_B64, + GFX9_DS_XOR_B64, + GFX9_DS_MSKOR_B64, + GFX9_DS_WRITE_B64, + GFX9_DS_WRITE2_B64, + GFX9_DS_WRITE2ST64_B64, + GFX9_DS_CMPST_B64, + GFX9_DS_CMPST_F64, + GFX9_DS_MIN_F64, + GFX9_DS_MAX_F64, + GFX9_DS_WRITE_B8_D16_HI, + GFX9_DS_WRITE_B16_D16_HI, + GFX9_DS_READ_U8_D16, + GFX9_DS_READ_U8_D16_HI, + GFX9_DS_READ_I8_D16, + GFX9_DS_READ_I8_D16_HI, + GFX9_DS_READ_U16_D16, + GFX9_DS_READ_U16_D16_HI =3D 91, + GFX9_DS_ADD_RTN_U64 =3D 96, + GFX9_DS_SUB_RTN_U64, + GFX9_DS_RSUB_RTN_U64, + GFX9_DS_INC_RTN_U64, + GFX9_DS_DEC_RTN_U64, + GFX9_DS_MIN_RTN_I64, + GFX9_DS_MAX_RTN_I64, + GFX9_DS_MIN_RTN_U64, + GFX9_DS_MAX_RTN_U64, + GFX9_DS_AND_RTN_B64, + GFX9_DS_OR_RTN_B64, + GFX9_DS_XOR_RTN_B64, + GFX9_DS_MSKOR_RTN_B64, + GFX9_DS_WRXCHG_RTN_B64, + GFX9_DS_WRXCHG2_RTN_B64, + GFX9_DS_WRXCHG2ST64_RTN_B64, + GFX9_DS_CMPST_RTN_B64, + GFX9_DS_CMPST_RTN_F64, + GFX9_DS_MIN_RTN_F64, + GFX9_DS_MAX_RTN_F64 =3D 115, + GFX9_DS_READ_B64 =3D 118, + GFX9_DS_READ2_B64 =3D 119, + GFX9_DS_READ2ST64_B64 =3D 120, + /* ISA opcodes 121-125 are reserved gaps. */ + GFX9_DS_CONDXCHG32_RTN_B64 =3D 126, + /* ISA opcode 127 is a reserved gap. */ + GFX9_DS_ADD_SRC2_U32 =3D 128, + GFX9_DS_SUB_SRC2_U32, + GFX9_DS_RSUB_SRC2_U32, + GFX9_DS_INC_SRC2_U32, + GFX9_DS_DEC_SRC2_U32, + GFX9_DS_MIN_SRC2_I32, + GFX9_DS_MAX_SRC2_I32, + GFX9_DS_MIN_SRC2_U32, + GFX9_DS_MAX_SRC2_U32, + GFX9_DS_AND_SRC2_B32, + GFX9_DS_OR_SRC2_B32, + GFX9_DS_XOR_SRC2_B32 =3D 139, + GFX9_DS_WRITE_SRC2_B32 =3D 141, + GFX9_DS_MIN_SRC2_F32 =3D 146, + GFX9_DS_MAX_SRC2_F32 =3D 147, + GFX9_DS_ADD_SRC2_F32 =3D 149, + GFX9_DS_GWS_SEMA_RELEASE_ALL =3D 152, + GFX9_DS_GWS_INIT, + GFX9_DS_GWS_SEMA_V, + GFX9_DS_GWS_SEMA_BR, + GFX9_DS_GWS_SEMA_P, + GFX9_DS_GWS_BARRIER =3D 157, + GFX9_DS_READ_ADDTID_B32 =3D 182, + GFX9_DS_CONSUME =3D 189, + GFX9_DS_APPEND, + GFX9_DS_ORDERED_COUNT, + GFX9_DS_ADD_SRC2_U64, + GFX9_DS_SUB_SRC2_U64, + GFX9_DS_RSUB_SRC2_U64, + GFX9_DS_INC_SRC2_U64, + GFX9_DS_DEC_SRC2_U64, + GFX9_DS_MIN_SRC2_I64, + GFX9_DS_MAX_SRC2_I64, + GFX9_DS_MIN_SRC2_U64, + GFX9_DS_MAX_SRC2_U64, + GFX9_DS_AND_SRC2_B64, + GFX9_DS_OR_SRC2_B64, + GFX9_DS_XOR_SRC2_B64 =3D 203, + GFX9_DS_WRITE_SRC2_B64 =3D 205, + GFX9_DS_MIN_SRC2_F64 =3D 210, + GFX9_DS_MAX_SRC2_F64 =3D 211, + GFX9_DS_WRITE_B96 =3D 222, + GFX9_DS_WRITE_B128 =3D 223, + GFX9_DS_READ_B96 =3D 254, + GFX9_DS_READ_B128 =3D 255, +}; + +#define GFX9_DS_ENCODING 0x36 +#define GFX9_DS_GDS 1 +#define GFX9_DS_LDS 0 + +//////////////////////////////////////////////////// +/* Vector Memory Buffer Formats */ +struct amdgcn_gfx9_mtbuf { + u64 offset:12; + u64 offen:1; + u64 idxen:1; + u64 glc:1; + u64 op:4; + u64 dfmt:4; + u64 nfmt:3; + u64 encoding:6; + u64 vaddr:8; + u64 vdata:8; + u64 srsrc:5; + u64 dummy:1; + u64 slc:1; + u64 tfe:1; + u64 soffset:8; +}; + +enum amdgcn_gfx9_mtbuf_opcode { + GFX9_TBUFFER_LOAD_FORMAT_X, + GFX9_TBUFFER_LOAD_FORMAT_XY, + GFX9_TBUFFER_LOAD_FORMAT_XYZ, + GFX9_TBUFFER_LOAD_FORMAT_XYZW, + GFX9_TBUFFER_STORE_FORMAT_X, + GFX9_TBUFFER_STORE_FORMAT_XY, + GFX9_TBUFFER_STORE_FORMAT_XYZ, + GFX9_TBUFFER_STORE_FORMAT_XYZW, + GFX9_TBUFFER_LOAD_FORMAT_D16_X, + GFX9_TBUFFER_LOAD_FORMAT_D16_XY, + GFX9_TBUFFER_LOAD_FORMAT_D16_XYZ, + GFX9_TBUFFER_LOAD_FORMAT_D16_XYZW, + GFX9_TBUFFER_STORE_FORMAT_D16_X, + GFX9_TBUFFER_STORE_FORMAT_D16_XY, + GFX9_TBUFFER_STORE_FORMAT_D16_XYZ, + GFX9_TBUFFER_STORE_FORMAT_D16_XYZW, +}; + +#define GFX9_MUBUF_ENCODING 0x38 +#define GFX9_MUBUF_SOFFSET_SGPR_BASE 0 +#define GFX9_MUBUF_SOFFSET_VCC_LO 106 +#define GFX9_MUBUF_SOFFSET_VCC_HI 107 +#define GFX9_MUBUF_SOFFSET_TTPM_BASE 108 +#define GFX9_MUBUF_SOFFSET_M0 124 +#define GFX9_MUBUF_SOFFSET_NULL 125 +#define GFX9_MUBUF_SOFFSET_EXEC_LO 126 +#define GFX9_MUBUF_SOFFSET_EXEC_HI 127 +#define GFX9_MUBUF_SOFFSET_INTEGER_0 128 +#define GFX9_MUBUF_SOFFSET_INTEGER_MINUS_1 193 +#define GFX9_MUBUF_SOFFSET_SHARED_BASE 235 +#define GFX9_MUBUF_SOFFSET_SHARED_LIMIT 236 +#define GFX9_MUBUF_SOFFSET_PRIVATE_BASE 237 +#define GFX9_MUBUF_SOFFSET_PRIVATE_LIMIT 238 +#define GFX9_MUBUF_SOFFSET_POPS_EXITING_WAVE_ID 239 +#define GFX9_MUBUF_SOFFSET_VCCZ 251 +#define GFX9_MUBUF_SOFFSET_EXECZ 252 +#define GFX9_MUBUF_SOFFSET_SCC 253 + +struct amdgcn_gfx9_mubuf { + u64 offset:12; + u64 offen:1; + u64 idxen:1; + u64 glc:1; + u64 dummy1:1; + u64 lds:1; + u64 slc:1; + u64 op:7; + u64 dummy2:1; + u64 encoding:6; + u64 vaddr:8; + u64 vdata:8; + u64 srsrc:5; + u64 dummy3:2; + u64 tfe:1; + u64 soffset:8; +}; + +enum amdgcn_gfx9_mubuf_opcode { + GFX9_BUFFER_LOAD_FORMAT_X, + GFX9_BUFFER_LOAD_FORMAT_XY, + GFX9_BUFFER_LOAD_FORMAT_XYZ, + GFX9_BUFFER_LOAD_FORMAT_XYZW, + GFX9_BUFFER_STORE_FORMAT_X, + GFX9_BUFFER_STORE_FORMAT_XY, + GFX9_BUFFER_STORE_FORMAT_XYZ, + GFX9_BUFFER_STORE_FORMAT_XYZW, + GFX9_BUFFER_LOAD_FORMAT_D16_X, + GFX9_BUFFER_LOAD_FORMAT_D16_XY, + GFX9_BUFFER_LOAD_FORMAT_D16_XYZ, + GFX9_BUFFER_LOAD_FORMAT_D16_XYZW, + GFX9_BUFFER_STORE_FORMAT_D16_X, + GFX9_BUFFER_STORE_FORMAT_D16_XY, + GFX9_BUFFER_STORE_FORMAT_D16_XYZ, + GFX9_BUFFER_STORE_FORMAT_D16_XYZW, + GFX9_BUFFER_LOAD_UBYTE, + GFX9_BUFFER_LOAD_SBYTE, + GFX9_BUFFER_LOAD_USHORT, + GFX9_BUFFER_LOAD_SSHORT, + GFX9_BUFFER_LOAD_DWORD, + GFX9_BUFFER_LOAD_DWORDX2, + GFX9_BUFFER_LOAD_DWORDX3, + GFX9_BUFFER_LOAD_DWORDX4, + GFX9_BUFFER_STORE_BYTE, + GFX9_BUFFER_STORE_BYTE_D16_HI, + GFX9_BUFFER_STORE_SHORT, + GFX9_BUFFER_STORE_SHORT_D16_HI, + GFX9_BUFFER_STORE_DWORD, + GFX9_BUFFER_STORE_DWORDX2, + GFX9_BUFFER_STORE_DWORDX3, + GFX9_BUFFER_STORE_DWORDX4, + GFX9_BUFFER_LOAD_UBYTE_D16, + GFX9_BUFFER_LOAD_UBYTE_D16_HI, + GFX9_BUFFER_LOAD_SBYTE_D16, + GFX9_BUFFER_LOAD_SBYTE_D16_HI, + GFX9_BUFFER_LOAD_SHORT_D16, + GFX9_BUFFER_LOAD_SHORT_D16_HI, + GFX9_BUFFER_LOAD_FORMAT_D16_HI_X, + GFX9_BUFFER_STORE_FORMAT_D16_HI_X =3D 39, + GFX9_BUFFER_STORE_LDS_DWORD =3D 61, + GFX9_BUFFER_WBINVL1, + GFX9_BUFFER_WBINVL1_VOL, + GFX9_BUFFER_ATOMIC_SWAP, + GFX9_BUFFER_ATOMIC_CMPSWAP, + GFX9_BUFFER_ATOMIC_ADD, + GFX9_BUFFER_ATOMIC_SUB, + GFX9_BUFFER_ATOMIC_SMIN, + GFX9_BUFFER_ATOMIC_UMIN, + GFX9_BUFFER_ATOMIC_SMAX, + GFX9_BUFFER_ATOMIC_UMAX, + GFX9_BUFFER_ATOMIC_AND, + GFX9_BUFFER_ATOMIC_OR, + GFX9_BUFFER_ATOMIC_XOR, + GFX9_BUFFER_ATOMIC_INC, + GFX9_BUFFER_ATOMIC_DEC =3D 76, + GFX9_BUFFER_ATOMIC_SWAP_X2 =3D 96, + GFX9_BUFFER_ATOMIC_CMPSWAP_X2, + GFX9_BUFFER_ATOMIC_ADD_X2, + GFX9_BUFFER_ATOMIC_SUB_X2, + GFX9_BUFFER_ATOMIC_SMIN_X2, + GFX9_BUFFER_ATOMIC_UMIN_X2, + GFX9_BUFFER_ATOMIC_SMAX_X2, + GFX9_BUFFER_ATOMIC_UMAX_X2, + GFX9_BUFFER_ATOMIC_AND_X2, + GFX9_BUFFER_ATOMIC_OR_X2, + GFX9_BUFFER_ATOMIC_XOR_X2, + GFX9_BUFFER_ATOMIC_INC_X2, + GFX9_BUFFER_ATOMIC_DEC_X2, +}; + +/* Vector Memory Image Format */ +struct amdgcn_gfx9_mimg { +}; + +#define GFX9_FLAT_ENCODING 0x37 +#define GFX9_FLAT_SADDR_SGPR_BASE 0 +#define GFX9_FLAT_SADDR_VCC_LO 106 +#define GFX9_FLAT_SADDR_VCC_HI 107 +#define GFX9_FLAT_SADDR_TTPM_BASE 108 +#define GFX9_FLAT_SADDR_M0 124 +#define GFX9_FLAT_SADDR_NULL 125 +#define GFX9_FLAT_SADDR_EXEC_LO 126 +#define GFX9_FLAT_SADDR_EXEC_HI 127 +#define GFX9_FLAT_SADDR_INTEGER_0 128 +#define GFX9_FLAT_SADDR_INTEGER_MINUS_1 193 +#define GFX9_FLAT_SADDR_SHARED_BASE 235 +#define GFX9_FLAT_SADDR_SHARED_LIMIT 236 +#define GFX9_FLAT_SADDR_PRIVATE_BASE 237 +#define GFX9_FLAT_SADDR_PRIVATE_LIMIT 238 +#define GFX9_FLAT_SADDR_POPS_EXITING_WAVE_ID 239 +#define GFX9_FLAT_SADDR_VCCZ 251 +#define GFX9_FLAT_SADDR_EXECZ 252 +#define GFX9_FLAT_SADDR_SCC 253 + +/* + * Scalar SGPR that provides an offset address. To disable, set this field= to + * 0x7F. Meaning of this field is different for Scratch and Global: + * Flat: Unused. + * Scratch: Use an SGPR (instead of VGPR) for the address. + * Global: Use the SGPR to provide a base address; the VGPR provides a + * 32-bit offset. + */ +#define GFX9_FLAT_SADDR_DISABLE 0x7f +#define GFX9_FLAT_SEG_FLAT 0 +#define GFX9_FLAT_SEG_SCRATCH 1 +#define GFX9_FLAT_SEG_GLOBAL 2 + +/* Flat Formats */ +struct amdgcn_gfx9_flat { + u64 offset:13; + u64 lds:1; + u64 seg:2; + u64 glc:1; + u64 slc:1; + u64 op:7; + u64 dummy:1; + u64 encoding:6; + u64 addr:8; + u64 data:8; + u64 saddr:7; + u64 nv:1; + u64 vdst:8; +}; + +enum amdgcn_gfx9_global_opcode { + GFX9_GLOBAL_LOAD_UBYTE =3D 16, + GFX9_GLOBAL_LOAD_SBYTE, + GFX9_GLOBAL_LOAD_USHORT, + GFX9_GLOBAL_LOAD_SSHORT, + GFX9_GLOBAL_LOAD_DWORD, + GFX9_GLOBAL_LOAD_DWORDX2, + GFX9_GLOBAL_LOAD_DWORDX3, + GFX9_GLOBAL_LOAD_DWORDX4, + GFX9_GLOBAL_STORE_BYTE, + GFX9_GLOBAL_STORE_BYTE_D16_HI, + GFX9_GLOBAL_STORE_SHORT, + GFX9_GLOBAL_STORE_SHORT_D16_HI, + GFX9_GLOBAL_STORE_DWORD, + GFX9_GLOBAL_STORE_DWORDX2, + GFX9_GLOBAL_STORE_DWORDX3, + GFX9_GLOBAL_STORE_DWORDX4, + GFX9_GLOBAL_LOAD_UBYTE_D16, + GFX9_GLOBAL_LOAD_UBYTE_D16_HI, + GFX9_GLOBAL_LOAD_SBYTE_D16, + GFX9_GLOBAL_LOAD_SBYTE_D16_HI, + GFX9_GLOBAL_LOAD_SHORT_D16, + GFX9_GLOBAL_LOAD_SHORT_D16_HI =3D 37, + GFX9_GLOBAL_ATOMIC_SWAP =3D 64, + GFX9_GLOBAL_ATOMIC_CMPSWAP, + GFX9_GLOBAL_ATOMIC_ADD, + GFX9_GLOBAL_ATOMIC_SUB, + GFX9_GLOBAL_ATOMIC_SMIN, + GFX9_GLOBAL_ATOMIC_UMIN, + GFX9_GLOBAL_ATOMIC_SMAX, + GFX9_GLOBAL_ATOMIC_UMAX, + GFX9_GLOBAL_ATOMIC_AND, + GFX9_GLOBAL_ATOMIC_OR, + GFX9_GLOBAL_ATOMIC_XOR, + GFX9_GLOBAL_ATOMIC_INC, + GFX9_GLOBAL_ATOMIC_DEC =3D 76, + GFX9_GLOBAL_ATOMIC_SWAP_X2 =3D 96, + GFX9_GLOBAL_ATOMIC_CMPSWAP_X2, + GFX9_GLOBAL_ATOMIC_ADD_X2, + GFX9_GLOBAL_ATOMIC_SUB_X2, + GFX9_GLOBAL_ATOMIC_SMIN_X2, + GFX9_GLOBAL_ATOMIC_UMIN_X2, + GFX9_GLOBAL_ATOMIC_SMAX_X2, + GFX9_GLOBAL_ATOMIC_UMAX_X2, + GFX9_GLOBAL_ATOMIC_AND_X2, + GFX9_GLOBAL_ATOMIC_OR_X2, + GFX9_GLOBAL_ATOMIC_XOR_X2, + GFX9_GLOBAL_ATOMIC_INC_X2, + GFX9_GLOBAL_ATOMIC_DEC_X2, +}; + +/* Export Format */ +struct amdgcn_gfx9_exp { +}; + +union amdgcn_gfx9_insn { + struct amdgcn_gfx9_sop2 sop2; + struct amdgcn_gfx9_sopk sopk; + struct amdgcn_gfx9_sop1 sop1; + struct amdgcn_gfx9_sopc sopc; + struct amdgcn_gfx9_sopp sopp; + struct amdgcn_gfx9_smem smem; + struct amdgcn_gfx9_vop2 vop2; + struct amdgcn_gfx9_vop1 vop1; + struct amdgcn_gfx9_vopc vopc; + struct amdgcn_gfx9_vop3a vop3a; + struct amdgcn_gfx9_vop3b vop3b; + struct amdgcn_gfx9_vop3p vop3p; + struct amdgcn_gfx9_sdwa sdwa; + struct amdgcn_gfx9_sdwab sdwab; + struct amdgcn_gfx9_dpp16 dpp16; + struct amdgcn_gfx9_dpp8 dpp8; + struct amdgcn_gfx9_vintrp vintrp; + struct amdgcn_gfx9_ds ds; + struct amdgcn_gfx9_mtbuf mtbuf; + struct amdgcn_gfx9_mubuf mubuf; + struct amdgcn_gfx9_mimg mimg; + struct amdgcn_gfx9_flat flat; + struct amdgcn_gfx9_exp exp; +}; + +/* Cast macro - convert u32 *buf position to GFX9 insn union pointer. */ +#define I9(buf, n) ((union amdgcn_gfx9_insn *)&(buf)[(n)]) + +inline u32 gfx9_get_param_base(struct amdgcn_param32 param) +{ + return gfx9_param_base[param.type]; +} + +inline u32 emit_gfx9_s_load_dwordx2(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int offset) +{ + /* s_load_dwordx2 , , + * sdata: register, load to. + * sbase: sgpr-pair, load from. + * offset: offset of kernarg_address in the hsa_kernel_dispatch_packet. + * + * sdata =3D *(sbase + offset); + * param =3D (__global struct _knod_bpf_param *)pkt.kernarg_address; + */ + + insn->smem.sbase =3D KNOD_AMDGPU_REG_PAIR(src.v); + insn->smem.sdata =3D dst.v; + insn->smem.dummy1 =3D 0; + insn->smem.soe =3D 0; + insn->smem.nv =3D 0; + insn->smem.glc =3D 0; + insn->smem.imm =3D 1; + insn->smem.op =3D GFX9_S_LOAD_DWORDX2; + insn->smem.encoding =3D GFX9_SMEM_ENCODING; + insn->smem.offset =3D offset; + insn->smem.dummy2 =3D 0; + insn->smem.soffset =3D 0; + + return 8; +} + +inline u32 emit_gfx9_v_bfe_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_BFE_I32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_bfe_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* v_bfe_u32 v0, v0, 8 + * : destination vgpr. + * : initialized to workitem in first bitfields. + * 8: bitfield size to extract from. + * + * extract workitem ID from + */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_BFE_U32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_bfi_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_BFI_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_lshl_add_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* v_lshl_add_u32 , , , + * + * =3D ( << ) + ; + */ + + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_LSHL_ADD_U32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_lshl_or_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* v_lshl_or_b32 , , , + * + * D.u =3D (S0.u << S1.u[4:0]) | S2.u + */ + + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_LSHL_OR_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_mad_u64_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 vdst, + struct amdgcn_param32 sdst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param64 src2) +{ + /* v_mad_u64_u32 , , , , + * : destination vgpr. + * : destination sgpr. + * : source vgpr. + * : source vgpr. + * : source vgpr. + * + * {vcc_out, D.u64} =3D S0.u32 * S1.u32 + S2.u64. + * ctx =3D ¶m->sub[idx].ctx; + */ + + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.hi.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3b.vdst =3D vdst.lo.v; + insn->vop3b.sdst =3D sdst.v; + insn->vop3b.clmp =3D 0; + insn->vop3b.op =3D GFX9_V_MAD_U64_U32; + insn->vop3b.encoding =3D GFX9_VOP3B_ENCODING; + insn->vop3b.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3b.src2 =3D gfx9_get_param_base(src2.lo) + src2.lo.v; + insn->vop3b.omod =3D 0; + insn->vop3b.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3b.src0 =3D gfx9_get_param_base(src0); + insn->vop3b.literal =3D src0.v; + return 12; + } + insn->vop3b.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_s_mov_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->sop1.ssrc0 =3D gfx9_get_param_base(src); + insn->sop1.literal =3D src.v; + } else { + insn->sop1.ssrc0 =3D gfx9_get_param_base(src) + src.v; + } + insn->sop1.op =3D GFX9_S_MOV_B32; + insn->sop1.sdst =3D gfx9_get_param_base(dst) + dst.v; + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) + return 8; + return 4; +} + +/* + * v_readfirstlane_b32 sdst, vsrc + * Copies VGPR[vsrc] from the first active lane to SGPR[sdst]. + * VOP1 encoding, dst =3D SGPR (not VGPR). + */ +inline u32 emit_gfx9_v_readfirstlane_b32(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 vsrc) +{ + insn->vop1.encoding =3D GFX9_VOP1_ENCODING; + insn->vop1.vdst =3D sdst; + insn->vop1.op =3D GFX9_V_READFIRSTLANE_B32; + insn->vop1.src0 =3D GFX9_SRC_VGPR_BASE + vsrc; + return 4; +} + +inline u32 emit_gfx9_v_mov_b32_e32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + if (dst.type !=3D AMDGCN_PARAM_TYPE_VGPR) + WARN_ON_ONCE(1); + insn->vop1.encoding =3D GFX9_VOP1_ENCODING; + insn->vop1.vdst =3D dst.v; + insn->vop1.op =3D GFX9_V_MOV_B32; + if (src.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop1.src0 =3D GFX9_VOP1_SRC_LITERAL_CONST; + insn->vop1.literal =3D src.v; + + return 8; + } + insn->vop1.src0 =3D gfx9_get_param_base(src) + src.v; + + return 4; +} + +inline u32 emit_gfx9_v_add_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_ADD_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_add_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_ADD_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_sub_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_SUB_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_addc_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_ADDC_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + WARN_ON_ONCE(1); + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_xor_b32_e32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_XOR_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_or_b32_e32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_OR_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_cndmask_b32_e32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_CNDMASK_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_and_b32_e32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_AND_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_sub_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_SUB_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_subrev_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* D.u =3D S1.u - S0.u; */ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_SUBREV_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_subbrev_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* D.u =3D S1.u - S0.u; */ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_SUBBREV_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_subb_co_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) + +{ + /* dst =3D src0 - src1 */ + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_SUBB_CO_U32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_mul_lo_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_MUL_LO_U32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +/* v_mbcnt_lo_u32_b32 vdst, src0, vsrc1 + * vdst =3D popcount(src0 & ((1 << lane_id[4:0]) - 1)) + vsrc1 + */ +inline u32 emit_gfx9_v_mbcnt_lo_u32_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_MBCNT_LO_U32_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +/* v_mbcnt_hi_u32_b32 vdst, src0, vsrc1 + * vdst =3D popcount(src0 & ((1 << lane_id[5]) - 1)) + vsrc1 + */ +inline u32 emit_gfx9_v_mbcnt_hi_u32_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_MBCNT_HI_U32_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +inline u32 emit_gfx9_v_mul_hi_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_MUL_HI_U32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_lshlrev_b64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 << s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_LSHLREV_B64; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1.lo) + src1.lo.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx9_v_lshlrev_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 << s0 */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_LSHLREV_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_lshrrev_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 << s0 */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_LSHRREV_B32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +inline u32 emit_gfx9_v_lshrrev_b64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 >> s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_LSHRREV_B64; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1.lo) + src1.lo.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx9_v_ashrrev_i64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + /* dst =3D s1 >> s0 */ + insn->vop3a.vdst =3D dst.lo.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_ASHRREV_I64; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1.lo) + src1.lo.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.lo.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo); + insn->vop3a.literal =3D src0.lo.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0.lo) + src0.lo.v; + + return 8; +} + +inline u32 emit_gfx9_v_ashrrev_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + /* dst =3D s1 >> s0 */ + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + /* + * VOP2 ops encoded in VOP3A use opcode 0x100 + vop2_op (V_ASHRREV_I32 + * is a VOP2 instruction). Without the +0x100 the op field decodes as a + * different VOP3A instruction, so the arithmetic shift (used for 64-bit + * sign-extension, e.g. bpf_xdp_adjust_head's delta) produced garbage. + */ + insn->vop3a.op =3D GFX9_V_ASHRREV_I32 + 0x100; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + /* 128: src2=3D0 reads s0 on gfx9 */ + insn->vop3a.src2 =3D GFX9_SRC_INTEGER_0; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_alignbit_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* D.u =3D ({S0,S1} >> S2.u[4:0]) & 0xffffffff. */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_ALIGNBIT_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop3a.src0 =3D gfx9_get_param_base(src0); + insn->vop3a.literal =3D src0.v; + return 12; + } + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 8; +} + +inline u32 emit_gfx9_v_perm_b32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + /* + * v_perm_b32: D.u[31:24] =3D src[sel[14:12]], + * D.u[23:16] =3D src[sel[10:8]], ... + * For bswap32: sel =3D 0x00010203 reverses bytes. + */ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + /* VOP3 does not support literal constants on GFX9 */ + WARN_ON(src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src1.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST || + src2.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + insn->vop3a.vdst =3D dst.v; + insn->vop3a.abs =3D 0; + insn->vop3a.op_sel =3D 0; + insn->vop3a.clmp =3D 0; + insn->vop3a.op =3D GFX9_V_PERM_B32; + insn->vop3a.encoding =3D GFX9_VOP3A_ENCODING; + insn->vop3a.src0 =3D gfx9_get_param_base(src0) + src0.v; + insn->vop3a.src1 =3D gfx9_get_param_base(src1) + src1.v; + insn->vop3a.src2 =3D gfx9_get_param_base(src2) + src2.v; + insn->vop3a.omod =3D 0; + insn->vop3a.neg =3D 0; + + return 8; +} + +inline u32 emit_gfx9_v_cmp_eq_u64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 =3D=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_EQ_U64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_eq_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* + * VOPC src0 supports SGPR/inline/VGPR but NOT literal + * (no 8-byte path) + */ + WARN_ON(dst.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST); + WARN_ON(src.type !=3D AMDGCN_PARAM_TYPE_VGPR); + /* VCC =3D S0 =3D=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_EQ_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_gt_u64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 > S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_GT_U64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_gt_i64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 > S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_GT_I64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_ge_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 >=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_GE_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_gt_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 > S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_GT_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_lt_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 < S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_LT_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_le_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 <=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_LE_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_gt_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 > S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_GT_I32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_ge_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 >=3D S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_GE_I32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_lt_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 < S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_LT_I32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_le_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* VCC =3D S0 <=3D S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMP_LE_I32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmpx_lt_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + /* EXEC &=3D (S0 < S1) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst) + dst.v; + insn->vopc.vsrc1 =3D src.v; + insn->vopc.op =3D GFX9_V_CMPX_LT_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_ge_u64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 >=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_GE_U64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_ge_i64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 >=3D S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_GE_I64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_lt_u64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 < S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_LT_U64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_lt_i64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 < S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_LT_I64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_le_u64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 <=3D S1 */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_LE_U64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_v_cmp_le_i64(union amdgcn_gfx9_insn *insn, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + /* VCC =3D S0 <=3D S1 (signed) */ + insn->vopc.src0 =3D gfx9_get_param_base(dst.lo) + dst.lo.v; + insn->vopc.vsrc1 =3D src.lo.v; + insn->vopc.op =3D GFX9_V_CMP_LE_I64; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_buffer_load_ubyte(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_ubyte , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.op =3D GFX9_BUFFER_LOAD_UBYTE; + insn->mubuf.dummy2 =3D 0; + insn->mubuf.encoding =3D GFX9_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy3 =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX9_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx9_buffer_load_ushort(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_ushort , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.op =3D GFX9_BUFFER_LOAD_USHORT; + insn->mubuf.dummy2 =3D 0; + insn->mubuf.encoding =3D GFX9_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy3 =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX9_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx9_buffer_load_dword(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dword , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.op =3D GFX9_BUFFER_LOAD_DWORD; + insn->mubuf.dummy2 =3D 0; + insn->mubuf.encoding =3D GFX9_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy3 =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX9_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx9_buffer_load_dwordx2(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dwordx2 , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.op =3D GFX9_BUFFER_LOAD_DWORDX2; + insn->mubuf.dummy2 =3D 0; + insn->mubuf.encoding =3D GFX9_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy3 =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX9_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx9_buffer_load_dwordx4(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* buffer_load_dwordx4 , , s[0:3], 0 offen offset: */ + insn->mubuf.offset =3D off; + insn->mubuf.offen =3D 1; + insn->mubuf.idxen =3D 0; + insn->mubuf.glc =3D 0; + insn->mubuf.dummy1 =3D 0; + insn->mubuf.lds =3D 0; + insn->mubuf.op =3D GFX9_BUFFER_LOAD_DWORDX4; + insn->mubuf.dummy2 =3D 0; + insn->mubuf.encoding =3D GFX9_MUBUF_ENCODING; + insn->mubuf.vaddr =3D src.v; + insn->mubuf.vdata =3D dst.v; + insn->mubuf.srsrc =3D 0; /* s[0:3] */ + insn->mubuf.dummy3 =3D 0; + insn->mubuf.tfe =3D 0; + insn->mubuf.soffset =3D GFX9_MUBUF_SOFFSET_INTEGER_0; + + return 8; +} + +inline u32 emit_gfx9_global_load_ubyte(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_ubyte , , off offset: */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_LOAD_UBYTE; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; /* ignored? */ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_load_ushort(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_ushort , , off offset: */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_LOAD_USHORT; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; /* ignored? */ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_load_dword(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dword , , off offset: */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_LOAD_DWORD; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; /* ignored? */ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_load_dwordx2(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dwordx2 , , off offset: */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_LOAD_DWORDX2; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; /* ignored? */ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_load_dwordx4(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dwordx4 , , off offset: */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_LOAD_DWORDX4; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; /* ignored? */ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_load_dwordx4_glc(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, + short off) +{ + /* global_load_dwordx4 , , off offset: glc slc + * GLC=3D1: bypass L1 (TCP). SLC=3D1: bypass L2 (TCC). + * Both needed for VRAM written by external DMA (NIC via PCIe) + * since L2 is not invalidated on external writes. + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 1; + insn->flat.slc =3D 1; + insn->flat.op =3D GFX9_GLOBAL_LOAD_DWORDX4; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D src.v; + insn->flat.data =3D 0; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D dst.v; + + return 8; +} + +inline u32 emit_gfx9_global_store_byte(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_byte , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_STORE_BYTE; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx9_global_store_short(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_short , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_STORE_SHORT; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx9_global_store_dword(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dword , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_STORE_DWORD; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +/* global_atomic_add vdst, addr, data, off (GLC=3D1: return old value) + * old_val =3D *(u32 *)(addr + off); *(u32 *)(addr + off) +=3D data; vdst = =3D old_val + */ +/* GFX9 global atomic: opcode only differs, all else identical */ +#define DEFINE_GFX9_GLOBAL_ATOMIC(name, opcode) \ +inline u32 emit_gfx9_global_atomic_##name(union amdgcn_gfx9_insn *insn,\ + struct amdgcn_param32 vdst, \ + struct amdgcn_param32 addr, \ + struct amdgcn_param32 data, \ + int off, int glc) \ +{ \ + insn->flat.offset =3D off; \ + insn->flat.lds =3D 0; \ + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; \ + insn->flat.glc =3D glc; \ + insn->flat.slc =3D 0; \ + insn->flat.op =3D (opcode); \ + insn->flat.dummy =3D 0; \ + insn->flat.encoding =3D GFX9_FLAT_ENCODING; \ + insn->flat.addr =3D addr.v; \ + insn->flat.data =3D data.v; \ + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; \ + insn->flat.nv =3D 0; \ + insn->flat.vdst =3D vdst.v; \ + return 8; \ +} + +DEFINE_GFX9_GLOBAL_ATOMIC(add, GFX9_GLOBAL_ATOMIC_ADD) +DEFINE_GFX9_GLOBAL_ATOMIC(and, GFX9_GLOBAL_ATOMIC_AND) +DEFINE_GFX9_GLOBAL_ATOMIC(or, GFX9_GLOBAL_ATOMIC_OR) +DEFINE_GFX9_GLOBAL_ATOMIC(xor, GFX9_GLOBAL_ATOMIC_XOR) +DEFINE_GFX9_GLOBAL_ATOMIC(swap, GFX9_GLOBAL_ATOMIC_SWAP) +DEFINE_GFX9_GLOBAL_ATOMIC(cmpswap, GFX9_GLOBAL_ATOMIC_CMPSWAP) +DEFINE_GFX9_GLOBAL_ATOMIC(add_x2, GFX9_GLOBAL_ATOMIC_ADD_X2) +DEFINE_GFX9_GLOBAL_ATOMIC(and_x2, GFX9_GLOBAL_ATOMIC_AND_X2) +DEFINE_GFX9_GLOBAL_ATOMIC(or_x2, GFX9_GLOBAL_ATOMIC_OR_X2) +DEFINE_GFX9_GLOBAL_ATOMIC(xor_x2, GFX9_GLOBAL_ATOMIC_XOR_X2) +DEFINE_GFX9_GLOBAL_ATOMIC(swap_x2, GFX9_GLOBAL_ATOMIC_SWAP_X2) +DEFINE_GFX9_GLOBAL_ATOMIC(cmpswap_x2, GFX9_GLOBAL_ATOMIC_CMPSWAP_X2) + +inline u32 emit_gfx9_global_store_dwordx2(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dwordx2 , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_STORE_DWORDX2; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx9_global_store_dwordx4(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src, int off) +{ + /* global_store_dwordx4 , , off offset: + * *(char *)(dst + off) =3D src; + */ + insn->flat.offset =3D off; + insn->flat.lds =3D 0; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.glc =3D 0; + insn->flat.slc =3D 0; + insn->flat.op =3D GFX9_GLOBAL_STORE_DWORDX4; + insn->flat.dummy =3D 0; + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.addr =3D dst.v; + insn->flat.data =3D src.v; + insn->flat.saddr =3D GFX9_FLAT_SADDR_DISABLE; + insn->flat.nv =3D 0; + insn->flat.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx9_ds_read2_b64(union amdgcn_gfx9_insn *insn, + int dst, int src, short off0, short off1) +{ + /* + * ds_read2_b64 v[+3:], v off + * offset0: offset1: + */ + + insn->ds.offset0 =3D off0; + insn->ds.offset1 =3D off1; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_READ2_B64; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D src; + insn->ds.data0 =3D 0; + insn->ds.data1 =3D 0; + insn->ds.vdst =3D dst; + + return 8; +} + +inline u32 emit_gfx9_ds_read_b64(union amdgcn_gfx9_insn *insn, + int dst, int src, short off) +{ + /* ds_read_b64 v[+1:], v offset: */ + + insn->ds.offset0 =3D off & 0xff; + insn->ds.offset1 =3D off >> 8; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_READ_B64; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D src; + insn->ds.data0 =3D 0; + insn->ds.data1 =3D 0; + insn->ds.vdst =3D dst; + + return 8; +} + +inline u32 emit_gfx9_ds_read_b32(union amdgcn_gfx9_insn *insn, + int dst, int src, short off) +{ + /* ds_read_b32 v, v offset: */ + + insn->ds.offset0 =3D off & 0xff; + insn->ds.offset1 =3D off >> 8; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_READ_B32; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D src; + insn->ds.data0 =3D 0; + insn->ds.data1 =3D 0; + insn->ds.vdst =3D dst; + + return 8; +} + +inline u32 emit_gfx9_ds_read_u16(union amdgcn_gfx9_insn *insn, + int dst, int src, short off) +{ + /* ds_read_u16 v, v offset: */ + + insn->ds.offset0 =3D off & 0xff; + insn->ds.offset1 =3D off >> 8; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_READ_U16; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D src; + insn->ds.data0 =3D 0; + insn->ds.data1 =3D 0; + insn->ds.vdst =3D dst; + + return 8; +} + +inline u32 emit_gfx9_ds_read_u8(union amdgcn_gfx9_insn *insn, + int dst, int src, short off) +{ + /* ds_read_u8 v, v offset: */ + + insn->ds.offset0 =3D off & 0xff; + insn->ds.offset1 =3D off >> 8; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_READ_U8; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D src; + insn->ds.data0 =3D 0; + insn->ds.data1 =3D 0; + insn->ds.vdst =3D dst; + + return 8; +} + +inline u32 emit_gfx9_ds_write2_b64(union amdgcn_gfx9_insn *insn, + int dst, int src0, int src1, + short off0, short off1) +{ + /* + * ds_write2_b64 v[+3:], v off + * offset0: offset1: + */ + + insn->ds.offset0 =3D off0; + insn->ds.offset1 =3D off1; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.op =3D GFX9_DS_WRITE2_B64; + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.addr =3D dst; + insn->ds.data0 =3D src0; + insn->ds.data1 =3D src1; + insn->ds.vdst =3D 0; + + return 8; +} + +inline u32 emit_gfx9_s_branch(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_BRANCH; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_cbranch_vccz(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_VCCZ; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_cbranch_vccnz(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_VCCNZ; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_branch_fixup(union amdgcn_gfx9_insn *insn, + short off) +{ + WARN_ON(insn->sopp.op !=3D GFX9_S_BRANCH && + insn->sopp.op !=3D GFX9_S_CBRANCH_SCC0 && + insn->sopp.op !=3D GFX9_S_CBRANCH_VCCZ && + insn->sopp.op !=3D GFX9_S_CBRANCH_VCCNZ && + insn->sopp.op !=3D GFX9_S_CBRANCH_EXECZ && + insn->sopp.op !=3D GFX9_S_CBRANCH_EXECNZ); + insn->sopp.simm16 =3D off; + + return 4; +} + +inline u32 emit_gfx9_s_waitcnt_lgkmcnt(union amdgcn_gfx9_insn *insn) +{ + struct amdgcn_gfx9_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D -1; + vmcnt.vmcnt2 =3D -1; + vmcnt.expcnt =3D -1; + vmcnt.dummy1 =3D 0; + vmcnt.dummy2 =3D 0; + vmcnt.lgkmcnt =3D 0; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX9_S_WAITCNT; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_waitcnt_vmcnt(union amdgcn_gfx9_insn *insn) +{ + struct amdgcn_gfx9_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D 0; + vmcnt.vmcnt2 =3D 0; + vmcnt.expcnt =3D -1; + vmcnt.dummy1 =3D 0; + vmcnt.dummy2 =3D 0; + vmcnt.lgkmcnt =3D -1; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX9_S_WAITCNT; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_waitcnt_vmcnt_lgkmcnt(union amdgcn_gfx9_insn *insn) +{ + struct amdgcn_gfx9_sopp_vmcnt vmcnt; + u16 simm16; + /* s_waitcnt lgkmcnt (0) + * wait for memory + */ + vmcnt.vmcnt1 =3D 0; + vmcnt.vmcnt2 =3D 0; + vmcnt.expcnt =3D -1; + vmcnt.dummy1 =3D 0; + vmcnt.dummy2 =3D 0; + vmcnt.lgkmcnt =3D 0; + memcpy(&simm16, &vmcnt, sizeof(u16)); + insn->sopp.simm16 =3D simm16; + insn->sopp.op =3D GFX9_S_WAITCNT; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_nop(union amdgcn_gfx9_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX9_S_NOP; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_endpgm(union amdgcn_gfx9_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX9_S_ENDPGM; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +inline u32 emit_gfx9_s_icache_inv(union amdgcn_gfx9_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX9_S_ICACHE_INV; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +/* Structurized CFG instructions. + * These use raw SGPR indices for 64-bit pair operations involving + * EXEC (126) and VCC (106) special registers. + */ + +/* s_and_saveexec_b64 s[sdst:sdst+1], s[ssrc:ssrc+1] + * sdst =3D EXEC; EXEC &=3D ssrc; SCC =3D (EXEC !=3D 0) + */ +inline u32 emit_gfx9_s_and_saveexec_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX9_S_AND_SAVEEXEC_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + + return 4; +} + +/* s_bcnt1_i32_b64 sdst, s[ssrc:ssrc+1] + * sdst =3D popcount(ssrc). SCC =3D (sdst !=3D 0) + */ +inline u32 emit_gfx9_s_bcnt1_i32_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX9_S_BCNT1_I32_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + + return 4; +} + +/* s_mov_b64 s[sdst:sdst+1], s[ssrc:ssrc+1] (or special src like 0) */ +inline u32 emit_gfx9_s_mov_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX9_S_MOV_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + + return 4; +} + +/* s_and_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] */ +inline u32 emit_gfx9_s_and_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_AND_B64; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_or_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] */ +inline u32 emit_gfx9_s_or_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_OR_B64; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_andn2_b64 s[sdst:sdst+1], s[ssrc0:ssrc0+1], s[ssrc1:ssrc1+1] + * sdst =3D ssrc0 & ~ssrc1 + */ +inline u32 emit_gfx9_s_andn2_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_ANDN2_B64; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_and_b32 s[sdst], s[ssrc0], s[ssrc1] */ +inline u32 emit_gfx9_s_and_b32(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_AND_B32; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_or_b32 s[sdst], s[ssrc0], s[ssrc1] */ +inline u32 emit_gfx9_s_or_b32(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_OR_B32; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_andn2_b32 s[sdst], s[ssrc0], s[ssrc1] + * sdst =3D ssrc0 & ~ssrc1 + */ +inline u32 emit_gfx9_s_andn2_b32(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_ANDN2_B32; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_cbranch_execz off - branch if EXEC =3D=3D 0 */ +inline u32 emit_gfx9_s_cbranch_execz(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_EXECZ; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +/* s_cbranch_execnz off - branch if EXEC !=3D 0 */ +inline u32 emit_gfx9_s_cbranch_execnz(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_EXECNZ; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +/* s_sub_u32 sdst, ssrc0, ssrc1 - sdst =3D ssrc0 - ssrc1; SCC =3D borrow */ +inline u32 emit_gfx9_s_sub_u32(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc0, u8 ssrc1) +{ + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; + insn->sop2.sdst =3D sdst; + insn->sop2.op =3D GFX9_S_SUB_U32; + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + + return 4; +} + +/* s_cbranch_scc0 off - branch if SCC =3D=3D 0 (no borrow) */ +inline u32 emit_gfx9_s_cbranch_scc0(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_SCC0; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + + return 4; +} + +static inline void __emit_gfx9_sop2(union amdgcn_gfx9_insn *insn, + int op, int sdst, int ssrc0, int ssrc1) +{ + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; + insn->sop2.op =3D op; + insn->sop2.sdst =3D sdst; + insn->sop2.ssrc0 =3D ssrc0; + insn->sop2.ssrc1 =3D ssrc1; +} + +static inline void __emit_gfx9_sop1(union amdgcn_gfx9_insn *insn, + int op, int sdst, int ssrc0) +{ + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + insn->sop1.op =3D op; + insn->sop1.sdst =3D sdst; + insn->sop1.ssrc0 =3D ssrc0; +} + +static inline void __emit_gfx9_sopp(union amdgcn_gfx9_insn *insn, + int op, u16 simm16) +{ + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + insn->sopp.op =3D op; + insn->sopp.simm16 =3D simm16; +} + +static inline void __emit_gfx9_sopc(union amdgcn_gfx9_insn *insn, + int op, int ssrc0, int ssrc1) +{ + insn->sopc.encoding =3D GFX9_SOPC_ENCODING; + insn->sopc.op =3D op; + insn->sopc.ssrc0 =3D ssrc0; + insn->sopc.ssrc1 =3D ssrc1; +} + +static inline void __emit_gfx9_vop1(union amdgcn_gfx9_insn *insn, + int op, int vdst, int src0) +{ + insn->vop1.encoding =3D GFX9_VOP1_ENCODING; + insn->vop1.op =3D op; + insn->vop1.vdst =3D vdst; + insn->vop1.src0 =3D src0; +} + +static inline void __emit_gfx9_vop2(union amdgcn_gfx9_insn *insn, + int op, int vdst, int vsrc1, int src0) +{ + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + insn->vop2.op =3D op; + insn->vop2.vdst =3D vdst; + insn->vop2.vsrc1 =3D vsrc1; + insn->vop2.src0 =3D src0; +} + +static inline void __emit_gfx9_vopc(union amdgcn_gfx9_insn *insn, + int op, int vsrc1, int src0) +{ + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + insn->vopc.op =3D op; + insn->vopc.vsrc1 =3D vsrc1; + insn->vopc.src0 =3D src0; +} + +static inline void __emit_gfx9_smem(union amdgcn_gfx9_insn *insn, + int op, int sdata, int sbase_pair, + u32 offset) +{ + insn->smem.encoding =3D GFX9_SMEM_ENCODING; + insn->smem.op =3D op; + insn->smem.sdata =3D sdata; + insn->smem.sbase =3D sbase_pair; + insn->smem.imm =3D 1; + insn->smem.offset =3D offset; + insn->smem.soffset =3D GFX9_SRC_NULL; +} + +static inline void __emit_gfx9_ds(union amdgcn_gfx9_insn *insn, + int op, int addr, int data0, int vdst, + int off0, int off1) +{ + insn->ds.encoding =3D GFX9_DS_ENCODING; + insn->ds.op =3D op; + insn->ds.gds =3D GFX9_DS_LDS; + insn->ds.addr =3D addr; + insn->ds.data0 =3D data0; + insn->ds.vdst =3D vdst; + insn->ds.offset0 =3D off0; + insn->ds.offset1 =3D off1; +} + +static inline void __emit_gfx9_global(union amdgcn_gfx9_insn *insn, + int op, int vdst, int vaddr, + int vdata, int saddr, int offset) +{ + insn->flat.encoding =3D GFX9_FLAT_ENCODING; + insn->flat.seg =3D GFX9_FLAT_SEG_GLOBAL; + insn->flat.op =3D op; + insn->flat.vdst =3D vdst; + insn->flat.addr =3D vaddr; + insn->flat.data =3D vdata; + insn->flat.saddr =3D saddr; + insn->flat.offset =3D offset; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX9 Param-Aware Emit Functions + * + * Used by shader-emitters (aesgcm_shader.h, ipsec_fused_gfx9.h, ...) that + * construct param32 operands via P_S/P_V/P_I/P_L helpers. Paired with + * the GFX10 equivalents in knod_gfx10_insn.h. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static inline int __p2e9(struct amdgcn_param32 p) +{ + if (p.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) + return gfx9_get_param_base(p); + return gfx9_get_param_base(p) + p.v; +} + +/* --- SOP2 family (param32 version) --- */ + +#define DEFINE_GFX9_SOP2_P(name, opcode) \ +inline u32 emit_gfx9_##name(union amdgcn_gfx9_insn *insn, \ + struct amdgcn_param32 dst, \ + struct amdgcn_param32 src0, \ + struct amdgcn_param32 src1) \ +{ \ + insn->sop2.sdst =3D __p2e9(dst); \ + insn->sop2.ssrc0 =3D __p2e9(src0); \ + insn->sop2.ssrc1 =3D __p2e9(src1); \ + insn->sop2.op =3D opcode; \ + insn->sop2.encoding =3D GFX9_SOP2_ENCODING; \ + if (knod_param_is_literal(src0)) { \ + insn->sop2.literal =3D src0.v; \ + return 8; \ + } \ + if (knod_param_is_literal(src1)) { \ + insn->sop2.literal =3D src1.v; \ + return 8; \ + } \ + return 4; \ +} + +DEFINE_GFX9_SOP2_P(s_add_u32, GFX9_S_ADD_U32) +DEFINE_GFX9_SOP2_P(s_sub_u32_p, GFX9_S_SUB_U32) +DEFINE_GFX9_SOP2_P(s_addc_u32, GFX9_S_ADDC_U32) +DEFINE_GFX9_SOP2_P(s_subb_u32, GFX9_S_SUBB_U32) +DEFINE_GFX9_SOP2_P(s_and_b32_p, GFX9_S_AND_B32) +DEFINE_GFX9_SOP2_P(s_lshl_b32, GFX9_S_LSHL_B32) +DEFINE_GFX9_SOP2_P(s_lshr_b32, GFX9_S_LSHR_B32) +DEFINE_GFX9_SOP2_P(s_mul_i32, GFX9_S_MUL_I32) +DEFINE_GFX9_SOP2_P(s_xor_b32, GFX9_S_XOR_B32) + +#undef DEFINE_GFX9_SOP2_P + +/* --- SOPC family (param32 version) --- */ + +#define DEFINE_GFX9_SOPC_P(name, opcode) \ +inline u32 emit_gfx9_##name(union amdgcn_gfx9_insn *insn, \ + struct amdgcn_param32 src0, \ + struct amdgcn_param32 src1) \ +{ \ + insn->sopc.ssrc0 =3D __p2e9(src0); \ + insn->sopc.ssrc1 =3D __p2e9(src1); \ + insn->sopc.op =3D opcode; \ + insn->sopc.encoding =3D GFX9_SOPC_ENCODING; \ + if (knod_param_is_literal(src0)) { \ + insn->sopc.literal =3D src0.v; \ + return 8; \ + } \ + if (knod_param_is_literal(src1)) { \ + insn->sopc.literal =3D src1.v; \ + return 8; \ + } \ + return 4; \ +} + +DEFINE_GFX9_SOPC_P(s_cmp_eq_u32, GFX9_S_CMP_EQ_U32) +DEFINE_GFX9_SOPC_P(s_cmp_lg_u32, GFX9_S_CMP_LG_U32) +DEFINE_GFX9_SOPC_P(s_cmp_ge_u32, GFX9_S_CMP_GE_U32) +DEFINE_GFX9_SOPC_P(s_cmp_lt_u32, GFX9_S_CMP_LT_U32) + +#undef DEFINE_GFX9_SOPC_P + +/* --- SOPP family --- */ + +inline u32 emit_gfx9_s_barrier(union amdgcn_gfx9_insn *insn) +{ + insn->sopp.simm16 =3D 0; + insn->sopp.op =3D GFX9_S_BARRIER; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + return 4; +} + +/* waitcnt with arbitrary vm/lgkm */ +#define GFX9_WAITCNT(vm, lgkm) (((lgkm) << 8) | (7 << 4) | (vm)) + +inline u32 emit_gfx9_s_waitcnt(union amdgcn_gfx9_insn *insn, + int vm, int lgkm) +{ + insn->sopp.simm16 =3D GFX9_WAITCNT(vm, lgkm); + insn->sopp.op =3D GFX9_S_WAITCNT; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + return 4; +} + +inline u32 emit_gfx9_s_cbranch_scc1(union amdgcn_gfx9_insn *insn, + short off) +{ + insn->sopp.simm16 =3D off; + insn->sopp.op =3D GFX9_S_CBRANCH_SCC1; + insn->sopp.encoding =3D GFX9_SOPP_ENCODING; + return 4; +} + +/* --- SOP1 family --- */ + +inline u32 emit_gfx9_s_not_b64(union amdgcn_gfx9_insn *insn, + u8 sdst, u8 ssrc) +{ + insn->sop1.ssrc0 =3D ssrc; + insn->sop1.op =3D GFX9_S_NOT_B64; + insn->sop1.sdst =3D sdst; + insn->sop1.encoding =3D GFX9_SOP1_ENCODING; + return 4; +} + +/* --- VOPC (v_cmp_ne_u32 param variant) --- */ + +inline u32 emit_gfx9_v_cmp_ne_u32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + insn->vopc.vsrc1 =3D src1.v; + insn->vopc.op =3D GFX9_V_CMP_NE_U32; + insn->vopc.encoding =3D GFX9_VOPC_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vopc.src0 =3D gfx9_get_param_base(src0); + insn->vopc.literal =3D src0.v; + return 8; + } + insn->vopc.src0 =3D gfx9_get_param_base(src0) + src0.v; + return 4; +} + +/* --- SMEM family (param32 version) --- */ + +#define DEFINE_GFX9_SMEM_P(name, opcode) \ +inline u32 emit_gfx9_##name(union amdgcn_gfx9_insn *insn, \ + struct amdgcn_param32 dst, \ + struct amdgcn_param32 src, int offset) \ +{ \ + insn->smem.sdata =3D dst.v; \ + insn->smem.sbase =3D src.v / 2; \ + insn->smem.op =3D opcode; \ + insn->smem.imm =3D 1; \ + insn->smem.offset =3D offset; \ + insn->smem.encoding =3D GFX9_SMEM_ENCODING; \ + return 8; \ +} + +DEFINE_GFX9_SMEM_P(s_load_dword, GFX9_S_LOAD_DWORD) +DEFINE_GFX9_SMEM_P(s_load_dwordx4, GFX9_S_LOAD_DWORDX4) +DEFINE_GFX9_SMEM_P(s_load_dwordx8, GFX9_S_LOAD_DWORDX8) + +#undef DEFINE_GFX9_SMEM_P + +/* s_dcache_inv - no operands, just opcode + encoding */ +inline u32 emit_gfx9_s_dcache_inv(union amdgcn_gfx9_insn *insn) +{ + insn->smem.sdata =3D 0; + insn->smem.sbase =3D 0; + insn->smem.op =3D GFX9_S_DCACHE_INV; + insn->smem.imm =3D 0; + insn->smem.offset =3D 0; + insn->smem.encoding =3D GFX9_SMEM_ENCODING; + return 8; +} + +/* --- SOPK: s_getreg_b32 --- */ + +/* + * HWREG encoding for s_getreg_b32 simm16: + * bits[5:0] =3D hwreg_id + * bits[10:6] =3D offset (bit position) + * bits[15:11] =3D size - 1 (in bits) + */ +#define GFX9_HWREG(id, off, sz) (((sz) - 1) << 11 | (off) << 6 | (id)) +#define GFX9_HW_REG_LDS_ALLOC 6 + +inline u32 emit_gfx9_s_getreg_b32(union amdgcn_gfx9_insn *insn, + int sdst, u16 hwreg) +{ + insn->sopk.encoding =3D GFX9_SOPK_ENCODING; + insn->sopk.op =3D GFX9_S_GETREG_B32; + insn->sopk.sdst =3D sdst; + insn->sopk.simm16 =3D hwreg; + return 4; +} + +/* --- DS family --- */ + +inline u32 emit_gfx9_gds_write_b32(union amdgcn_gfx9_insn *insn, + int addr, int data0) +{ + __emit_gfx9_ds(insn, GFX9_DS_WRITE_B32, addr, data0, 0, 0, 0); + insn->ds.gds =3D GFX9_DS_GDS; + return 8; +} + +inline u32 emit_gfx9_gds_read_b32(union amdgcn_gfx9_insn *insn, + int vdst, int addr, int offset) +{ + __emit_gfx9_ds(insn, GFX9_DS_READ_B32, addr, 0, vdst, offset, 0); + insn->ds.gds =3D GFX9_DS_GDS; + return 8; +} + +inline u32 emit_gfx9_ds_write_b32(union amdgcn_gfx9_insn *insn, + int addr, int data0) +{ + __emit_gfx9_ds(insn, GFX9_DS_WRITE_B32, addr, data0, 0, 0, 0); + return 8; +} + +inline u32 emit_gfx9_ds_write_b32_off(union amdgcn_gfx9_insn *insn, + int addr, int data0, int offset) +{ + __emit_gfx9_ds(insn, GFX9_DS_WRITE_B32, addr, data0, 0, offset, 0); + return 8; +} + +inline u32 emit_gfx9_ds_read_b32_off(union amdgcn_gfx9_insn *insn, + int vdst, int addr, int offset) +{ + __emit_gfx9_ds(insn, GFX9_DS_READ_B32, addr, 0, vdst, offset, 0); + return 8; +} + +/* ds_write_b128 v, v[:+3] - write 128 bits to LDS */ +inline u32 emit_gfx9_ds_write_b128(union amdgcn_gfx9_insn *insn, + int addr, int data0) +{ + __emit_gfx9_ds(insn, GFX9_DS_WRITE_B128, addr, data0, 0, 0, 0); + return 8; +} + +/* ds_read_b128 v[:+3], v - read 128 bits from LDS */ +inline u32 emit_gfx9_ds_read_b128(union amdgcn_gfx9_insn *insn, + int vdst, int addr) +{ + __emit_gfx9_ds(insn, GFX9_DS_READ_B128, addr, 0, vdst, 0, 0); + return 8; +} + +/* v_max_i32 vdst, src0, vsrc1 - VOP2 */ +inline u32 emit_gfx9_v_max_i32(union amdgcn_gfx9_insn *insn, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + WARN_ON(dst.type !=3D AMDGCN_PARAM_TYPE_VGPR); + WARN_ON(src1.type !=3D AMDGCN_PARAM_TYPE_VGPR); + + insn->vop2.vsrc1 =3D src1.v; + insn->vop2.vdst =3D dst.v; + insn->vop2.op =3D GFX9_V_MAX_I32; + insn->vop2.encoding =3D GFX9_VOP2_ENCODING; + if (src0.type =3D=3D AMDGCN_PARAM_TYPE_LITERAL_CONST) { + insn->vop2.src0 =3D gfx9_get_param_base(src0); + insn->vop2.literal =3D src0.v; + return 8; + } + insn->vop2.src0 =3D gfx9_get_param_base(src0) + src0.v; + + return 4; +} + +#endif --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C63953B3BF5 for ; Sun, 19 Jul 2026 18:01:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484158; cv=none; b=Q+ibPfKbsDCy0FO8oNKXLNOtFzBgmv8LBMoUqLLIUTTdR1ClUs/yd0ecvf1lWMbJ5S/LL/pZi/dj7NBfkBaia328BIM/bZNzImkVQvrevL1Vp4e8pGkKFESJMX4Y4qADSL+UpGVq3Kvs/KNvYhubQ1mDOX8Yo9MmlRLQeoJtW8g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484158; c=relaxed/simple; bh=3dv/ffJumcA7GMH0JRcM4W9HUXMxzguBrELKWNvRCeE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YpE90tj1ApCLiB2n1cHKdEvfebqAsWOJtpIXMA0Cd3HFEJVdaDsbUoT4w2t/PivOgoUwXBT6bjNNaipAShFYaZAfTRgwYfkNoP9UQx9JW3KkbUfnDVwHVnYBBPB6V0jold5QmUhUW+yP0EMGWr5/Smlfm0LhE3FN8y1s+bGHPDo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f2Z1haqy; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f2Z1haqy" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2ce7d2adef4so87728215ad.3 for ; Sun, 19 Jul 2026 11:01:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484091; x=1785088891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=azzV2bYeVsLKU7C2phxsGcESC9yqicjrMHqaNHRc6gc=; b=f2Z1haqy7ya7uCoe5IRLGsUVinghy7bzLydr0Te+d4jseB/eQIeFWHBWlOFt9SuciJ TGTp9lvxkgXfKbMcEMYBzfp1IAiW81LYXhpAitoOld/Gi0+RRFdEBoFl+w10aRpSKw0M +09mwgGY5Mr4XsLmhxAj8CxsCHhlYuUphGfcB9dxj0+/7JPG2MXhFC2OLK+6CKU3ehOI dkgq6IRuoAIBhkTlaVJiRecJ7HypG/EBRykBHeAGJJqv6WuXQGqgTH5PUZRfdN/e3Omn pJryOcNHKYeVqvFMYXzfCbP6qDj1i+scGmZDPkbS8/jrm9uUJcCpwJmA/8Dcz1+HIf9z Q2IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484091; x=1785088891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=azzV2bYeVsLKU7C2phxsGcESC9yqicjrMHqaNHRc6gc=; b=AfrHrzMlkLl+vUi5shDjeTCgN7DOoqoVV7HUAPHSnREWmYJblGrtTWfwt+Js5M19SV 4fJWZ8V6YU6BHWAr4LfPGBlr3N41N/U5JQY7rhchRh6DDYE4C21nJ+Y9ouaXbU9vrxBC PXBxD3PnkjXOXUs58A1y0+NeKivrIv9JTmm9G/eTpljNSlXLc/k0vpDdzaP56xkL/sX8 9BkzJJSeZb3MZIbuYTkFUmWU2regS0n79c5VFWsaevNGHJblMDC/tvgDz/ND9cOKJmee yuzSktaEmzsGI8dl+2xZL3YFu/GijMKy2EvlAQOoGeTfz9NKuOgIE89hyLYBLDs4j02h f+7A== X-Forwarded-Encrypted: i=1; AHgh+RpWaKlkHIMxDSSGq70QuM8YM1axKnL421Vj9cDicz4n9lEXiAVQbPlATpe9Rim2aaGbGdMzqlG/5ArWNNQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyneGzK2EKjox+KxaQOF5GhpjPR/oq17qJwdvuCvZlkd5rOGrGV fuoHYQ5D99WuQuh9bjAoSwMuaQ0eLhJ0r81fwaFTvyeYUzh5r8NN6HD0 X-Gm-Gg: AfdE7cnn6SigUgiaFz4mBleUxxVF4R9k+ra2M3L8mp+GLhNGHDKYt0HLUhHGM7igWrT X7sFm0lY634SKua4qQatjl/tU8pSgILv+UdI7nrv0iR/HeYHRxU9HiPEMAga+9dGrkgRFTCHowQ X04Deti7kXch9MPZob2zovZsjpspeHWTz5EuZM7sN6FviXdYVYYPOiypSMDg1DjRGJx8FOJv3Cj KvcOsEhYkka1OTicIeTAxYPJNxTl7TFs9jXg0orXROqqaWgnnt/Mcl5jcp35YZ/m3FfN3U0aJOu +CMgbPNA73LNbDTqjChBmbPf1Sdnbrp1q9n+P2ocdCZ6BiYYsI+Xn3xRL8W5On3w9CtAYFG6qSc 8zTmPpfErflfRSWLcAiWIMx6s3ef3k0E0hqngQqZWr2hkPNgDGWDKZjDyAhqb6ZDY2w2s8s/shV td X-Received: by 2002:a17:903:3807:b0:2ce:9439:59a5 with SMTP id d9443c01a7336-2cf349a15c6mr111239485ad.36.1784484089456; Sun, 19 Jul 2026 11:01:29 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.01.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:01:27 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 09/13] drm/amdkfd: add BPF-to-GPU JIT offload Date: Sun, 19 Jul 2026 17:58:53 +0000 Message-ID: <20260719175857.4071636-10-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the knod BPF feature: an XDP program attached in offload mode is JIT-compiled from eBPF to an AMD GCN shader and dispatched on the GPU against packets DMA'd into GPU memory, keeping the PASS/DROP/TX verdict path off the host CPU. Built as a separate module (knod_bpf). Signed-off-by: Taehee Yoo (cherry picked from commit 132d51819ffced59b5890e14bd39cab2e9c12eb4) --- drivers/gpu/drm/amd/amdkfd/Kconfig | 11 + drivers/gpu/drm/amd/amdkfd/Makefile | 2 + drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c | 11554 +++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h | 597 + 4 files changed, 12164 insertions(+) create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkf= d/Kconfig index d93f1af749ff..708aa5fc051b 100644 --- a/drivers/gpu/drm/amd/amdkfd/Kconfig +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig @@ -49,3 +49,14 @@ config HSA_AMD_KNOD =20 Say N to drop the KNOD core along with the BPF and IPsec offloads layered on top of it. If unsure, say Y. + +config HSA_AMD_KNOD_BPF + tristate "KNOD BPF" + depends on HSA_AMD_KNOD + help + GPU-accelerated XDP/BPF packet processing via KNOD. BPF programs + attached in XDP mode are JIT-compiled to AMD GCN shaders and run + on the GPU against packets DMA'd directly into GPU memory, keeping + the verdict path (PASS/DROP/TX) off the host CPU. + + If unsure, say N. diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdk= fd/Makefile index 1834faa54863..4df3850e1466 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -75,3 +75,5 @@ endif ifneq ($(CONFIG_HSA_AMD_KNOD),) AMDKFD_FILES +=3D $(AMDKFD_PATH)/kfd_knod.o endif + +obj-$(CONFIG_HSA_AMD_KNOD_BPF) +=3D $(AMDKFD_PATH)/knod/knod_bpf.o diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c b/drivers/gpu/drm/a= md/amdkfd/knod/knod_bpf.c new file mode 100644 index 000000000000..f4f48e1b9f1c --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c @@ -0,0 +1,11554 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "kfd_priv.h" +#include "kfd_hsa.h" +#include "knod_bpf.h" +#include "kfd_migrate.h" +#include "kfd_events.h" +#include "kfd_device_queue_manager.h" +#include +#include +#include +#include + +/*+--------+---------+-------+------+--+-----+------+------+--------+ + *| v0-v21 | v22-v59 |v60-v61| v62 |63|64-65|66-67 |68-69 | v70-127| + *+--------+---------+-------+------+--+-----+------+------+--------+ + *|BPF REGS|TMP REGS |CTX REG| WIDX |R |DATA |D_END |PGBASE|PKTCACHE| + *+--------+---------+-------+------+--+-----+------+------+--------+ + *+-----------------+ + *| v128-v255 | + *+-----------------+ + *| BPF STACK(512B) | + *+-----------------+ + */ + +/* Temp register map + *+-------------+-------------+---------------+---------------+ + *|TREG0 - TREG2|TREG3 - TREG9|TREG10 - TREG16|TREG17 - TREG18| + *+-------------+-------------+---------------+---------------+ + *| General Use | Key cache A | Key in MAP | JHASH Temp Reg| + *+-------------+-------------+---------------+---------------+ + * Available Key cache size is 56. + * So, key size of map can't be exceed 56B. + */ + +#define KNOD_AMDGPU_VREG0_LO 0 /* v0 */ +#define KNOD_AMDGPU_VREG0_HI 1 +#define KNOD_AMDGPU_VREG1_LO 2 +#define KNOD_AMDGPU_VREG1_HI 3 +#define KNOD_AMDGPU_VREG2_LO 4 +#define KNOD_AMDGPU_VREG2_HI 5 +#define KNOD_AMDGPU_VREG3_LO 6 +#define KNOD_AMDGPU_VREG3_HI 7 +#define KNOD_AMDGPU_VREG4_LO 8 +#define KNOD_AMDGPU_VREG4_HI 9 +#define KNOD_AMDGPU_VREG5_LO 10 +#define KNOD_AMDGPU_VREG5_HI 11 +#define KNOD_AMDGPU_VREG6_LO 12 +#define KNOD_AMDGPU_VREG6_HI 13 +#define KNOD_AMDGPU_VREG7_LO 14 +#define KNOD_AMDGPU_VREG7_HI 15 +#define KNOD_AMDGPU_VREG8_LO 16 +#define KNOD_AMDGPU_VREG8_HI 17 +#define KNOD_AMDGPU_VREG9_LO 18 +#define KNOD_AMDGPU_VREG9_HI 19 +#define KNOD_AMDGPU_FRAME_POINTER_VREG_LO 20 /* v20 */ +#define KNOD_AMDGPU_FRAME_POINTER_VREG_HI 21 /* v20 */ + +#define KNOD_AMDGPU_TMP_VREG0_LO 22 +#define KNOD_AMDGPU_TMP_VREG0_HI 23 +#define KNOD_AMDGPU_TMP_VREG1_LO 24 +#define KNOD_AMDGPU_TMP_VREG1_HI 25 +#define KNOD_AMDGPU_TMP_VREG2_LO 26 +#define KNOD_AMDGPU_TMP_VREG2_HI 27 +#define KNOD_AMDGPU_TMP_VREG3_LO 28 +#define KNOD_AMDGPU_TMP_VREG3_HI 29 +#define KNOD_AMDGPU_TMP_VREG4_LO 30 +#define KNOD_AMDGPU_TMP_VREG4_HI 31 +#define KNOD_AMDGPU_TMP_VREG5_LO 32 +#define KNOD_AMDGPU_TMP_VREG5_HI 33 +#define KNOD_AMDGPU_TMP_VREG6_LO 34 +#define KNOD_AMDGPU_TMP_VREG6_HI 35 +#define KNOD_AMDGPU_TMP_VREG7_LO 36 +#define KNOD_AMDGPU_TMP_VREG7_HI 37 +#define KNOD_AMDGPU_TMP_VREG8_LO 38 +#define KNOD_AMDGPU_TMP_VREG8_HI 39 +#define KNOD_AMDGPU_TMP_VREG9_LO 40 +#define KNOD_AMDGPU_TMP_VREG9_HI 41 +#define KNOD_AMDGPU_TMP_VREG10_LO 42 +#define KNOD_AMDGPU_TMP_VREG10_HI 43 +#define KNOD_AMDGPU_TMP_VREG11_LO 44 +#define KNOD_AMDGPU_TMP_VREG11_HI 45 +#define KNOD_AMDGPU_TMP_VREG12_LO 46 +#define KNOD_AMDGPU_TMP_VREG12_HI 47 +#define KNOD_AMDGPU_TMP_VREG13_LO 48 +#define KNOD_AMDGPU_TMP_VREG13_HI 49 +#define KNOD_AMDGPU_TMP_VREG14_LO 50 +#define KNOD_AMDGPU_TMP_VREG14_HI 51 +#define KNOD_AMDGPU_TMP_VREG15_LO 52 +#define KNOD_AMDGPU_TMP_VREG15_HI 53 +#define KNOD_AMDGPU_TMP_VREG16_LO 54 +#define KNOD_AMDGPU_TMP_VREG16_HI 55 +#define KNOD_AMDGPU_TMP_VREG17_LO 56 +#define KNOD_AMDGPU_TMP_VREG17_HI 57 +#define KNOD_AMDGPU_TMP_VREG18_LO 58 +#define KNOD_AMDGPU_TMP_VREG18_HI 59 +#define KNOD_AMDGPU_TMP_VREG_MAX KNOD_AMDGPU_TMP_VREG18_HI +#define KNOD_AMDGPU_CTX_VREG_LO 60 +#define KNOD_AMDGPU_CTX_VREG_HI 61 +#define KNOD_AMDGPU_IDX_VREG 62 +#define KNOD_AMDGPU_RESERVED 63 +/* + * After prologue step 4, IDX_VREG is no longer needed. + * v62:v63 are repurposed to hold slot_addr (spsc_bd GTT address) + * through BPF execution and into the epilogue. + * + * BACKLOG_IDX_VREG (v58) saves the backlog index from IDX_VREG + * before step 6 overwrites it. Used in epilogue for XDP_PASS. + */ +#define KNOD_AMDGPU_BACKLOG_IDX_VREG KNOD_AMDGPU_TMP_VREG18_LO /* v58 */ +#define KNOD_AMDGPU_SLOT_VREG_LO KNOD_AMDGPU_IDX_VREG /* v62 */ +#define KNOD_AMDGPU_SLOT_VREG_HI KNOD_AMDGPU_RESERVED /* v63 */ +/* + * DATA/DATA_END VGPRs: hold packet gaddr and end address. + * Set in prologue, read by BPF ctx->data / ctx->data_end accesses. + * Replaces GTT round-trip (prologue store -> BPF load). + */ +#define KNOD_AMDGPU_DATA_VREG_LO 64 +#define KNOD_AMDGPU_DATA_VREG_HI 65 +#define KNOD_AMDGPU_DATA_END_VREG_LO 66 +#define KNOD_AMDGPU_DATA_END_VREG_HI 67 +#define KNOD_AMDGPU_PAGE_BASE_VREG_LO 68 +#define KNOD_AMDGPU_PAGE_BASE_VREG_HI 69 +#define KNOD_AMDGPU_PKT_CACHE_VREG0 70 +#define KNOD_AMDGPU_PKT_CACHE_VREG_MAX 127 /* 0 ~ 127 vgprs are available = */ +#define KNOD_AMDGPU_STACK_VREG0 128 +#define KNOD_AMDGPU_STACK_VREG_MAX 255 /* 128 ~ 255 vgprs are available */ + +#define KNOD_BPF_PROG_BUF_SIZE 32768 + +/* Index for r64. + * r64[TREG64_0] + */ +#define TREG64_0 0 +#define TREG64_1 1 +#define TREG64_2 2 +#define TREG64_3 3 +#define KEY_IN_PKT_64 TREG64_3 +#define TREG64_4 4 +#define TREG64_5 5 +#define TREG64_6 6 +#define TREG64_7 7 +#define TREG64_8 8 +#define TREG64_9 9 +#define TREG64_10 10 +#define KEY_IN_MAP_64 TREG64_10 +#define TREG64_11 11 +#define TREG64_12 12 +#define TREG64_13 13 +#define TREG64_14 14 +#define TREG64_15 15 +#define TREG64_16 16 +#define TREG64_17 17 +#define TREG64_18 18 + +#define MAX_MAP_KEY_SIZE 56 + +/* Index for r32. + * r32[TREG32_0_LO] + */ +#define TREG32_0_LO 0 +#define TREG32_0_HI 1 +#define TREG32_1_LO 2 +#define TREG32_1_HI 3 +#define TREG32_2_LO 4 +#define TREG32_2_HI 5 +#define TREG32_3_LO 6 +#define KEY_IN_PKT_32 TREG32_3_LO +#define TREG32_3_HI 7 +#define TREG32_4_LO 8 +#define TREG32_4_HI 9 +#define TREG32_5_LO 10 +#define TREG32_5_HI 11 +#define TREG32_6_LO 12 +#define TREG32_6_HI 13 +#define TREG32_7_LO 14 +#define TREG32_7_HI 15 +#define TREG32_8_LO 16 +#define TREG32_8_HI 17 +#define TREG32_9_LO 18 +#define TREG32_9_HI 19 +#define TREG32_10_LO 20 +#define KEY_IN_MAP_32 TREG32_10_LO +#define TREG32_10_HI 21 +#define TREG32_11_LO 22 +#define TREG32_11_HI 23 +#define TREG32_12_LO 24 +#define TREG32_12_HI 25 +#define TREG32_13_LO 26 +#define TREG32_13_HI 27 +#define TREG32_14_LO 28 +#define TREG32_14_HI 29 +#define TREG32_15_LO 30 +#define TREG32_15_HI 31 +#define TREG32_16_LO 32 +#define TREG32_16_HI 33 +#define TREG32_17_LO 34 +#define TREG32_17_HI 35 +#define TREG32_18_LO 36 +#define TREG32_18_HI 37 +#define TREG32_MAX TREG32_18_HI + +/*+--------+--------------------------------------+---+---+ + *| s[0:3] |s[4:5] s[6:7] s[8:9] s[10:11] s[12:13]|s14|s15| + *+--------+--------------------------------------+---+---+ + *| PSB | USER SGPRs (disp/queue/karg/id/flat) |WGX|QID| + *+--------+--------------------------------------+---+---+ + *+----------------+------+---+---+------+-------+-------------------+ + *| s[16:27] |s28:29|s30|s31|s32:33|s34:35 | s[36:105] | + *+----------------+------+---+---+------+-------+-------------------+ + *|TMP_SREG 0-5 |PARAM |FP | - | GFX9 | DONE | EXEC_SAVE PAIRS | + *|(6 x 64-bit) |SREG | | |BROKE!| MASK | (max 35, GFX10) | + *+----------------+------+---+---+------+-------+-------------------+ + * Implicit: VCC =3D s[106:107] EXEC =3D s[126:127] + * + * user_sgpr_count=3D14, same on GFX9 and GFX10. + * enable_sgpr_private_segment_size is disabled so that workgroup_id_y + * lands at s15 and TMP_SREG0_LO stays at s16 (keeps 64-bit SGPR pair + * alignment; avoids shifting the entire TMP/PARAM/FRAME layout). + */ +#define KNOD_AMDGPU_PSB_SREG 0 /* s[0:3] private_segment_buffer */ +#define KNOD_AMDGPU_DISPATCH_PTR_SREG 4 /* s[4:5] dispatch_ptr */ +#define KNOD_AMDGPU_ARG_SREG 4 /* alias for dispatch_ptr */ +#define KNOD_AMDGPU_QUEUE_PTR_SREG 6 /* s[6:7] queue_ptr */ +#define KNOD_AMDGPU_KERNARG_PTR_SREG 8 /* s[8:9] kernarg_segment_ptr */ +#define KNOD_AMDGPU_DISPATCH_ID_SREG 10 /* s[10:11] dispatch_id */ +#define KNOD_AMDGPU_FLAT_SCR_INIT_SREG 12 /* s[12:13] flat_scratch_init */ +#define KNOD_AMDGPU_WORKGROUP_ID_X_SREG 14 /* s14 workgroup_id_x */ +#define KNOD_AMDGPU_WORKGROUP_ID_Y_SREG 15 /* s15 workgroup_id_y =3D queue= _id */ +#define KNOD_AMDGPU_TMP_SREG0_LO 16 +#define KNOD_AMDGPU_TMP_SREG0_HI 17 +#define KNOD_AMDGPU_TMP_SREG1_LO 18 +#define KNOD_AMDGPU_TMP_SREG1_HI 19 +#define KNOD_AMDGPU_TMP_SREG2_LO 20 +#define KNOD_AMDGPU_TMP_SREG2_HI 21 +#define KNOD_AMDGPU_TMP_SREG3_LO 22 +#define KNOD_AMDGPU_TMP_SREG3_HI 23 +#define KNOD_AMDGPU_TMP_SREG4_LO 24 +#define KNOD_AMDGPU_TMP_SREG4_HI 25 +#define KNOD_AMDGPU_TMP_SREG5_LO 26 +#define KNOD_AMDGPU_TMP_SREG5_HI 27 +#define KNOD_AMDGPU_PARAM_SREG_LO 28 /* s28 */ +#define KNOD_AMDGPU_PARAM_SREG_HI 29 /* s29 */ +#define KNOD_AMDGPU_FRAME_POINTER_SREG 30 /* s30 */ + +/* Structurized CFG: EXEC mask save/restore SGPRs. + * done_mask tracks lanes that have reached BPF_EXIT. + * exec_save pairs store EXEC at branch points for restore at merge points. + * GFX9: s[0:101] addressable (102 SGPRs), GFX10: s[0:105] (106 SGPRs). + * NOTE: s[32:33] is corrupted by GFX9 hardware - do NOT use on GFX9. + * GFX10 uses s[32:33] for done_mask and starts exec_save at s[34]. + */ +/* Common SGPR special register indices (same on GFX9 and GFX10) */ +#define AMDGCN_SREG_VCC_LO 106 +#define AMDGCN_SREG_EXEC_LO 126 +#define AMDGCN_SREG_INTEGER_0 128 +#define AMDGCN_SREG_INTEGER_1 129 + +/* s[34:35] - must not overlap TMP_SREGs */ +#define KNOD_AMDGPU_DONE_MASK_SREG 34 +#define KNOD_AMDGPU_EXEC_SAVE_SREG_BASE 36 /* s[36:37], s[38:39], ... */ +#define KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX9 100 +#define KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX10 104 +/* exec_save can fill up to each ISA's top usable SGPR pair. GFX9 lays the + * initial in-bounds EXEC snapshot immediately after the pairs a program + * actually uses, so small programs keep the old 64-SGPR occupancy window. + * GFX10 keeps the original high fixed snapshot pair. + */ +#define KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX9 99 +#define KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX10 103 +#define KNOD_AMDGPU_MAX_EXEC_SAVE_PAIRS_GFX9 \ + ((KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX9 - KNOD_AMDGPU_EXEC_SAVE_SREG_BASE += 1) / 2) +#define KNOD_AMDGPU_MAX_EXEC_SAVE_PAIRS_GFX10 \ + ((KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX10 - KNOD_AMDGPU_EXEC_SAVE_SREG_BASE = + 1) / 2) + +static u8 knod_bpf_gfx9_sgpr_granule(unsigned int sgprs_used) +{ + if (sgprs_used <=3D 16) + return 0; + + return 2 * (DIV_ROUND_UP(sgprs_used, 16) - 1); +} + +unsigned int knod_bpf_workgroups =3D KNOD_BPF_WORKGROUPS_DEFAULT; +MODULE_PARM_DESC(workgroups, "Workgroup size, multiple of 64, Min(64) Defa= ult/Max(256)"); +module_param_named(workgroups, knod_bpf_workgroups, int, 0600); + +unsigned int knod_bpf_expire =3D KNOD_BPF_EXPIRE_DEFAULT; +MODULE_PARM_DESC(queue_expire, "Queue expire time(ms), Min(1), Default(10)= , Max(1000)"); +module_param_named(queue_expire, knod_bpf_expire, int, 0600); + +unsigned int knod_bpf_pkt_cache; +MODULE_PARM_DESC(packet_cache, "Use packet cache, 0=3DOff(Default), 1=3DOn= "); +module_param_named(packet_cache, knod_bpf_pkt_cache, int, 0600); + +unsigned int knod_bpf_wave32; +MODULE_PARM_DESC(wave32, "Use wave32 0=3DOff(Default), 1=3DOn"); +module_param_named(wave32, knod_bpf_wave32, int, 0600); + +#define KNOD_EA(extack, msg) NL_SET_ERR_MSG_MOD((extack), msg) + +DEFINE_STATIC_KEY_FALSE(knod_stats_key); + +static const u32 bl_bounds[KNOD_BL_BUCKETS - 1] =3D { + 16, 64, 256, 1024, 4096, 8192, 16384 +}; + +static const char * const lat_labels[] =3D { + "< 1us", "1-2us", "2-4us", "4-8us", "8-16us", + "16-32us", "32-64us", "64-128us", "128-256us", ">=3D 256us", +}; + +static const char * const bl_labels[] =3D { + "1-16", "17-64", "65-256", "257-1K", + "1K-4K", "4K-8K", "8K-16K", ">=3D 16K", +}; + +static LIST_HEAD(priv_list); +struct amdgcn_param64 r64[20], sr64[6], p64[4], bpf_reg64[11]; +struct amdgcn_param32 r32[40]; /* last two is CTX */ +struct amdgcn_param32 stack[128]; +struct amdgcn_param32 pkt_cache[64]; + +struct amdgcn_label { + struct knod_insn_meta *meta; + int insn_idx; +}; + +struct amdgcn_branch_fixup { + struct amdgcn_label *target_label; + struct knod_insn_meta *meta; + int insn_idx; +}; + +struct knod_accel_xdp_ops accel_xdp_ops; + +static int knod_prog_prepare_insns(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog); +static int knod_bpf_worker(void *arg); +static void knod_bpf_drain_worker(struct knod_bpf_priv *priv); +static void knod_prog_free(struct knod_prog *knod_prog); +static void knod_emit_pass_addr_store(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta); +static void knod_setup_bpf_prog(struct bpf_prog *prog); + +static void knod_bpf_gpu_mem_fence(struct knod_bpf_priv *priv) +{ + if (!priv) + return; + + /* drain the WC store buffer before the GPU reads the map */ + wmb(); +} + +static unsigned int knod_bpf_active_rxq_count(struct net_device *netdev) +{ + unsigned int nr_rxq; + + if (!netdev) + return 0; + + nr_rxq =3D READ_ONCE(netdev->real_num_rx_queues); + if (!nr_rxq) + nr_rxq =3D netdev->num_rx_queues; + + return min_t(unsigned int, nr_rxq, KNOD_SPSC_MAX); +} + +static void knod_bpf_fill_dispatch(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw, + struct knod_dispatch_params *p) +{ + struct knod_bpf_param *param =3D sqw->param->kaddr; + + p->workgroup_size_x =3D knod_bpf_workgroups; + p->grid_size_x =3D priv->batch_size; + p->grid_size_y =3D param->nr_queues; + p->private_segment_size =3D 8192; + p->group_segment_size =3D 8192; + p->kernel_object =3D + (u64)priv->knod->kernels[READ_ONCE(priv->active_idx)]->gaddr; + p->kernarg_address =3D sqw->param->gaddr; +} + +static void debug_kernel_descriptor(struct kernel_descriptor *kernel_code) +{ + knod_jit_dbg(" kernel_code->group_segment_fixed_size =3D %d\n", + kernel_code->group_segment_fixed_size); + knod_jit_dbg(" kernel_code->private_segment_fixed_size =3D %d\n", + kernel_code->private_segment_fixed_size); + knod_jit_dbg(" kernel_code->kernarg_size =3D %d\n", + kernel_code->kernarg_size); + knod_jit_dbg(" kernel_code->kernel_code_entry_byte_offset =3D %lld\n", + kernel_code->kernel_code_entry_byte_offset); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc3.accum_offset =3D %d\n", + kernel_code->compute_pgm_rsrc3.accum_offset); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc3.reserved0 =3D %d\n", + kernel_code->compute_pgm_rsrc3.reserved0); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc3.tg_split =3D %d\n", + kernel_code->compute_pgm_rsrc3.tg_split); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc3.reserved1 =3D %d\n", + kernel_code->compute_pgm_rsrc3.reserved1); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.granulated_workitem_vgpr_co= unt =3D %d\n", + kernel_code->compute_pgm_rsrc1.granulated_workitem_vgpr_count); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.granulated_wavefront_sgpr_c= ount =3D %d\n", + kernel_code->compute_pgm_rsrc1.granulated_wavefront_sgpr_count); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.priority =3D %d\n", + kernel_code->compute_pgm_rsrc1.priority); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.float_round_mode_32 =3D %d\= n", + kernel_code->compute_pgm_rsrc1.float_round_mode_32); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.float_round_mode_16_64 =3D = %d\n", + kernel_code->compute_pgm_rsrc1.float_round_mode_16_64); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.float_denorm_mode_32 =3D %d= \n", + kernel_code->compute_pgm_rsrc1.float_denorm_mode_32); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.float_denorm_mode_16_64 =3D= %d\n", + kernel_code->compute_pgm_rsrc1.float_denorm_mode_16_64); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.priv =3D %d\n", + kernel_code->compute_pgm_rsrc1.priv); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.enable_dx10_clamp =3D %d\n", + kernel_code->compute_pgm_rsrc1.enable_dx10_clamp); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.debug_mode =3D %d\n", + kernel_code->compute_pgm_rsrc1.debug_mode); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.enable_ieee_mode =3D %d\n", + kernel_code->compute_pgm_rsrc1.enable_ieee_mode); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.bulky =3D %d\n", + kernel_code->compute_pgm_rsrc1.bulky); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.cdbg_user =3D %d\n", + kernel_code->compute_pgm_rsrc1.cdbg_user); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.fp16_ovfl =3D %d\n", + kernel_code->compute_pgm_rsrc1.fp16_ovfl); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.reserved0 =3D %d\n", + kernel_code->compute_pgm_rsrc1.reserved0); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.wgp_mode =3D %d\n", + kernel_code->compute_pgm_rsrc1.wgp_mode); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.mem_ordered =3D %d\n", + kernel_code->compute_pgm_rsrc1.mem_ordered); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc1.fwd_progress =3D %d\n", + kernel_code->compute_pgm_rsrc1.fwd_progress); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_private_segment =3D = %d\n", + kernel_code->compute_pgm_rsrc2.enable_private_segment); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.user_sgpr_count =3D %d\n", + kernel_code->compute_pgm_rsrc2.user_sgpr_count); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_trap_handler =3D %d\= n", + kernel_code->compute_pgm_rsrc2.enable_trap_handler); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_x = =3D %d\n", + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_x); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_y = =3D %d\n", + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_y); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_z = =3D %d\n", + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_z); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_info = =3D %d\n", + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_info); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_vgpr_workitem_id =3D= %d\n", + kernel_code->compute_pgm_rsrc2.enable_vgpr_workitem_id); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_address_wa= tch =3D %d\n", + kernel_code->compute_pgm_rsrc2.enable_exception_address_watch); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_memory =3D= %d\n", + kernel_code->compute_pgm_rsrc2.enable_exception_memory); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.granulated_lds_size =3D %d\= n", + kernel_code->compute_pgm_rsrc2.granulated_lds_size); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_f= p_invalid_operation =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_invalid_operation); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_fp_denorma= l_source =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_fp_denormal_source); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_f= p_division_by_zero =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_division_by_zero); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_f= p_overflow =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_overflow); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_f= p_underflow =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_underflow); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_f= p_inexact =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_inexact); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.enable_exception_int_divide= _by_zero =3D %d\n", + kernel_code->compute_pgm_rsrc2 + .enable_exception_int_divide_by_zero); + knod_jit_dbg(" kernel_code->compute_pgm_rsrc2.reserved0 =3D %d\n", + kernel_code->compute_pgm_rsrc2.reserved0); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_private_segment_b= uffer =3D %d\n", + kernel_code->code_properties + .enable_sgpr_private_segment_buffer); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_dispatch_ptr =3D = %d\n", + kernel_code->code_properties.enable_sgpr_dispatch_ptr); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_queue_ptr =3D %d\= n", + kernel_code->code_properties.enable_sgpr_queue_ptr); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_kernarg_segment_p= tr =3D %d\n", + kernel_code->code_properties.enable_sgpr_kernarg_segment_ptr); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_dispatch_id =3D %= d\n", + kernel_code->code_properties.enable_sgpr_dispatch_id); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_flat_scratch_init= =3D %d\n", + kernel_code->code_properties.enable_sgpr_flat_scratch_init); + knod_jit_dbg(" kernel_code->code_properties.enable_sgpr_private_segment_s= ize =3D %d\n", + kernel_code->code_properties.enable_sgpr_private_segment_size); + knod_jit_dbg(" kernel_code->code_properties.reserved0 =3D %d\n", + kernel_code->code_properties.reserved0); + knod_jit_dbg(" kernel_code->code_properties.enable_wavefront_size32 =3D %= d\n", + kernel_code->code_properties.enable_wavefront_size32); + knod_jit_dbg(" kernel_code->code_properties.uses_dynamic_stack =3D %d\n", + kernel_code->code_properties.uses_dynamic_stack); + knod_jit_dbg(" kernel_code->code_properties.reserved1 =3D %d\n", + kernel_code->code_properties.reserved1); +} + +static void kfd_kernel_gfx9_init(struct knod *knod) +{ + struct kernel_descriptor *kernel_code =3D knod->kernels[0]->kaddr; + + kernel_code->group_segment_fixed_size =3D 0; + kernel_code->private_segment_fixed_size =3D 8192; + kernel_code->kernarg_size =3D 64; + kernel_code->kernel_code_entry_byte_offset =3D 1024; + + /* GFX10+ or GFX90A+ */ + kernel_code->compute_pgm_rsrc3.accum_offset =3D 0; + kernel_code->compute_pgm_rsrc3.reserved0 =3D 0; + kernel_code->compute_pgm_rsrc3.tg_split =3D 0; + kernel_code->compute_pgm_rsrc3.reserved1 =3D 0; + + kernel_code->compute_pgm_rsrc1.granulated_workitem_vgpr_count =3D + (256 / 4) - 1; + /* + * Start with the small GFX9 window. BPF install updates each slot + * descriptor when a program needs a larger exec_save/initial_exec + * range. + */ + kernel_code->compute_pgm_rsrc1.granulated_wavefront_sgpr_count =3D + knod_bpf_gfx9_sgpr_granule(52); + kernel_code->compute_pgm_rsrc1.priority =3D 0; + kernel_code->compute_pgm_rsrc1.float_round_mode_32 =3D 0; + kernel_code->compute_pgm_rsrc1.float_round_mode_16_64 =3D 0; + kernel_code->compute_pgm_rsrc1.float_denorm_mode_32 =3D 3; + kernel_code->compute_pgm_rsrc1.float_denorm_mode_16_64 =3D 3; + kernel_code->compute_pgm_rsrc1.priv =3D 0; + kernel_code->compute_pgm_rsrc1.enable_dx10_clamp =3D 1; + kernel_code->compute_pgm_rsrc1.debug_mode =3D 0; + kernel_code->compute_pgm_rsrc1.enable_ieee_mode =3D 1; + kernel_code->compute_pgm_rsrc1.bulky =3D 0; + kernel_code->compute_pgm_rsrc1.cdbg_user =3D 0; + kernel_code->compute_pgm_rsrc1.fp16_ovfl =3D 0; + kernel_code->compute_pgm_rsrc1.reserved0 =3D 0; + kernel_code->compute_pgm_rsrc1.wgp_mode =3D 0; + kernel_code->compute_pgm_rsrc1.mem_ordered =3D 0; + kernel_code->compute_pgm_rsrc1.fwd_progress =3D 0; + + kernel_code->compute_pgm_rsrc2.enable_private_segment =3D 0; + kernel_code->compute_pgm_rsrc2.user_sgpr_count =3D 14; /* 4+2+2+2+2+2 */ + kernel_code->compute_pgm_rsrc2.enable_trap_handler =3D 0; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_x =3D 1; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_y =3D 1; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_z =3D 0; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_info =3D 0; + kernel_code->compute_pgm_rsrc2.enable_vgpr_workitem_id =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_address_watch =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_memory =3D 0; + kernel_code->compute_pgm_rsrc2.granulated_lds_size =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_invalid_operation =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_fp_denormal_source =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_division_by_zero =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_overflow =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_underflow =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_fp_inexact =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_int_divide_by_zero =3D 0; + kernel_code->compute_pgm_rsrc2.reserved0 =3D 0; + + /* + * User SGPR layout - loaded in fixed order, disabled entries are + * skipped (not reserved). The resulting SGPR map depends on which + * flags are enabled: + * + * enable_sgpr_private_segment_buffer -> 4 SGPRs (s[0:3]) + * enable_sgpr_dispatch_ptr -> 2 SGPRs (s[4:5]) + * enable_sgpr_queue_ptr -> 2 SGPRs + * enable_sgpr_kernarg_segment_ptr -> 2 SGPRs + * enable_sgpr_dispatch_id -> 2 SGPRs + * enable_sgpr_flat_scratch_init -> 2 SGPRs + * enable_sgpr_private_segment_size -> 1 SGPR + * + * System SGPRs (WorkgroupId etc.) follow immediately after the + * last user SGPR. user_sgpr_count must match the total above. + */ + /* 4 SGPRs */ + kernel_code->code_properties.enable_sgpr_private_segment_buffer =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_dispatch_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_queue_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_kernarg_segment_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_dispatch_id =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_flat_scratch_init =3D 1; + /* disabled -> s14/s15 free for workgroup_id */ + kernel_code->code_properties.enable_sgpr_private_segment_size =3D 0; + /* total =3D 14 SGPRs */ + kernel_code->code_properties.reserved0 =3D 0; + /* GFX10+ */ + kernel_code->code_properties.enable_wavefront_size32 =3D 0; + kernel_code->code_properties.uses_dynamic_stack =3D 0; + kernel_code->code_properties.reserved1 =3D 0; + + debug_kernel_descriptor(kernel_code); +} + +static void kfd_kernel_gfx10_init(struct knod *knod) +{ + struct kernel_descriptor *kernel_code =3D knod->kernels[0]->kaddr; + + kernel_code->group_segment_fixed_size =3D 0; + kernel_code->private_segment_fixed_size =3D 8192; + kernel_code->kernarg_size =3D 64; + kernel_code->kernel_code_entry_byte_offset =3D 1024; + + /* + * User SGPR layout - loaded in fixed order, disabled entries are + * skipped (not reserved). The resulting SGPR map depends on which + * flags are enabled: + * + * enable_sgpr_private_segment_buffer -> 4 SGPRs (s[0:3]) + * enable_sgpr_dispatch_ptr -> 2 SGPRs (s[4:5]) + * enable_sgpr_queue_ptr -> 2 SGPRs + * enable_sgpr_kernarg_segment_ptr -> 2 SGPRs + * enable_sgpr_dispatch_id -> 2 SGPRs + * enable_sgpr_flat_scratch_init -> 2 SGPRs + * enable_sgpr_private_segment_size -> 1 SGPR + * + * System SGPRs (WorkgroupId etc.) follow immediately after the + * last user SGPR. user_sgpr_count must match the total above. + */ + /* 4 SGPRs */ + kernel_code->code_properties.enable_sgpr_private_segment_buffer =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_dispatch_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_queue_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_kernarg_segment_ptr =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_dispatch_id =3D 1; + /* 2 SGPRs */ + kernel_code->code_properties.enable_sgpr_flat_scratch_init =3D 1; + /* disabled -> s14/s15 free for workgroup_id */ + kernel_code->code_properties.enable_sgpr_private_segment_size =3D 0; + /* total =3D 14 SGPRs */ + kernel_code->code_properties.reserved0 =3D 0; + if (knod_bpf_wave32) + kernel_code->code_properties.enable_wavefront_size32 =3D 1; + else + kernel_code->code_properties.enable_wavefront_size32 =3D 0; + kernel_code->code_properties.uses_dynamic_stack =3D 0; + kernel_code->code_properties.reserved1 =3D 0; + + kernel_code->compute_pgm_rsrc3.accum_offset =3D 0; + kernel_code->compute_pgm_rsrc3.reserved0 =3D 0; + kernel_code->compute_pgm_rsrc3.tg_split =3D 0; + kernel_code->compute_pgm_rsrc3.reserved1 =3D 0; + + if (kernel_code->code_properties.enable_wavefront_size32 =3D=3D 1) + kernel_code->compute_pgm_rsrc1.granulated_workitem_vgpr_count =3D + (256 / 8) - 1; + else + kernel_code->compute_pgm_rsrc1.granulated_workitem_vgpr_count =3D + (256 / 4) - 1; + kernel_code->compute_pgm_rsrc1.granulated_wavefront_sgpr_count =3D 0; + kernel_code->compute_pgm_rsrc1.priority =3D 0; + kernel_code->compute_pgm_rsrc1.float_round_mode_32 =3D 0; + kernel_code->compute_pgm_rsrc1.float_round_mode_16_64 =3D 0; + kernel_code->compute_pgm_rsrc1.float_denorm_mode_32 =3D 3; + kernel_code->compute_pgm_rsrc1.float_denorm_mode_16_64 =3D 3; + kernel_code->compute_pgm_rsrc1.priv =3D 0; + kernel_code->compute_pgm_rsrc1.enable_dx10_clamp =3D 1; + kernel_code->compute_pgm_rsrc1.debug_mode =3D 0; + kernel_code->compute_pgm_rsrc1.enable_ieee_mode =3D 1; + kernel_code->compute_pgm_rsrc1.bulky =3D 0; + kernel_code->compute_pgm_rsrc1.cdbg_user =3D 0; + kernel_code->compute_pgm_rsrc1.fp16_ovfl =3D 0; + kernel_code->compute_pgm_rsrc1.reserved0 =3D 0; + kernel_code->compute_pgm_rsrc1.wgp_mode =3D 0; + kernel_code->compute_pgm_rsrc1.mem_ordered =3D 1; + kernel_code->compute_pgm_rsrc1.fwd_progress =3D 0; + + kernel_code->compute_pgm_rsrc2.enable_private_segment =3D 0; + kernel_code->compute_pgm_rsrc2.user_sgpr_count =3D 14; /* 4+2+2+2+2+2 */ + kernel_code->compute_pgm_rsrc2.enable_trap_handler =3D 0; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_x =3D 1; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_y =3D 1; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_id_z =3D 0; + kernel_code->compute_pgm_rsrc2.enable_sgpr_workgroup_info =3D 0; + kernel_code->compute_pgm_rsrc2.enable_vgpr_workitem_id =3D 1; + kernel_code->compute_pgm_rsrc2.enable_exception_address_watch =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_memory =3D 0; + kernel_code->compute_pgm_rsrc2.granulated_lds_size =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_invalid_operation =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_fp_denormal_source =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_division_by_zero =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_overflow =3D 0; + kernel_code->compute_pgm_rsrc2 + .enable_exception_ieee_754_fp_underflow =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_ieee_754_fp_inexact =3D 0; + kernel_code->compute_pgm_rsrc2.enable_exception_int_divide_by_zero =3D 0; + kernel_code->compute_pgm_rsrc2.reserved0 =3D 0; + + debug_kernel_descriptor(kernel_code); +} + +static int kfd_kernel_init(struct knod *knod, struct knod_bpf_priv *priv) +{ + struct kernel_descriptor *kd; + + if (!knod->kernels[1]) + return -ENOMEM; + + /* + * Pass-through starts on slot 0; the first XDP prog attach stages into + * slot 1 and flips the active index there, ping-ponging on each + * install. + */ + priv->active_idx =3D 0; + + if (priv->isa_version =3D=3D 9) + kfd_kernel_gfx9_init(knod); + else if (priv->isa_version =3D=3D 10) + kfd_kernel_gfx10_init(knod); + + /* + * Slot 1 must carry the same kernel-descriptor as slot 0 -- gfx init + * only touches slot 0, and slot 1's BO is otherwise uninitialised, + * which stalls the compute queue. Copy the kd + pre-code region. + */ + kd =3D knod->kernels[0]->kaddr; + memcpy(knod->kernels[1]->kaddr, knod->kernels[0]->kaddr, + kd->kernel_code_entry_byte_offset); + knod_bpf_gpu_mem_fence(priv); + + return 0; +} + +static struct knod_bpf_work_sq * +__knod_get_free_work_sq(struct knod_bpf_priv *priv) +{ + return list_first_entry_or_null(&priv->free_list_sqw, + struct knod_bpf_work_sq, list); +} + +/* Prepare a dispatch: peek SPSC rings and fill params, but do not submit. + * Returns the prepared sqw (with backlogs > 0), or NULL if nothing to do. + * + * A single in-flight AQL queue means the worker never has to reserve SPSC + * ranges ahead of the current dispatch. The SPSC acquired pointer is adva= nced + * only after the GPU finishes the dispatch that consumed those entries. + */ +static struct knod_bpf_work_sq *knod_prepare_bpf(struct knod_bpf_priv *pri= v) +{ + int i, cnt, backlogs =3D 0; + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_work_sq *sqw; + struct knod_bpf_param *param; + + if (READ_ONCE(priv->installing_kernel)) + return NULL; + + if (!priv->pass_prog_buf && !READ_ONCE(priv->prog)) + return NULL; + + sqw =3D __knod_get_free_work_sq(priv); + if (!sqw) + return NULL; + + param =3D (struct knod_bpf_param *)sqw->param->kaddr; + memset(sqw->queue_idx, 0, sizeof(sqw->queue_idx)); + + /* 2D dispatch: queue_id =3D workgroup_id_y, tid =3D workitem within WG. + * Per-queue bds live in sqw->bds[i * batch_size + tid] and shader + * indexes sub[] / sqw->bds[] using (queue_id * batch_size + tid). + * No cumulative start_idx -- each queue's slot range is fixed by i. + */ + for (i =3D 0; i < priv->nr_works; i++) { + int slot =3D i * priv->batch_size; + unsigned int skip =3D 0, j; + + /* Stage past every in-flight dispatch's claim on this queue so + * the new sqw reads disjoint SPSC slots. Peek self-limits: if + * the ring holds fewer entries past @skip, cnt shrinks (or 0). + */ + for (j =3D 0; j < priv->inflight_cnt; j++) + skip +=3D priv->inflight[j]->queue_idx[i]; + + param->queues[i].count =3D 0; + spsc_peek_at(&knodev->wpriv[i].spsc_bds, skip, + (void **)&sqw->bds[slot], + priv->batch_size, &cnt); + if (!cnt) { + sqw->queue_idx[i] =3D 0; + param->queues[i].count =3D 0; + continue; + } + + /* Fill queue descriptor for GPU direct SPSC read. + * ring_start is the absolute ring position where this sqw + * begins - shader reads slots[(ring_start + tid) & mask]. + * Offset by skip to keep staged sqws disjoint. + */ + param->queues[i].pool_gaddr =3D knodev->wpriv[i].spsc_pool_gaddr; + param->queues[i].base_gaddr =3D priv->queue_base_gaddr[i]; + param->queues[i].count =3D cnt; + param->queues[i].ring_start =3D + knodev->wpriv[i].spsc_bds.acquired + skip; + param->queues[i].ring_mask =3D + knodev->wpriv[i].spsc_bds.mask; + + backlogs +=3D cnt; + sqw->queue_idx[i] =3D cnt; + } + sqw->backlogs =3D backlogs; + param->nr_backlogs =3D backlogs; + param->nr_queues =3D priv->nr_works; + param->spsc_stride =3D ALIGN(sizeof(struct spsc_bd), SMP_CACHE_BYTES); + for (i =3D 0; i < priv->nr_works; i++) { + param->pass_count[i] =3D 0; + param->pass_meta_buf_gaddr[i] =3D priv->pass_meta_buf ? + priv->pass_meta_buf->gaddr + + (u64)i * priv->pass_pkts_per_queue * + KNOD_PASS_SLOT_SIZE : + 0; + } + param->ktime_ns =3D ktime_get_ns(); + + if (!sqw->backlogs) + return NULL; + + list_del_init(&sqw->list); + return sqw; +} + +/* Submit a prepared sqw: write AQL packet, ring doorbell, record stats. */ +static void knod_submit_bpf(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct amd_signal *signal =3D + (struct amd_signal *)priv->knod->kaql[0].queue_signal->kaddr; + struct knod_bpf_stats *stats =3D &priv->stats; + struct knod_dispatch_params p; + int i, bucket =3D KNOD_BL_BUCKETS - 1; + + /* The @inflight_cnt dispatches already in flight decrement the signal + * before this one, so this sqw completes when the signal drops below + * (current value - inflight_cnt). + */ + sqw->sigval =3D signal->value - priv->inflight_cnt; + sqw->expire =3D jiffies + msecs_to_jiffies(knod_bpf_expire); + if (static_branch_unlikely(&knod_stats_key)) { + sqw->dispatch_time =3D ktime_get(); + + stats->backlogs_total +=3D sqw->backlogs; + for (i =3D 0; i < KNOD_BL_BUCKETS - 1; i++) { + if (sqw->backlogs <=3D bl_bounds[i]) { + bucket =3D i; + break; + } + } + stats->backlogs_hist[bucket]++; + } + + knod_bpf_fill_dispatch(priv, sqw, &p); + /* publish dispatch params before the AQL packet becomes visible */ + wmb(); + knod_setup_header(priv->knod, &p, 0); +} + +/* Phase 1: advance SPSC consumer pointers so next dispatch can peek + * new entries. + */ +static void knod_complete_acquire(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct knod_dev *knodev =3D priv->knodev; + int i; + + for (i =3D 0; i < priv->nr_works; i++) { + if (sqw->queue_idx[i] >=3D 1) { + spsc_acquire(&knodev->wpriv[i].spsc_bds, NULL, + sqw->queue_idx[i], NULL); + } + } +} + +/* Phase 2: schedule NAPI and free sqw. Can run after the next dispatch + * has been submitted - napi_schedule overlaps with GPU execution. + */ +static void knod_complete_napi(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_stats *stats =3D &priv->stats; + ktime_t start; + int i; + + if (static_branch_unlikely(&knod_stats_key)) + start =3D ktime_get(); + + for (i =3D 0; i < priv->nr_works; i++) { + if (sqw->queue_idx[i] >=3D 1) + knod_napi_kick(&knodev->wpriv[i]); + } + + sqw->backlogs =3D 0; + sqw->expire =3D 0; + list_add_tail_rcu(&sqw->list, &priv->free_list_sqw); + + if (static_branch_unlikely(&knod_stats_key)) { + u64 ns =3D ktime_to_ns(ktime_sub(ktime_get(), start)); + + stats->decode_act_total_ns +=3D ns; + stats->decode_act_count++; + if (ns > stats->decode_act_max_ns) + stats->decode_act_max_ns =3D ns; + } +} + +static void knod_bpf_update_kernel_descriptor(struct knod_bpf_priv *priv, + struct kernel_descriptor *kd, + const struct knod_prog *knod_prog) +{ + unsigned int sgprs_used; + + if (priv->isa_version !=3D 9 || !knod_prog) + return; + + sgprs_used =3D knod_prog->initial_exec_sreg + 2; + kd->compute_pgm_rsrc1.granulated_wavefront_sgpr_count =3D + knod_bpf_gfx9_sgpr_granule(sgprs_used); +} + +/* + * Install kernel code into the inactive slot and atomically flip the acti= ve + * index. The active slot is never modified while the GPU dispatches it, = so + * the swap never races the live pipeline, and the worker is not touched: = new + * dispatches pick up the new slot, the in-flight one finishes on the old = slot. + */ +static void knod_bpf_install_kernel(struct knod_bpf_priv *priv, + const struct knod_prog *knod_prog, + const void *code, u32 size) +{ + struct kernel_descriptor *kd; + struct knod *knod =3D priv->knod; + struct knod_mem *slot; + u32 entry_off; + u32 image_len; + int idx; + + if (!code || !size || !knod->kernels[1]) + return; + + /* + * Before the worker runs, install in place; once it is dispatching, + * stage into the inactive slot and flip the active index so the live + * pipeline never reads a half-written slot. + */ + if (!priv->start || !knod->worker) + idx =3D priv->active_idx; + else + idx =3D priv->active_idx ^ 1; + + slot =3D knod->kernels[idx]; + kd =3D slot->kaddr; + knod_bpf_update_kernel_descriptor(priv, kd, knod_prog); + entry_off =3D kd->kernel_code_entry_byte_offset; + if (WARN_ON(entry_off >=3D slot->size)) + return; + if (WARN_ON(size > slot->size - entry_off)) + size =3D slot->size - entry_off; + image_len =3D entry_off + size; + + memcpy(slot->kaddr + entry_off, code, size); + if (image_len < slot->size) { + u32 clear_end =3D min_t(u32, slot->size, + entry_off + KNOD_BPF_PROG_BUF_SIZE); + + if (image_len < clear_end) + memset(slot->kaddr + image_len, 0, + clear_end - image_len); + } + /* + * kernels[] is write-combining VRAM. smp_wmb() is only a compiler + * barrier on x86 and does NOT drain the WC buffers, so the GPU could + * fetch half-written code and spin. wmb() (sfence) flushes WC to VRAM + * before we publish the new slot; the dispatch doorbell is ordered + * behind it. + */ + wmb(); + knod_bpf_gpu_mem_fence(priv); + WRITE_ONCE(priv->kernel_image_len[idx], image_len); + + if (idx !=3D priv->active_idx) + WRITE_ONCE(priv->active_idx, idx); +} + +/* + * Keep the just-built pass-kernel IR for the debugfs "insn" dump, so it c= an + * show the pass-through kernel when no XDP prog is attached. The machine= code + * already lives in the kernel slot; this only retains the meta list. Reb= uilt + * on every start (old metas freed first), released in knod_priv_exit(). + */ +static void knod_bpf_retain_pass_ir(struct knod_bpf_priv *priv, + struct knod_prog *src) +{ + struct knod_insn_meta *meta, *tmp; + struct knod_prog *dst =3D priv->pass_knod_prog; + + if (!dst) { + dst =3D kzalloc_obj(*dst, GFP_KERNEL); + if (!dst) + return; + INIT_LIST_HEAD(&dst->pre_insns); + INIT_LIST_HEAD(&dst->insns); + INIT_LIST_HEAD(&dst->post_insns); + priv->pass_knod_prog =3D dst; + } else { + list_for_each_entry_safe(meta, tmp, &dst->pre_insns, l) { + list_del(&meta->l); + kfree(meta); + } + list_for_each_entry_safe(meta, tmp, &dst->insns, l) { + list_del(&meta->l); + kfree(meta); + } + list_for_each_entry_safe(meta, tmp, &dst->post_insns, l) { + list_del(&meta->l); + kfree(meta); + } + } + list_splice_init(&src->pre_insns, &dst->pre_insns); + list_splice_init(&src->insns, &dst->insns); + list_splice_init(&src->post_insns, &dst->post_insns); +} + +static void knod_bpf_layout_sregs(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog) +{ + if (priv->isa_version !=3D 9) + return; + + knod_prog->initial_exec_sreg =3D + knod_prog->exec_save_base + knod_prog->exec_save_pairs_used * 2; +} + +static int knod_bpf_jit_pass_kernel(struct knod_bpf_priv *priv) +{ + struct list_head *lists[2]; + struct knod_insn_meta *meta, *tmp, *epi; + struct amdgcn_param32 p[3]; + struct knod *knod =3D priv->knod; + struct knod_prog pass_prog; + int pass_branch_idx; + u32 pass_dwords; + u8 *buf, *ptr; + u32 total =3D 0; + int i, j, li, err; + + memset(&pass_prog, 0, sizeof(pass_prog)); + INIT_LIST_HEAD(&pass_prog.pre_insns); + INIT_LIST_HEAD(&pass_prog.insns); + INIT_LIST_HEAD(&pass_prog.post_insns); + pass_prog.knod =3D knod; + pass_prog.knodev =3D priv->knodev; + if (priv->isa_version =3D=3D 10) { + pass_prog.done_mask_sreg =3D 32; + pass_prog.exec_save_base =3D 34; + pass_prog.initial_exec_sreg =3D + KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX10; + } else { + pass_prog.done_mask_sreg =3D KNOD_AMDGPU_DONE_MASK_SREG; + pass_prog.exec_save_base =3D KNOD_AMDGPU_EXEC_SAVE_SREG_BASE; + pass_prog.initial_exec_sreg =3D + KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX9; + } + + knod_bpf_layout_sregs(priv, &pass_prog); + err =3D knod_prog_prepare_insns(priv, &pass_prog); + if (err) + return err; + + epi =3D kzalloc_obj(*epi, GFP_KERNEL); + if (!epi) { + err =3D -ENOMEM; + goto free_pro; + } + + /* BPF/XDP actions are 32-bit values; mlx5 consumes bd->act as low32. */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_iset32(&p[1], XDP_PASS); + knod_emit(priv, epi, v_mov_b32_e32, p[0], p[1]); + + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_vset32(&p[1], KNOD_AMDGPU_SLOT_VREG_LO); + knod_emit(priv, epi, global_store_dword, p[0], p[1], + offsetof(struct spsc_bd, act)); + + /* XDP_PASS detection: v_cmp_eq_u32 XDP_PASS, R0 -> VCC */ + knod_iset32(&p[0], XDP_PASS); + knod_vset32(&p[1], KNOD_AMDGPU_VREG0_LO); + knod_emit(priv, epi, v_cmp_eq_u32, p[0], p[1]); + + pass_branch_idx =3D epi->amdgpu_insns; + knod_emit(priv, epi, s_cbranch_vccz, 0); + + /* EXEC &=3D VCC - only PASS lanes proceed */ + knod_emit(priv, epi, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* v_mov param addr to VGPR pair for pass_count atomic */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, epi, v_mov_b32_e32, p[0], p[1]); + + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_HI); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_HI); + knod_emit(priv, epi, v_mov_b32_e32, p[0], p[1]); + + /* v_mov v2, s15 (queue_idx -> VGPR) */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG1_LO); + knod_sset32(&p[1], KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_emit(priv, epi, v_mov_b32_e32, p[0], p[1]); + + /* v_lshlrev_b32 v2, 2, v2 (queue_idx * 4) */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG1_LO); + knod_iset32(&p[1], 2); + knod_vset32(&p[2], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, epi, v_lshlrev_b32, p[0], p[1], p[2]); + + /* v_add_u32 TMP9_LO, v2, TMP9_LO (param_addr +=3D queue_idx * 4) */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_VREG1_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, epi, v_add_u32, p[0], p[1], p[2]); + + /* v_mov TMP10_LO, 1 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_iset32(&p[1], 1); + knod_emit(priv, epi, v_mov_b32_e32, p[0], p[1]); + + /* global_atomic_add pass_count[q]++, GLC=3D1 -> old_val in TMP10_LO */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG10_LO); + knod_emit(priv, epi, global_atomic_add, p[0], p[1], p[2], + offsetof(struct knod_bpf_param, pass_count), 1); + + /* s_waitcnt vmcnt(0) */ + knod_emit(priv, epi, s_waitcnt_vmcnt); + + /* v_sub_u32 TMP9_LO, TMP9_LO, v2 (restore param_addr_lo) */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[2], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, epi, v_sub_u32, p[0], p[1], p[2]); + + /* old_val * 2 for pass_indices u16 stride */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_iset32(&p[1], 1); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG10_LO); + knod_emit(priv, epi, v_lshlrev_b32, p[0], p[1], p[2]); + + /* addr_lo +=3D old_val * 2 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG10_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, epi, v_add_u32, p[0], p[1], p[2]); + + /* global_store_short pass_indices[old_val], BACKLOG_IDX_VREG */ + knod_vset32(&p[0], KNOD_AMDGPU_BACKLOG_IDX_VREG); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, epi, global_store_short, p[0], p[1], + offsetof(struct knod_bpf_param, pass_indices)); + + /* Store len + src_addr to pass_meta_buf slot header */ + knod_emit_pass_addr_store(priv, epi); + + /* Patch s_cbranch_vccz offset (skip pass handling) */ + pass_dwords =3D 0; + for (j =3D pass_branch_idx + 1; j < epi->amdgpu_insns; j++) + pass_dwords +=3D epi->amdgpu_insn[j].size / 4; + emit_s_cbranch_vccz(priv->isa_version, + &epi->amdgpu_insn[pass_branch_idx], pass_dwords); + + knod_emit(priv, epi, s_endpgm); + + if (priv->isa_version >=3D 10) { + for (j =3D 0; j < 16 && epi->amdgpu_insns < KNOD_META_INSNS; j++) + knod_emit(priv, epi, s_code_end); + } + list_add_tail(&epi->l, &pass_prog.post_insns); + + /* Linearize prologue + epilogue into pass_prog_buf */ + lists[0] =3D &pass_prog.pre_insns; + lists[1] =3D &pass_prog.post_insns; + + for (li =3D 0; li < 2; li++) { + list_for_each_entry(meta, lists[li], l) + for (i =3D 0; i < meta->amdgpu_insns; i++) + total +=3D meta->amdgpu_insn[i].size; + } + + kfree(priv->pass_prog_buf); + buf =3D kzalloc(total, GFP_KERNEL); + if (!buf) { + err =3D -ENOMEM; + goto free_all; + } + + ptr =3D buf; + for (li =3D 0; li < 2; li++) { + list_for_each_entry(meta, lists[li], l) + for (i =3D 0; i < meta->amdgpu_insns; i++) { + memcpy(ptr, &meta->amdgpu_insn[i], + meta->amdgpu_insn[i].size); + ptr +=3D meta->amdgpu_insn[i].size; + } + } + + priv->pass_prog_buf =3D buf; + priv->pass_prog_size =3D total; + + knod_bpf_install_kernel(priv, &pass_prog, priv->pass_prog_buf, + priv->pass_prog_size); + /* Remember which slot now holds pass so detach can flip back to it. */ + priv->pass_idx =3D priv->active_idx; + + pr_info("knod_bpf: pass kernel JIT'd %u bytes\n", priv->pass_prog_size); + err =3D 0; + /* Retain the IR (moves the lists out) before the cleanup below + * frees. + */ + knod_bpf_retain_pass_ir(priv, &pass_prog); + +free_all: + list_for_each_entry_safe(meta, tmp, &pass_prog.post_insns, l) { + list_del_init(&meta->l); + kfree(meta); + } +free_pro: + list_for_each_entry_safe(meta, tmp, &pass_prog.pre_insns, l) { + list_del_init(&meta->l); + kfree(meta); + } + return err; +} + +static void knod_bpf_reset_sqw(struct knod_bpf_work_sq *sqw) +{ + if (!sqw) + return; + + sqw->backlogs =3D 0; + sqw->expire =3D 0; +} + +static void knod_bpf_wait_sqw(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct amd_signal *signal; + unsigned long deadline; + + if (!sqw) + return; + + signal =3D (struct amd_signal *) + priv->knod->kaql[0].queue_signal->kaddr; + deadline =3D jiffies + msecs_to_jiffies(1000); + + while (sqw->sigval <=3D READ_ONCE(signal->value) && + time_before(jiffies, deadline)) + usleep_range(100, 200); + + if (sqw->sigval <=3D READ_ONCE(signal->value)) + pr_warn("knod: timed out waiting for GPU dispatch completion\n"); +} + +static void knod_bpf_drain_worker(struct knod_bpf_priv *priv) +{ + struct knod_bpf_work_sq *sqw; + + /* stop() runs on interface-down AND on every feature switch, both + * with mlx5 RX possibly still producing into knodev->wpriv[].spsc_bds. + * So we only quiesce the GPU here; the NIC-owned RX SPSC rings are + * drained on interface-down by mlx5e_rx_offload_stop(). + */ + while (priv->inflight_cnt) { + sqw =3D priv->inflight[--priv->inflight_cnt]; + priv->inflight[priv->inflight_cnt] =3D NULL; + knod_bpf_wait_sqw(priv, sqw); + knod_bpf_reset_sqw(sqw); + list_add_tail_rcu(&sqw->list, &priv->free_list_sqw); + } +} + +static void knod_bpf_drain(struct knod_bpf_priv *priv) +{ + knod_bpf_drain_worker(priv); +} + +static void knod_bpf_stop_worker(struct knod_bpf_priv *priv) +{ + priv->start =3D 0; + if (priv->worker_task) { + kthread_stop(priv->worker_task); + put_task_struct(priv->worker_task); + priv->worker_task =3D NULL; + } + synchronize_net(); +} + +static void knod_bpf_configure_worker(struct knod_bpf_priv *priv) +{ + knod_bpf_stop_worker(priv); + knod_bpf_drain(priv); + + priv->inflight_cnt =3D 0; +} + +static int knod_bpf_start_worker(struct knod_bpf_priv *priv) +{ + struct task_struct *p; + + p =3D kthread_run(knod_bpf_worker, priv, "knod_%d_0", + priv->knodev->accel->id); + if (IS_ERR(p)) + return PTR_ERR(p); + + get_task_struct(p); + priv->worker_task =3D p; + return 0; +} + +static bool knod_bpf_uses_percpu(struct knod_bpf_priv *priv) +{ + struct knod_bpf_map *knod_map; + + list_for_each_entry(knod_map, &priv->knodev->accel->xdp.bound_maps, + list) + if (knod_map->knod_map_obj->map_type =3D=3D + BPF_MAP_TYPE_PERCPU_ARRAY) + return true; + return false; +} + +/* Fan out one workgroup per CU (rounded down to a power of two). PERCPU = maps + * keep one instance per RX queue, so a queue must stay on a single workgr= oup; + * force xgroups=3D1 when the program uses them. Non-PERCPU maps are glob= ally + * shared with atomics, so fan-out is safe there. This replaces the old m= anual + * xgroups knob. + */ +static unsigned int knod_bpf_auto_xgroups(struct knod_bpf_priv *priv) +{ + struct amdgpu_device *adev =3D NULL; + unsigned int cus, xgroups; + + if (knod_bpf_uses_percpu(priv)) + return 1; + + if (priv->knod->dev) + adev =3D priv->knod->dev->adev; + else if (priv->knod->process && priv->knod->process->pdds[0]) + adev =3D priv->knod->process->pdds[0]->dev->adev; + if (!adev || !priv->nr_works) + return 1; + + cus =3D adev->gfx.cu_info.number; + xgroups =3D cus / priv->nr_works; + if (!xgroups) + xgroups =3D 1; + return rounddown_pow_of_two(xgroups); +} + +/* Per-queue dispatch batch =3D workgroups * xgroups packets, capped by the + * static descriptor array and rounded down to a power of two (the shader + * derives the flat slot as queue_id << ilog2(batch_size) + local_idx). + */ +static unsigned int knod_bpf_batch_size(struct knod_bpf_priv *priv) +{ + unsigned int xgroups =3D knod_bpf_auto_xgroups(priv); + unsigned int max_flat =3D KNOD_BPF_BACKLOGS_MAX / priv->nr_works; + unsigned int batch =3D min_t(unsigned int, + knod_bpf_workgroups * xgroups, max_flat); + + if (!batch) + batch =3D knod_bpf_workgroups; + return rounddown_pow_of_two(batch); +} + +static void knod_bpf_start(struct knod_dev *knodev) +{ + struct knod_bpf_priv *priv =3D + (struct knod_bpf_priv *)knodev->accel->xdp.priv; + struct bpf_prog *prog; + unsigned int active_rxq; + int err; + + priv->start =3D 1; + active_rxq =3D knod_bpf_active_rxq_count(knodev->netdev); + if (active_rxq && active_rxq !=3D priv->nr_works) + pr_warn("knod_bpf: active rx queues changed from %d to %u; using initial= ized count\n", + priv->nr_works, active_rxq); + + priv->batch_size =3D knod_bpf_batch_size(priv); + + knod_jit_dbg(" batch_size =3D %d\n", priv->batch_size); + knod_bpf_configure_worker(priv); + pr_info("knod_bpf: using single AQL queue, rx_works=3D%d active_rxq=3D%u = batch_size=3D%d xgroups=3D%u\n", + priv->nr_works, active_rxq, priv->batch_size, + priv->batch_size / knod_bpf_workgroups); + + if (knod_bpf_jit_pass_kernel(priv)) + pr_warn("knod_bpf: pass kernel JIT failed\n"); + + prog =3D READ_ONCE(priv->prog); + if (prog) + knod_setup_bpf_prog(prog); + + priv->start =3D 1; + err =3D knod_bpf_start_worker(priv); + if (err) { + pr_err("knod_bpf: start_worker failed: %d\n", err); + priv->start =3D 0; + return; + } +} + +static void knod_bpf_stop(struct knod_dev *knodev) +{ + struct knod_bpf_priv *priv =3D + (struct knod_bpf_priv *)knodev->accel->xdp.priv; + + knod_bpf_stop_worker(priv); + knod_bpf_drain(priv); + + kfree(priv->pass_prog_buf); + priv->pass_prog_buf =3D NULL; + priv->pass_prog_size =3D 0; +} + +/* + * Flip the dispatched kernel back to pass-through when the XDP prog is + * detached. The pass slot already holds the pass code, so this is just an + * atomic index flip -- no re-copy. + */ +static void knod_bpf_reload_pass(struct knod_dev *knodev) +{ + struct knod_bpf_priv *priv =3D knodev->accel->xdp.priv; + + if (priv) + WRITE_ONCE(priv->active_idx, priv->pass_idx); +} + +static void knod_setup_bpf_prog(struct bpf_prog *prog) +{ + struct knod_prog *knod_prog =3D prog->aux->offload->dev_priv; + struct knod_dev *knodev =3D knod_prog->knodev; + struct knod_insn_meta *meta, *tmp; + struct knod_bpf_priv *priv; + u8 *kernel_ptr, *ptr; + u32 total_bytes; + u32 *debug_ptr; + int i; + + priv =3D (struct knod_bpf_priv *)knodev->accel->xdp.priv; + WRITE_ONCE(priv->installing_kernel, true); + + if (prog) { + WRITE_ONCE(priv->prog, NULL); + kernel_ptr =3D priv->prog_buf; + memset(priv->prog_buf, 0, KNOD_BPF_PROG_BUF_SIZE); + + list_for_each_entry(meta, &priv->knod_prog->pre_insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) { + ptr =3D (u8 *)&meta->amdgpu_insn[i]; + debug_ptr =3D (u32 *)ptr; + + memcpy(kernel_ptr, ptr, + meta->amdgpu_insn[i].size); + kernel_ptr +=3D meta->amdgpu_insn[i].size; + + if (meta->amdgpu_insn[i].size =3D=3D 4) { + knod_jit_dbg(" 0x%.8X\t%.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0]); + } else if (meta->amdgpu_insn[i].size =3D=3D 8) { + knod_jit_dbg(" 0x%.8X\t%.8X %.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0], debug_ptr[1]); + } else if (meta->amdgpu_insn[i].size =3D=3D 12) { + knod_jit_dbg(" 0x%.8X\t%.8X %.8X %.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0], + debug_ptr[1], debug_ptr[2]); + } else { + WARN_ON_ONCE(1); + } + } + } + + list_for_each_entry(meta, &priv->knod_prog->insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) { + ptr =3D (u8 *)&meta->amdgpu_insn[i]; + debug_ptr =3D (u32 *)ptr; + + memcpy(kernel_ptr, ptr, + meta->amdgpu_insn[i].size); + kernel_ptr +=3D meta->amdgpu_insn[i].size; + if (meta->amdgpu_insn[i].size =3D=3D 4) { + knod_jit_dbg(" 0x%.8X\t%.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0]); + } else if (meta->amdgpu_insn[i].size =3D=3D 8) { + knod_jit_dbg(" 0x%.8X\t%.8X %.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0], debug_ptr[1]); + } else if (meta->amdgpu_insn[i].size =3D=3D 12) { + knod_jit_dbg(" 0x%.8X\t%.8X %.8X %.8X\n", + meta->amdgpu_insn_idx, + debug_ptr[0], + debug_ptr[1], debug_ptr[2]); + } else { + WARN_ON_ONCE(1); + } + } + } + + list_for_each_entry(meta, &priv->knod_prog->post_insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) { + ptr =3D (u8 *)&meta->amdgpu_insn[i]; + debug_ptr =3D (u32 *)ptr; + + memcpy(kernel_ptr, ptr, + meta->amdgpu_insn[i].size); + kernel_ptr +=3D meta->amdgpu_insn[i].size; + if (meta->amdgpu_insn[i].size =3D=3D 4) { + knod_jit_dbg(" %.8X\n", debug_ptr[0]); + } else if (meta->amdgpu_insn[i].size =3D=3D 8) { + knod_jit_dbg(" %.8X %.8X\n", + debug_ptr[0], debug_ptr[1]); + } else if (meta->amdgpu_insn[i].size =3D=3D 12) { + knod_jit_dbg(" %.8X %.8X %.8X\n", + debug_ptr[0], + debug_ptr[1], debug_ptr[2]); + } else { + WARN_ON_ONCE(1); + } + } + } + total_bytes =3D kernel_ptr - (u8 *)priv->prog_buf; + + pr_debug("KNOD JIT: total binary size =3D %u bytes (limit %u)\n", + total_bytes, KNOD_BPF_PROG_BUF_SIZE); + if (WARN_ON(total_bytes > KNOD_BPF_PROG_BUF_SIZE)) + total_bytes =3D KNOD_BPF_PROG_BUF_SIZE; + knod_bpf_install_kernel(priv, knod_prog, priv->prog_buf, + total_bytes); + WRITE_ONCE(priv->prog, prog); + } else { + WRITE_ONCE(priv->prog, NULL); + list_for_each_entry_safe(meta, tmp, &priv->knod_prog->pre_insns, + l) { + list_del_init(&meta->l); + kfree(meta); + } + + list_for_each_entry_safe(meta, tmp, &priv->knod_prog->insns, + l) { + list_del_init(&meta->l); + kfree(meta); + } + + list_for_each_entry_safe(meta, tmp, + &priv->knod_prog->post_insns, l) { + list_del_init(&meta->l); + kfree(meta); + } + + /* bbs points into the metas just freed */ + kfree(priv->knod_prog->bbs); + priv->knod_prog->bbs =3D NULL; + priv->knod_prog->n_bbs =3D 0; + + if (priv->pass_prog_buf) + knod_bpf_install_kernel(priv, priv->pass_knod_prog, + priv->pass_prog_buf, + priv->pass_prog_size); + } + WRITE_ONCE(priv->installing_kernel, false); +} + +static int knod_bpf_map_hash_init_elem(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj) +{ + unsigned int *queue =3D (unsigned int *)knod_map->queue_mem->kaddr; + unsigned int *bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + void *elems =3D knod_map->hash_elems_mem->kaddr; + struct knod_bpf_hash_elem_obj *e; + int i, elem_size; + + elem_size =3D sizeof(struct knod_bpf_hash_elem_obj) + + roundup(knod_map_obj->key_size, 4) + + roundup(knod_map_obj->value_size, 4); + knod_map_obj->meta.hmeta.elem_size =3D elem_size; + + for (i =3D 0; i < knod_map_obj->meta.hmeta.n_buckets; i++) + bucket[i] =3D KNOD_BPF_HASH_NEXT_END; + + for (i =3D 0; i < knod_map_obj->max_entries; i++) { + e =3D elems + (i * elem_size); + e->next =3D KNOD_BPF_HASH_NEXT_END; + queue[i] =3D i; + } + knod_map_obj->meta.hmeta.cur =3D knod_map_obj->max_entries - 1; + + return 0; +} + +static inline unsigned char * +knod_bpf_hash_elem_kv(struct knod_bpf_hash_elem_obj *e) +{ + return (unsigned char *)e + offsetof(struct knod_bpf_hash_elem_obj, kv); +} + +static inline void * +knod_bpf_array_value_ptr(struct knod_bpf_map_obj *knod_map_obj, + unsigned int idx) +{ + return (unsigned char *)knod_map_obj + + offsetof(struct knod_bpf_map_obj, bucket) + + (size_t)idx * knod_map_obj->value_size; +} + +static int __knod_bpf_map_alloc(struct knod_dev *knodev, + struct bpf_offloaded_map *offmap) +{ + struct knod_bpf_priv *priv =3D + (struct knod_bpf_priv *)knodev->accel->xdp.priv; + struct knod_mem *mem, *queue_mem, *hash_elems_mem, *gc_mem; + int order, size, queue_size, i, value_size, nents, err; + int n_instances =3D 1; + int flags =3D KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT | + KFD_IOC_ALLOC_MEM_FLAGS_PUBLIC | + KFD_IOC_ALLOC_MEM_FLAGS_VRAM; + struct knod_bpf_map_obj *knod_map_obj; + struct knod *knod =3D priv->knod; + struct knod_bpf_map *knod_map; + unsigned int gc_size; + unsigned int *q; + + if (offmap->map.map_type =3D=3D BPF_MAP_TYPE_HASH) { + value_size =3D sizeof(unsigned int); + nents =3D roundup_pow_of_two(offmap->map.max_entries); + } else { + value_size =3D offmap->map.value_size; + nents =3D offmap->map.max_entries; + } + + /* PERCPU_ARRAY keeps one value array per GPU workgroup (percpu + * instance) so each CU updates its own copy - no cross-CU atomic + * contention. Instances map 1:1 to the per-cpu value buffer, so + * allocate num_possible_cpus of them (workgroup_id_y indexes into it). + */ + if (offmap->map.map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) + n_instances =3D num_possible_cpus(); + + size =3D sizeof(struct knod_bpf_map_obj) + + (value_size * nents * n_instances); + if (offmap->map.map_type =3D=3D BPF_MAP_TYPE_HASH) + size +=3D sizeof(unsigned int) * nents; + order =3D get_order(size); + + mem =3D knod_alloc_mem(knod, PAGE_SIZE << order, flags); + if (IS_ERR(mem)) + return -ENOMEM; + + memset(mem->kaddr, 0, size); + knod_map =3D kzalloc_obj(struct knod_bpf_map, GFP_KERNEL); + if (!knod_map) { + knod_free_mem(knod, mem); + return -ENOMEM; + } + + knod_map->mem =3D mem; + knod_map->queue_mem =3D NULL; + knod_map->hash_elems_mem =3D NULL; + knod_map->offmap =3D offmap; + knod_map->priv =3D priv; + if (offmap->dev_priv) + WARN_ON_ONCE(1); + offmap->dev_priv =3D knod_map; + + knod_map_obj =3D (struct knod_bpf_map_obj *)mem->kaddr; + knod_map_obj->key_size =3D offmap->map.key_size; + if (knod_map_obj->key_size > MAX_MAP_KEY_SIZE) { + pr_warn("request key size is %d, but max key size is %d\n", + knod_map_obj->key_size, MAX_MAP_KEY_SIZE); + return -ENOMEM; + } + knod_map_obj->value_size =3D offmap->map.value_size; + knod_map_obj->max_entries =3D nents; + knod_map_obj->id =3D offmap->map.id; + knod_map_obj->map_type =3D offmap->map.map_type; + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + knod_map_obj->meta.hmeta.n_buckets =3D nents; + if (offmap->map.map_flags & BPF_F_ZERO_SEED) + knod_map_obj->meta.hmeta.hashrnd =3D 0; + else + knod_map_obj->meta.hmeta.hashrnd =3D get_random_u32(); + } else { + knod_map_obj->meta.ameta.per_instance_size =3D value_size * nents; + knod_map_obj->meta.ameta.n_instances =3D n_instances; + } + knod_map->knod_map_obj =3D knod_map_obj; + /* map->flags =3D ? */ + knod_jit_dbg(" map_id =3D %d\n", knod_map_obj->id); + + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + queue_size =3D sizeof(unsigned int) * nents; + queue_size =3D PAGE_SIZE << get_order(queue_size); + queue_mem =3D knod_alloc_mem(knod, queue_size, flags); + if (IS_ERR(queue_mem)) { + knod_free_mem(knod, mem); + kfree(knod_map); + return -ENOMEM; + } + + memset(queue_mem->kaddr, 0, queue_mem->size); + q =3D queue_mem->kaddr; + for (i =3D 0; i < knod_map_obj->meta.hmeta.n_buckets; i++) + q[i] =3D i; + knod_map->queue_mem =3D queue_mem; + knod_map_obj->meta.hmeta.q =3D (struct _queue *)queue_mem->gaddr; + + queue_size =3D (sizeof(struct knod_bpf_hash_elem_obj) + + roundup(knod_map_obj->key_size, 4) + + roundup(knod_map_obj->value_size, 4)) * + knod_map_obj->max_entries; + queue_size =3D PAGE_SIZE << get_order(queue_size); + + hash_elems_mem =3D knod_alloc_mem(knod, queue_size, flags); + if (IS_ERR(hash_elems_mem)) { + knod_free_mem(knod, queue_mem); + knod_free_mem(knod, mem); + kfree(knod_map); + return -ENOMEM; + } + + memset(hash_elems_mem->kaddr, 0, queue_size); + knod_map->hash_elems_mem =3D hash_elems_mem; + knod_map_obj->meta.hmeta.elems =3D (void *)hash_elems_mem->gaddr; + knod_bpf_map_hash_init_elem(knod_map, knod_map_obj); + + /* GC list for GPU-side delete: elem_ids pending unlink */ + gc_size =3D sizeof(unsigned int) * nents; + gc_size =3D PAGE_SIZE << get_order(gc_size); + gc_mem =3D knod_alloc_mem(knod, gc_size, flags); + if (IS_ERR(gc_mem)) { + knod_free_mem(knod, hash_elems_mem); + knod_free_mem(knod, queue_mem); + knod_free_mem(knod, mem); + kfree(knod_map); + return -ENOMEM; + } + memset(gc_mem->kaddr, 0, gc_size); + knod_map->gc_mem =3D gc_mem; + knod_map_obj->meta.hmeta.gc_count =3D 0; + knod_map_obj->meta.hmeta.gc_list =3D (void *)gc_mem->gaddr; + } + + err =3D __knod_map_mem(knod, mem); + if (err) { + pr_err("knod_bpf: failed to GPU-map map BO\n"); + goto err_map; + } + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + err =3D __knod_map_mem(knod, queue_mem); + if (err) { + pr_err("knod_bpf: failed to GPU-map queue BO\n"); + goto err_map; + } + err =3D __knod_map_mem(knod, hash_elems_mem); + if (err) { + pr_err("knod_bpf: failed to GPU-map hash_elems BO\n"); + goto err_map; + } + err =3D __knod_map_mem(knod, knod_map->gc_mem); + if (err) { + pr_err("knod_bpf: failed to GPU-map gc BO\n"); + goto err_map; + } + } + knod_bpf_gpu_mem_fence(priv); + + mutex_lock(&knodev->lock); + list_add(&knod_map->list, &knodev->accel->xdp.bound_maps); + mutex_unlock(&knodev->lock); + return 0; + +err_map: + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + knod_free_mem(knod, knod_map->gc_mem); + knod_free_mem(knod, hash_elems_mem); + knod_free_mem(knod, queue_mem); + } + knod_free_mem(knod, mem); + kfree(knod_map); + return err; +} + +static void knod_bpf_map_setup(struct bpf_prog *prog) +{ + struct knod_prog *knod_prog =3D prog->aux->offload->dev_priv; + struct knod_dev *knodev =3D knod_prog->knodev; + struct knod_bpf_map *knod_map; + struct knod_bpf_map_obj *map; + struct knod_mem *mem; + + mutex_lock(&knodev->lock); + list_for_each_entry(knod_map, &knodev->accel->xdp.bound_maps, list) { + mem =3D knod_map->mem; + map =3D mem->kaddr; + map->id =3D knod_map->offmap->map.id; + map->map_type =3D knod_map->offmap->map.map_type; + knod_jit_dbg(" id =3D %d type =3D %d\n", knod_map->offmap->map.id, + knod_map->offmap->map.map_type); + } + mutex_unlock(&knodev->lock); +} + +static struct knod_bpf_hash_elem_obj * +knod_bpf_map_hash_pop(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj) +{ + void *elems =3D knod_map->hash_elems_mem->kaddr; + unsigned int *queue =3D (unsigned int *)knod_map->queue_mem->kaddr; + struct knod_bpf_hash_elem_obj *e; + int elem_id; + + if (knod_map_obj->meta.hmeta.cur < 1) + return NULL; + + elem_id =3D queue[knod_map_obj->meta.hmeta.cur]; + knod_jit_dbg(" elem_id =3D 0x%x\n", elem_id); + e =3D elems + (elem_id * knod_map_obj->meta.hmeta.elem_size); + knod_map_obj->meta.hmeta.cur--; + e->next =3D KNOD_BPF_HASH_NEXT_END; + + return e; +} + +static struct knod_bpf_hash_elem_obj * +knod_bpf_map_hash_alloc_elem(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj, + void *key, void *value) +{ + struct knod_bpf_hash_elem_obj *e; + + e =3D knod_bpf_map_hash_pop(knod_map, knod_map_obj); + if (!e) + return NULL; + + unsafe_memcpy(knod_bpf_hash_elem_kv(e), key, knod_map_obj->key_size, + "knod hash elems are variable-sized GPU map records"); + unsafe_memcpy(knod_bpf_hash_elem_kv(e) + knod_map_obj->key_size, + value, knod_map_obj->value_size, + "knod hash elems are variable-sized GPU map records"); + /* VRAM is ioremap_wc - drain new elem's next and kv stores before + * the caller publishes a pointer to this elem. + */ + wmb(); + return e; +} + +static int knod_bpf_map_hash_lookup_elem(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj, + void *key, + void *value) +{ + void *elems =3D knod_map->hash_elems_mem->kaddr; + unsigned int hash, elem_id, elem_size; + struct knod_bpf_hash_elem_obj *e; + unsigned int *bucket; + + hash =3D jhash((const void *)key, knod_map_obj->key_size, + knod_map_obj->meta.hmeta.hashrnd); + knod_jit_dbg(" hash =3D %x\n", hash); + hash =3D hash & (knod_map_obj->meta.hmeta.n_buckets - 1); + knod_jit_dbg(" hash =3D %x\n", hash); + bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + + elem_id =3D bucket[hash]; + if (elem_id =3D=3D KNOD_BPF_HASH_NEXT_END) + return -ENOENT; + + elem_size =3D knod_map_obj->meta.hmeta.elem_size; + + e =3D elems + (elem_id * elem_size); + while (1) { + if (!(e->next & KNOD_BPF_HASH_NEXT_DELETED) && + !memcmp(&e->kv[0], (const unsigned char *)key, + knod_map_obj->key_size)) { + memcpy(value, + (unsigned char *)&e->kv[0] + + knod_map_obj->key_size, + knod_map_obj->value_size); + return 0; + } + unsigned int real_next =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + + if (real_next =3D=3D KNOD_BPF_HASH_NEXT_END) + return -ENOENT; + e =3D elems + (real_next * elem_size); + } + + return -ENOENT; +} + +static int knod_bpf_map_hash_update_elem(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj, + void *key, + void *value) +{ + void *elems =3D knod_map->hash_elems_mem->kaddr; + unsigned int hash, elem_id, elem_size; + struct knod_bpf_hash_elem_obj *e, *ne; + unsigned int *bucket; + + hash =3D jhash((const void *)key, knod_map_obj->key_size, + knod_map_obj->meta.hmeta.hashrnd); + hash =3D hash & (knod_map_obj->meta.hmeta.n_buckets - 1); + bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + + elem_size =3D knod_map_obj->meta.hmeta.elem_size; + elem_id =3D bucket[hash]; + if (elem_id =3D=3D KNOD_BPF_HASH_NEXT_END) { + ne =3D knod_bpf_map_hash_alloc_elem(knod_map, knod_map_obj, key, + value); + if (!ne) + return -ENOMEM; + bucket[hash] =3D ((void *)ne - (void *)elems) / elem_size; + return 0; + } + + e =3D elems + (elem_id * elem_size); + while (1) { + if (!(e->next & KNOD_BPF_HASH_NEXT_DELETED) && + !memcmp(&e->kv[0], (const unsigned char *)key, + knod_map_obj->key_size)) { + unsafe_memcpy(knod_bpf_hash_elem_kv(e) + + knod_map_obj->key_size, + value, knod_map_obj->value_size, + "knod hash elems are variable-sized GPU map records"); + return 0; + } + unsigned int real_next =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + + if (real_next =3D=3D KNOD_BPF_HASH_NEXT_END) { + ne =3D knod_bpf_map_hash_alloc_elem(knod_map, + knod_map_obj, + key, value); + if (!ne) + return -ENOMEM; + e->next =3D (e->next & KNOD_BPF_HASH_NEXT_DELETED) | + (((void *)ne - (void *)elems) / elem_size); + return 0; + } + e =3D elems + (real_next * elem_size); + } + + return -ENOENT; +} + +static int knod_bpf_map_hash_delete_elem(struct knod_bpf_map *knod_map, + struct knod_bpf_map_obj *knod_map_obj, + void *key) +{ + void *elems =3D knod_map->hash_elems_mem->kaddr; + unsigned int *queue =3D knod_map->queue_mem->kaddr; + unsigned int hash, elem_id, elem_size, cur; + struct knod_bpf_hash_elem_obj *e, *pe; + unsigned int *bucket; + + hash =3D jhash((const void *)key, knod_map_obj->key_size, + knod_map_obj->meta.hmeta.hashrnd); + hash =3D hash & (knod_map_obj->meta.hmeta.n_buckets - 1); + bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + + elem_id =3D bucket[hash]; + if (elem_id =3D=3D KNOD_BPF_HASH_NEXT_END) + return -ENOENT; + + elem_size =3D knod_map_obj->meta.hmeta.elem_size; + + e =3D elems + (elem_id * elem_size); + pe =3D e; + while (1) { + if (!(e->next & KNOD_BPF_HASH_NEXT_DELETED) && + !memcmp(&e->kv[0], (const unsigned char *)key, + knod_map_obj->key_size)) { + unsigned int e_next =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + unsigned int del_id =3D ((void *)e - elems) / elem_size; + + /* Unlink (GPU is paused - safe) */ + if (pe !=3D e) + pe->next =3D (pe->next & + KNOD_BPF_HASH_NEXT_DELETED) | + e_next; + else + bucket[hash] =3D e_next; + + e->next =3D KNOD_BPF_HASH_NEXT_END; + + /* Return elem to queue */ + cur =3D knod_map_obj->meta.hmeta.cur; + queue[cur] =3D del_id; + knod_map_obj->meta.hmeta.cur =3D cur + 1; + return 0; + } + unsigned int real_next =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + + if (real_next =3D=3D KNOD_BPF_HASH_NEXT_END) + return -ENOENT; + pe =3D e; + e =3D elems + (real_next * elem_size); + } + + return -ENOENT; +} + +static int knod_bpf_map_hash_get_first_key(struct bpf_offloaded_map *offma= p, + void *nkey) +{ + struct knod_bpf_map *knod_map =3D (struct knod_bpf_map *)offmap->dev_priv; + struct knod_bpf_map_obj *knod_map_obj; + unsigned int *bucket, elem_size, i; + struct knod_bpf_hash_elem_obj *e; + void *elems; + + knod_map_obj =3D knod_map->knod_map_obj; + bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + elems =3D knod_map->hash_elems_mem->kaddr; + + elem_size =3D knod_map_obj->meta.hmeta.elem_size; + + for (i =3D 0; i < knod_map_obj->meta.hmeta.n_buckets; i++) { + unsigned int eid; + + if (bucket[i] =3D=3D KNOD_BPF_HASH_NEXT_END) + continue; + eid =3D bucket[i]; + while (eid !=3D KNOD_BPF_HASH_NEXT_END) { + e =3D elems + (eid * elem_size); + if (!(e->next & KNOD_BPF_HASH_NEXT_DELETED)) { + unsafe_memcpy(nkey, knod_bpf_hash_elem_kv(e), + knod_map_obj->key_size, + "knod hash elems are variable-sized GPU map records"); + return 0; + } + eid =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + } + } + + return -ENOENT; +} + +static int knod_bpf_map_hash_get_next_key(struct bpf_offloaded_map *offmap, + void *key, void *nkey) +{ + struct knod_bpf_map *knod_map =3D (struct knod_bpf_map *)offmap->dev_priv; + struct knod_bpf_map_obj *knod_map_obj; + unsigned int *bucket, elem_size, i; + struct knod_bpf_hash_elem_obj *e; + bool found =3D false; + unsigned int hash; + void *elems; + + knod_map_obj =3D knod_map->knod_map_obj; + + bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + elems =3D knod_map->hash_elems_mem->kaddr; + + hash =3D jhash((const void *)key, knod_map_obj->key_size, + knod_map_obj->meta.hmeta.hashrnd); + hash =3D hash & (knod_map_obj->meta.hmeta.n_buckets - 1); + elem_size =3D knod_map_obj->meta.hmeta.elem_size; + + for (i =3D hash; i < knod_map_obj->meta.hmeta.n_buckets; i++) { + unsigned int eid; + + if (bucket[i] =3D=3D KNOD_BPF_HASH_NEXT_END) + continue; + + eid =3D bucket[i]; + while (eid !=3D KNOD_BPF_HASH_NEXT_END) { + e =3D elems + (eid * elem_size); + if (!(e->next & KNOD_BPF_HASH_NEXT_DELETED)) { + if (found && + memcmp(&e->kv[0], + (const unsigned char *)key, + knod_map_obj->key_size)) { + unsafe_memcpy(nkey, + knod_bpf_hash_elem_kv(e), + knod_map_obj->key_size, + "knod hash elems are variable-sized GPU map records"); + return 0; + } + if (!memcmp(&e->kv[0], + (const unsigned char *)key, + knod_map_obj->key_size)) + found =3D true; + } + eid =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + } + } + + return -ENOENT; +} + +static void knod_bpf_map_free(struct knod_dev *knodev, + struct bpf_offloaded_map *offmap) +{ + struct knod_bpf_map *knod_map =3D offmap->dev_priv; + struct knod_bpf_priv *priv =3D knodev->accel->xdp.priv; + + if (!knod_map) + return; + /* + * Defer the BO free: an in-flight prog dispatch may still reference + * this map's VRAM. Move it from bound_maps onto dead_maps under + * knodev->lock (the lock that guards the add); the worker reaps it + * from there after its next completion, by which point the in-flight + * dispatch on the old slot has retired (clean atomic flip). + */ + mutex_lock(&knodev->lock); + list_del(&knod_map->list); + list_add(&knod_map->list, &priv->dead_maps); + mutex_unlock(&knodev->lock); + offmap->dev_priv =3D NULL; +} + +static int __knod_bpf_map_lookup_elem(struct bpf_offloaded_map *offmap, + void *key, void *value) +{ + unsigned int idx =3D *(unsigned int *)key; + struct knod_bpf_map_obj *knod_map_obj; + struct knod_bpf_map *knod_map; + void *bucket; + u32 stride; + int i; + + knod_map =3D (struct knod_bpf_map *)offmap->dev_priv; + if (!knod_map || !knod_map->mem || !knod_map->mem->kaddr || + (knod_map->hash_elems_mem && !knod_map->hash_elems_mem->kaddr)) { + pr_err("knod_bpf: lookup on freed/invalid map (dev_priv=3D%p)\n", + offmap->dev_priv); + return -ENODEV; + } + knod_map_obj =3D knod_map->knod_map_obj; + + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_ARRAY) { + if (*(unsigned int *)key >=3D knod_map_obj->max_entries) + return -ENOENT; + bucket =3D knod_bpf_array_value_ptr(knod_map_obj, idx); + + unsafe_memcpy(value, bucket, knod_map_obj->value_size, + "knod array values live in a variable-sized GPU map tail"); + } else if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + if (idx >=3D knod_map_obj->max_entries) + return -ENOENT; + stride =3D round_up(knod_map_obj->value_size, 8); + bucket =3D &knod_map_obj->bucket[0]; + bucket +=3D (idx * knod_map_obj->value_size); + for (i =3D 0; i < knod_map_obj->meta.ameta.n_instances; i++) + unsafe_memcpy(value + i * stride, + bucket + i * + knod_map_obj->meta.ameta + .per_instance_size, + knod_map_obj->value_size, + "knod percpu array values live in a variable-sized GPU map tail"= ); + } else if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + return knod_bpf_map_hash_lookup_elem(knod_map, knod_map_obj, + key, value); + } + + return 0; +} + +static void knod_bpf_map_op_begin(struct knod_bpf_priv *priv) +{ + /* + * Serialize concurrent map ops but do NOT park the worker: stopping it + * mid-flight strands the in-flight dispatch and stalls the GPU compute + * queue. A map value updated while the GPU reads it may be seen torn, + * which is a transient inconsistency the BPF prog tolerates. + */ + mutex_lock(&priv->map_op_lock); +} + +static void knod_bpf_map_op_end(struct knod_bpf_priv *priv) +{ + knod_bpf_gpu_mem_fence(priv); + mutex_unlock(&priv->map_op_lock); +} + +static int __knod_bpf_map_update_elem(struct bpf_offloaded_map *offmap, + void *key, void *value, u64 flags) +{ + struct knod_bpf_map *knod_map =3D (struct knod_bpf_map *)offmap->dev_priv; + struct knod_bpf_map_obj *knod_map_obj; + struct knod_bpf_priv *priv; + unsigned int idx =3D *(unsigned int *)key; + struct knod_dev *knodev; + void *bucket; + u32 stride; + int i; + + if (!knod_map || !knod_map->mem || !knod_map->mem->kaddr) + return -ENODEV; + knod_map_obj =3D knod_map->knod_map_obj; + priv =3D knod_map->priv; + knodev =3D priv->knodev; + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_ARRAY) { + if (idx >=3D knod_map_obj->max_entries) + return -ENOENT; + + bucket =3D knod_bpf_array_value_ptr(knod_map_obj, idx); + unsafe_memcpy(bucket, value, knod_map_obj->value_size, + "knod array values live in a variable-sized GPU map tail"); + knod_bpf_gpu_mem_fence(priv); + return 0; + } else if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + if (idx >=3D knod_map_obj->max_entries) + return -ENOENT; + stride =3D round_up(knod_map_obj->value_size, 8); + bucket =3D &knod_map_obj->bucket[0]; + bucket +=3D (idx * knod_map_obj->value_size); + for (i =3D 0; i < knod_map_obj->meta.ameta.n_instances; i++) + unsafe_memcpy(bucket + i * + knod_map_obj->meta.ameta + .per_instance_size, + value + i * stride, + knod_map_obj->value_size, + "knod percpu array values live in a variable-sized GPU map tail"= ); + knod_bpf_gpu_mem_fence(priv); + return 0; + } else if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + int ret =3D -ENOENT; + + mutex_lock(&knodev->lock); + list_for_each_entry(knod_map, &knodev->accel->xdp.bound_maps, + list) { + if (knod_map->knod_map_obj =3D=3D knod_map_obj) { + mutex_unlock(&knodev->lock); + knod_bpf_map_op_begin(priv); + ret =3D knod_bpf_map_hash_update_elem(knod_map, + knod_map_obj, + key, value); + knod_bpf_map_op_end(priv); + return ret; + } + } + mutex_unlock(&knodev->lock); + } + + return -ENOENT; +} + +static int __knod_bpf_map_delete_elem(struct bpf_offloaded_map *offmap, + void *key) +{ + struct knod_bpf_map *knod_map =3D (struct knod_bpf_map *)offmap->dev_priv; + struct knod_bpf_map_obj *knod_map_obj; + struct knod_bpf_priv *priv; + int ret; + + if (!knod_map || !knod_map->mem || !knod_map->mem->kaddr) + return -ENODEV; + knod_map_obj =3D knod_map->knod_map_obj; + priv =3D knod_map->priv; + + if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_ARRAY || + knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) + return 0; + else if (knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) { + knod_bpf_map_op_begin(priv); + ret =3D knod_bpf_map_hash_delete_elem(knod_map, knod_map_obj, + key); + knod_bpf_map_op_end(priv); + return ret; + } + + return -ENOENT; +} + +static void knod_bpf_map_gc_process(struct knod_bpf_map *knod_map) +{ + struct knod_bpf_map_obj *knod_map_obj =3D knod_map->knod_map_obj; + unsigned int *gc_list =3D knod_map->gc_mem->kaddr; + unsigned int *queue =3D knod_map->queue_mem->kaddr; + void *elems =3D knod_map->hash_elems_mem->kaddr; + unsigned int *bucket =3D (unsigned int *)&knod_map_obj->bucket[0]; + unsigned int elem_size =3D knod_map_obj->meta.hmeta.elem_size; + unsigned int gc_count, cur, i; + + gc_count =3D READ_ONCE(knod_map_obj->meta.hmeta.gc_count); + if (!gc_count) + return; + + for (i =3D 0; i < gc_count; i++) { + unsigned int del_id =3D gc_list[i]; + struct knod_bpf_hash_elem_obj *del_elem =3D + elems + (del_id * elem_size); + unsigned int hash, eid; + struct knod_bpf_hash_elem_obj *e, *pe; + + hash =3D jhash(&del_elem->kv[0], knod_map_obj->key_size, + knod_map_obj->meta.hmeta.hashrnd); + hash =3D hash & (knod_map_obj->meta.hmeta.n_buckets - 1); + + eid =3D bucket[hash]; + pe =3D NULL; + while (eid !=3D KNOD_BPF_HASH_NEXT_END) { + e =3D elems + (eid * elem_size); + if (e =3D=3D del_elem) { + unsigned int next =3D e->next & + KNOD_BPF_HASH_NEXT_MASK; + if (pe) + pe->next =3D + (pe->next & + KNOD_BPF_HASH_NEXT_DELETED) | + next; + else + bucket[hash] =3D next; + + e->next =3D KNOD_BPF_HASH_NEXT_END; + + cur =3D knod_map_obj->meta.hmeta.cur; + queue[cur] =3D del_id; + knod_map_obj->meta.hmeta.cur =3D cur + 1; + break; + } + pe =3D e; + eid =3D e->next & KNOD_BPF_HASH_NEXT_MASK; + } + } + + WRITE_ONCE(knod_map_obj->meta.hmeta.gc_count, 0); +} + +/* + * Per-loop map maintenance, run from the worker loop head (outside any + * rcu_read_lock_bh, since knod_free_mem() may sleep). All bound_maps acc= ess + * is serialized under knodev->lock -- the same lock map_alloc/map_free us= e: + * GC live HASH maps, then reap maps that detach moved onto dead_maps. The + * worker only reaches here after completing the previous dispatch, so the + * clean atomic flip guarantees the GPU no longer reads a reaped map's BOs. + */ +#define KNOD_BPF_MAPS_TICK_INTERVAL 65536 + +static void knod_bpf_maps_tick(struct knod_bpf_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_map *knod_map, *tmp; + LIST_HEAD(reap); + + if (list_empty(&knodev->accel->xdp.bound_maps) && + list_empty(&priv->dead_maps)) + return; + + if (list_empty(&priv->dead_maps) && + (++priv->maps_tick_skip & (KNOD_BPF_MAPS_TICK_INTERVAL - 1))) + return; + + mutex_lock(&knodev->lock); + list_for_each_entry(knod_map, &knodev->accel->xdp.bound_maps, list) { + if (knod_map->knod_map_obj->map_type =3D=3D BPF_MAP_TYPE_HASH) + knod_bpf_map_gc_process(knod_map); + } + list_splice_init(&priv->dead_maps, &reap); + mutex_unlock(&knodev->lock); + + list_for_each_entry_safe(knod_map, tmp, &reap, list) { + if (knod_map->gc_mem) + knod_free_mem(priv->knod, knod_map->gc_mem); + if (knod_map->queue_mem) + knod_free_mem(priv->knod, knod_map->queue_mem); + if (knod_map->hash_elems_mem) + knod_free_mem(priv->knod, knod_map->hash_elems_mem); + if (knod_map->mem) + knod_free_mem(priv->knod, knod_map->mem); + kfree(knod_map); + } +} + +/* Completion mode: 0 =3D event (default, sleep on the AQL signal interrup= t), + * 1 =3D poll (busy-spin the signal value). Selectable via debugfs. + */ +static bool knod_bpf_poll_mode; + +/* Max spacing (microseconds) between dispatch-ahead submissions. Once a + * dispatch is in flight the worker waits up to this long before submitting + * the next so it batches the packets arriving meanwhile, letting inflight + * grow >=3D 2 without degenerating into one-packet dispatches. This is a + * ceiling only: an empty pipe submits at once to keep the GPU fed, and a + * completed dispatch is always retired without waiting. To actually build + * depth the value must be below the GPU execution time of a dispatch. + * 0 disables spacing (submit as soon as the ring has anything). + */ +static u32 knod_bpf_dispatch_delay_us =3D 20; + +static void knod_bpf_wait_event(struct knod_bpf_priv *priv) +{ + struct kfd_event_data events =3D { + .event_id =3D priv->knod->aql_event[0].id, + }; + u32 timeout_ms =3D knod_bpf_expire; + u32 wait_result; + + knod_wait_on_events(priv->knod->process, 1, &events, true, + &timeout_ms, &wait_result); +} + +static bool knod_bpf_submit_work(struct knod_bpf_priv *priv) +{ + struct knod_bpf_work_sq *sqw; + struct knod_bpf_stats *stats =3D &priv->stats; + ktime_t dispatch_start; + + if (priv->inflight_cnt >=3D KNOD_BPF_INFLIGHT) + return false; + + /* Pace dispatch-ahead so the next dispatch batches the packets that + * arrive during this window instead of firing one-packet dispatches. + * An empty pipe skips the wait so the GPU is never left idle. + */ + if (priv->inflight_cnt && + ktime_before(ktime_get(), priv->next_dispatch_time)) + return false; + + if (static_branch_unlikely(&knod_stats_key)) + dispatch_start =3D ktime_get(); + + sqw =3D knod_prepare_bpf(priv); + if (!sqw) + return false; + + if (static_branch_unlikely(&knod_stats_key)) { + u64 dns =3D ktime_to_ns(ktime_sub(ktime_get(), + dispatch_start)); + + stats->dispatch_total_ns +=3D dns; + stats->dispatch_count++; + if (dns > stats->dispatch_max_ns) + stats->dispatch_max_ns =3D dns; + } + + knod_submit_bpf(priv, sqw); + priv->inflight[priv->inflight_cnt++] =3D sqw; + priv->next_dispatch_time =3D + ktime_add_us(ktime_get(), + READ_ONCE(knod_bpf_dispatch_delay_us)); + return true; +} + +static void knod_bpf_record_completion(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct knod_bpf_stats *stats =3D &priv->stats; + u64 ns; + int bucket; + + if (!static_branch_unlikely(&knod_stats_key)) + return; + + ns =3D ktime_to_ns(ktime_sub(ktime_get(), sqw->dispatch_time)); + stats->completion_total_ns +=3D ns; + stats->completion_count++; + + if (ns > stats->completion_max_ns) + stats->completion_max_ns =3D ns; + + if (ns < 1000) + bucket =3D 0; + else + bucket =3D min(ilog2(ns / 1000) + 1, + KNOD_LAT_BUCKETS - 1); + stats->completion_hist[bucket]++; +} + +static bool knod_bpf_poll_complete(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + struct amd_signal *signal; + + if (!sqw) + return false; + + signal =3D (struct amd_signal *) + priv->knod->kaql[0].queue_signal->kaddr; + + if (sqw->sigval > READ_ONCE(signal->value)) { + knod_bpf_record_completion(priv, sqw); + return true; + } + + if (time_after(jiffies, sqw->expire)) { + pr_warn_ratelimited("knod_bpf: poll expire (sigval=3D%lld signal=3D%lld = expire_ms=3D%u)\n", + sqw->sigval, READ_ONCE(signal->value), knod_bpf_expire); + knod_bpf_record_completion(priv, sqw); + return true; + } + + return false; +} + +static void knod_bpf_schedule_pending_napi(struct knod_bpf_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + int qi; + + for (qi =3D 0; qi < priv->nr_works; qi++) { + if (spsc_pending(&knodev->wpriv[qi].spsc_bds) && + knodev->wpriv[qi].napi) + napi_schedule(knodev->wpriv[qi].napi); + } +} + +static int knod_bpf_worker(void *arg) +{ + struct knod_bpf_priv *priv =3D arg; + struct knod_bpf_work_sq *sqw; + bool progressed; + + while (!kthread_should_stop()) { + if (kthread_should_park()) { + knod_bpf_drain_worker(priv); + kthread_parkme(); + continue; + } + + knod_bpf_maps_tick(priv); + + progressed =3D false; + + rcu_read_lock_bh(); + /* Retire completed dispatches oldest-first: the signal is + * monotonic so inflight[0] finishes before inflight[1..]. + */ + while (priv->inflight_cnt && + knod_bpf_poll_complete(priv, priv->inflight[0])) { + sqw =3D priv->inflight[0]; + if (--priv->inflight_cnt) + memmove(priv->inflight, priv->inflight + 1, + priv->inflight_cnt * + sizeof(priv->inflight[0])); + priv->inflight[priv->inflight_cnt] =3D NULL; + knod_complete_acquire(priv, sqw); + knod_complete_napi(priv, sqw); + progressed =3D true; + } + + /* Keep the pipe full: dispatch ahead up to KNOD_BPF_INFLIGHT. + * Staging self-limits, so this stops once the ring is drained. + */ + while (knod_bpf_submit_work(priv)) + progressed =3D true; + rcu_read_unlock_bh(); + + if (!priv->inflight_cnt) { + knod_bpf_schedule_pending_napi(priv); + usleep_range(100, 200); + } else if (!progressed) { + /* Room to dispatch ahead but the pacing window has not + * opened yet: spin so the next submit fires on time and + * a completion is retired the instant it lands. Block + * on the event only when the pipe is full (nothing to + * submit) or spacing is disabled. + */ + if (priv->inflight_cnt < KNOD_BPF_INFLIGHT && + ktime_before(ktime_get(), priv->next_dispatch_time)) + cpu_relax(); + else if (knod_bpf_poll_mode) + cpu_relax(); + else + knod_bpf_wait_event(priv); + } + } + + return 0; +} + +static void knod_bpf_sq_init(struct knod_bpf_priv *priv) +{ + struct knod_bpf_work_sq *sqw; + int i; + + priv->worker_task =3D NULL; + priv->inflight_cnt =3D 0; + INIT_LIST_HEAD(&priv->free_list_sqw); + + for (i =3D 0; i < 32; i++) { + sqw =3D kvzalloc_obj(struct knod_bpf_work_sq, GFP_KERNEL); + if (!sqw) + continue; + + sqw->param =3D knod_alloc_mem(priv->knod, + sizeof(struct knod_bpf_param), + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (!sqw->param) { + kvfree(sqw); + continue; + } + memset(sqw->param->kaddr, 0, sizeof(struct knod_bpf_param)); + INIT_LIST_HEAD(&sqw->list); + list_add(&sqw->list, &priv->free_list_sqw); + sqw->backlogs =3D 0; + } +} + +static void knod_bpf_free_sqw(struct knod_bpf_priv *priv, + struct knod_bpf_work_sq *sqw) +{ + if (!sqw) + return; + + knod_free_mem(priv->knod, sqw->param); + kfree(sqw); +} + +static void knod_bpf_free_sqw_list(struct knod_bpf_priv *priv, + struct list_head *head) +{ + struct knod_bpf_work_sq *sqw, *tmp; + + list_for_each_entry_safe(sqw, tmp, head, list) { + list_del(&sqw->list); + knod_bpf_free_sqw(priv, sqw); + } +} + +static void knod_bpf_sq_exit(struct knod_bpf_priv *priv) +{ + if (!priv->knod) + return; + + knod_bpf_stop_worker(priv); + knod_bpf_drain(priv); + + knod_bpf_free_sqw_list(priv, &priv->free_list_sqw); + priv->inflight_cnt =3D 0; +} + +static void knod_priv_exit(struct knod_bpf_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_map *knod_map, *tmp; + LIST_HEAD(reap); + + knod_bpf_sq_exit(priv); + + /* + * The dispatch worker is not stopped until the next feature registers + * its own worker, so it may still be running knod_bpf_maps_tick() here. + * Serialize under knodev->lock and splice both lists to a local one: + * whichever side splices first frees them, the other sees them empty. + * Free outside the lock since knod_free_mem() may sleep. + */ + mutex_lock(&knodev->lock); + list_splice_init(&knodev->accel->xdp.bound_maps, &reap); + list_splice_init(&priv->dead_maps, &reap); + mutex_unlock(&knodev->lock); + + list_for_each_entry_safe(knod_map, tmp, &reap, list) { + if (knod_map->gc_mem) + knod_free_mem(priv->knod, knod_map->gc_mem); + if (knod_map->queue_mem) + knod_free_mem(priv->knod, knod_map->queue_mem); + if (knod_map->hash_elems_mem) + knod_free_mem(priv->knod, knod_map->hash_elems_mem); + if (knod_map->mem) + knod_free_mem(priv->knod, knod_map->mem); + kfree(knod_map); + } + + kfree(priv->prog_buf); + kfree(priv->pass_prog_buf); + if (priv->pass_knod_prog) { + knod_prog_free(priv->pass_knod_prog); + priv->pass_knod_prog =3D NULL; + } + /* kernels[] are owned by knod (freed in knod_release_ctx), not here */ + if (priv->pass_meta_buf) + knod_free_mem(priv->knod, priv->pass_meta_buf); +} + +static int knod_priv_init(struct knod_bpf_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + int pass_meta_buf_size; + int index; + + priv->prog =3D NULL; + mutex_init(&priv->map_op_lock); + INIT_LIST_HEAD(&priv->dead_maps); + priv->maps_tick_skip =3D 0; + + priv->nr_works =3D knod_bpf_active_rxq_count(knodev->netdev); + if (!priv->nr_works) { + pr_warn("knod_bpf: no active RX queues for %s\n", + knodev->netdev ? knodev->netdev->name : ""); + return -EINVAL; + } + + priv->prog_buf =3D kzalloc(KNOD_BPF_PROG_BUF_SIZE, GFP_KERNEL); + if (!priv->prog_buf) + return -ENOMEM; + + for (index =3D 0; index < priv->nr_works; index++) + priv->queue_base_gaddr[index] =3D priv->knod->buf[index]->gaddr; + + /* Per-queue PASS slot count; sizes the shader pass_meta_buf below. + * At most one PASS packet per dispatched slot, i.e. batch_size. + */ + priv->pass_pkts_per_queue =3D knod_bpf_batch_size(priv); + + /* Allocate GTT buffer for per-queue shader PASS copy */ + pass_meta_buf_size =3D priv->nr_works * priv->pass_pkts_per_queue * + KNOD_PASS_SLOT_SIZE; + priv->pass_meta_buf =3D knod_alloc_mem(priv->knod, pass_meta_buf_size, + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(priv->pass_meta_buf)) { + pr_warn("KNOD: failed to allocate pass_meta_buf\n"); + priv->pass_meta_buf =3D NULL; + knod_priv_exit(priv); + return -ENOMEM; + } + pr_debug("KNOD: pass_meta_buf gaddr=3D0x%llx..0x%llx size=3D%d nr_q=3D%d = pass_pkts_per_queue=3D%u\n", + priv->pass_meta_buf->gaddr, + priv->pass_meta_buf->gaddr + pass_meta_buf_size, + priv->pass_meta_buf->size, priv->nr_works, + priv->pass_pkts_per_queue); + + /* GPU->host delivery pages come from the framework per-queue page_pool + * (knodev->wpriv[q].pass_pool): the producer allocs from it and the + * NAPI drain recycles, so no per-feature delivery BO is allocated here. + */ + + knod_bpf_sq_init(priv); + + return 0; +} + +static struct knod_bpf_priv *__knod_accel_xdp_init(struct knod_accel *acce= l, + struct knod_dev *knodev) +{ + struct knod *knod =3D (struct knod *)knodev->accel->priv; + struct knod_bpf_priv *priv; + + priv =3D kzalloc_obj(struct knod_bpf_priv, GFP_KERNEL); + if (!priv) + return ERR_PTR(-ENOMEM); + + INIT_LIST_HEAD(&priv->list); + if (knod_bpf_workgroups % 64) { + knod_bpf_workgroups /=3D 64; + knod_bpf_workgroups++; + knod_bpf_workgroups *=3D 64; + } + + if (knod_bpf_workgroups < KNOD_BPF_WORKGROUPS_MIN || + knod_bpf_workgroups > KNOD_BPF_WORKGROUPS_MAX) + knod_bpf_workgroups =3D KNOD_BPF_WORKGROUPS_DEFAULT; + + if (knod_bpf_expire < KNOD_BPF_EXPIRE_MIN || + knod_bpf_expire > KNOD_BPF_EXPIRE_MAX) + knod_bpf_expire =3D KNOD_BPF_EXPIRE_DEFAULT; + pr_debug("workgroup size %d\n", knod_bpf_workgroups); + pr_debug("expire time =3D %dms\n", knod_bpf_expire); + pr_debug("packet cache =3D %d", knod_bpf_pkt_cache); + + INIT_LIST_HEAD(&accel->xdp.bound_maps); + accel->flags |=3D KNOD_FLAGS_XDP; + accel->xdp.priv =3D priv; + list_add(&priv->list, &priv_list); + + priv->knod =3D knod; + priv->accel =3D accel; + priv->knodev =3D knodev; + priv->dev =3D knodev->netdev; + + priv->isa_version =3D knod->isa_version; + + /* + * Only permanent per-attach state is set up here; the GPU compute + * buffers (knod_priv_init/kfd_kernel_init) are allocated by + * ->activate() when the BPF feature is selected. + */ + + return priv; +} + +/* Feature select: allocate the BPF GPU compute resources. */ +static int knod_bpf_activate(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv =3D accel->xdp.priv; + struct knod *knod =3D accel->priv; + + /* + * Pin the module while BPF is the selected feature: the core calls + * into these ops, so it must not be unloaded until feature->none. + * (No-op when built in - THIS_MODULE is NULL.) + */ + if (!try_module_get(THIS_MODULE)) + return -ENODEV; + + if (knod_priv_init(priv)) { + WARN_ON_ONCE(1); + module_put(THIS_MODULE); + return -EINVAL; + } + if (kfd_kernel_init(knod, priv)) { + knod_priv_exit(priv); + module_put(THIS_MODULE); + return -ENOMEM; + } + + priv->start =3D 0; + return 0; +} + +/* Feature deselect: free the BPF GPU compute resources. */ +static void knod_bpf_deactivate(struct knod_dev *knodev) +{ + struct knod_bpf_priv *priv =3D knodev->accel->xdp.priv; + + knod_priv_exit(priv); + module_put(THIS_MODULE); +} + +/* True while a user XDP prog or offloaded map is still bound to this acce= l. */ +static bool knod_bpf_busy(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv =3D accel->xdp.priv; + + if (!priv) + return false; + return READ_ONCE(priv->prog) || !list_empty(&accel->xdp.bound_maps); +} + +static void __knod_accel_xdp_exit(struct knod_accel *accel, + struct knod_bpf_priv *priv) +{ + /* GPU compute buffers are freed by ->deactivate(); free the rest. */ + memset(&accel->xdp, 0, sizeof(struct knod_accel_xdp)); + accel->flags &=3D ~KNOD_FLAGS_XDP; + list_del(&priv->list); + kfree(priv); +} + +static struct knod_insn_meta *knod_bpf_goto_meta(struct knod_prog *knod_pr= og, + struct knod_insn_meta *meta, + unsigned int insn_idx) +{ + unsigned int forward, backward, i; + + backward =3D meta->bpf_insn_idx - insn_idx; + forward =3D insn_idx - meta->bpf_insn_idx; + + if (min(forward, backward) > knod_prog->n_insns - insn_idx - 1) { + backward =3D knod_prog->n_insns - insn_idx - 1; + meta =3D knod_prog_last_meta(knod_prog); + } + if (min(forward, backward) > insn_idx && backward > insn_idx) { + forward =3D insn_idx; + meta =3D knod_prog_first_meta(knod_prog); + } + + if (forward < backward) + for (i =3D 0; i < forward; i++) + meta =3D knod_meta_next(meta); + else + for (i =3D 0; i < backward; i++) + meta =3D knod_meta_prev(meta); + + return meta; +} + +static int knod_bpf_check_stack_access(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + const struct bpf_reg_state *reg, + struct bpf_verifier_env *env) +{ + s32 old_off, new_off; + + if (reg->frameno !=3D env->cur_state->curframe) + meta->flags |=3D FLAG_INSN_PTR_CALLER_STACK_FRAME; + + if (!tnum_is_const(reg->var_off)) { + knod_jit_dbg(" variable ptr stack access\n"); + return -EINVAL; + } + + if (meta->ptr.type =3D=3D NOT_INIT) + return 0; + + old_off =3D meta->ptr.var_off.value; + new_off =3D reg->var_off.value; + + meta->ptr_not_const |=3D old_off !=3D new_off; + + if (!meta->ptr_not_const) + return 0; + + if (old_off % 4 =3D=3D new_off % 4) + return 0; + + knod_jit_dbg(" stack access changed location was:%d is:%d\n", + old_off, new_off); + return -EINVAL; +} + +static struct knod_insn_meta * +knod_bpf_lookup_prev_meta_by_dreg(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + int dreg_id) +{ + list_for_each_entry_continue_reverse(meta, &knod_prog->insns, l) { + if (!is_mbpf_alu(meta) && + !is_mbpf_load(meta) && + !is_mbpf_store(meta)) + continue; + if (meta->insn.dst_reg =3D=3D dreg_id) + return meta; + } + + return NULL; +} + +static int knod_bpf_check_ptr(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + struct bpf_verifier_env *env, u8 reg_no) +{ + const struct bpf_reg_state *reg =3D cur_regs(env) + reg_no; + int err; + + if (reg->type !=3D PTR_TO_CTX && + reg->type !=3D PTR_TO_STACK && + reg->type !=3D PTR_TO_MAP_VALUE && + reg->type !=3D PTR_TO_PACKET) { + knod_jit_dbg(" unsupported ptr type: %d\n", reg->type); + return -EINVAL; + } + + if (reg->type =3D=3D PTR_TO_STACK) { + err =3D knod_bpf_check_stack_access(knod_prog, meta, reg, env); + if (err) + return err; + } + + if (meta->ptr.type !=3D NOT_INIT && meta->ptr.type !=3D reg->type) { + knod_jit_dbg(" ptr type changed for instruction %d -> %d\n", + meta->ptr.type, + reg->type); + return -EINVAL; + } + + meta->ptr =3D *reg; + + return 0; +} + +static int knod_bpf_update_ptr_off(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + struct bpf_verifier_env *env) +{ + struct knod_bpf_reg_state *sreg =3D &meta->sreg; + struct knod_bpf_reg_state *dreg =3D &meta->dreg; + struct knod_insn_meta *prev_meta; + + if (is_mbpf_load(meta)) { + if (sreg->reg.type =3D=3D PTR_TO_PACKET || + sreg->reg.type =3D=3D PTR_TO_STACK) { + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.src_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + if (sreg->reg.type =3D=3D PTR_TO_PACKET) + sreg->packet_off =3D prev_meta->dreg.packet_off; + else + sreg->stack_off =3D prev_meta->dreg.stack_off; + } + } else if (is_mbpf_store(meta)) { + if (dreg->reg.type =3D=3D PTR_TO_PACKET || + dreg->reg.type =3D=3D PTR_TO_PACKET) { + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + if (dreg->reg.type =3D=3D PTR_TO_PACKET) + dreg->packet_off =3D prev_meta->dreg.packet_off; + else + dreg->stack_off =3D prev_meta->dreg.stack_off; + } + } + + return 0; +} + +static int knod_bpf_check_store(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + struct bpf_verifier_env *env) +{ + const struct bpf_reg_state *reg =3D cur_regs(env) + meta->insn.dst_reg; + + if (reg->type =3D=3D PTR_TO_CTX) { + if (knod_prog->type =3D=3D BPF_PROG_TYPE_XDP) { + /* XDP ctx accesses must be 4B in size */ + switch (meta->insn.off) { + case offsetof(struct xdp_md, rx_queue_index): + knod_jit_dbg(" queue selection not supported by FW\n"); + return -EOPNOTSUPP; + } + } + knod_jit_dbg(" unsupported store to context field\n"); + return -EOPNOTSUPP; + } + + return knod_bpf_check_ptr(knod_prog, meta, env, meta->insn.dst_reg); +} + +/* NOTE: + * knod_bpf_lookup_prev_meta_by_dreg(), src_reg vs dst_reg ???????/ + */ +static int knod_bpf_check_alu(struct knod_prog *knod_prog, + struct knod_insn_meta *meta, + struct bpf_verifier_env *env) +{ + const struct bpf_reg_state *sreg =3D cur_regs(env) + meta->insn.src_reg; + const struct bpf_reg_state *dreg =3D cur_regs(env) + meta->insn.dst_reg; + struct knod_bpf_reg_state *ksreg =3D &meta->sreg; + struct knod_bpf_reg_state *kdreg =3D &meta->dreg; + struct knod_insn_meta *prev_meta; + int imm; + + meta->umin_src =3D min(meta->umin_src, reg_umin(sreg)); + meta->umax_src =3D max(meta->umax_src, reg_umax(sreg)); + meta->umin_dst =3D min(meta->umin_dst, reg_umin(dreg)); + meta->umax_dst =3D max(meta->umax_dst, reg_umax(dreg)); + + /* AMDGPU doesn't have divide instructions, we support divide by + * constant through reciprocal multiplication. Given NFP support + * multiplication no bigger than u32, we'd require divisor and dividend + * no bigger than that as well. + * + * Also eBPF doesn't support signed divide and has enforced this on C + * language level by failing compilation. However LLVM assembler hasn't + * enforced this, so it is possible for negative constant to leak in as + * a BPF_K operand through assembly code, we reject such cases as well. + */ + if (is_mbpf_div(meta)) { + if (meta->umax_dst > U32_MAX) { + knod_jit_dbg(" dividend is not within u32 value range\n"); + return -EINVAL; + } + if (mbpf_src(meta) =3D=3D BPF_X) { + if (meta->umin_src !=3D meta->umax_src) { + knod_jit_dbg(" divisor is not constant\n"); + return -EINVAL; + } + if (meta->umax_src > U32_MAX) { + knod_jit_dbg(" divisor is not within u32 value range\n"); + return -EINVAL; + } + } + if (mbpf_src(meta) =3D=3D BPF_K && meta->insn.imm < 0) { + knod_jit_dbg(" divide by negative constant is not supported\n"); + return -EINVAL; + } + } + + if (dreg->type =3D=3D PTR_TO_STACK) { + imm =3D meta->insn.imm; + + switch (meta->insn.code) { + /* ALU + * If a destination register contains a pointer of STACK, + * offset should not be minus. + */ + case BPF_ALU | BPF_MOV | BPF_X: + case BPF_ALU64 | BPF_MOV | BPF_X: + //r[d] =3D r[s]; + kdreg->stack_off =3D ksreg->stack_off; + break; + case BPF_ALU | BPF_MOV | BPF_K: + case BPF_ALU64 | BPF_MOV | BPF_K: + //r[d] =3D imm; + kdreg->stack_off =3D ksreg->stack_off; + break; + case BPF_ALU | BPF_XOR | BPF_X: + case BPF_ALU64 | BPF_XOR | BPF_X: + //r[d] ^=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_XOR | BPF_K: + case BPF_ALU64 | BPF_XOR | BPF_K: + //r[d] ^=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off ^ imm; + break; + case BPF_ALU | BPF_MOD | BPF_X: + case BPF_ALU64 | BPF_MOD | BPF_X: + //r[d] %=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_MOD | BPF_K: + case BPF_ALU64 | BPF_MOD | BPF_K: + //r[d] %=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off % imm; + break; + case BPF_ALU | BPF_AND | BPF_X: + case BPF_ALU64 | BPF_AND | BPF_X: + //r[d] &=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_AND | BPF_K: + case BPF_ALU64 | BPF_AND | BPF_K: + //r[d] &=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off & imm; + break; + case BPF_ALU | BPF_OR | BPF_X: + case BPF_ALU64 | BPF_OR | BPF_X: + //r[d] |=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_OR | BPF_K: + case BPF_ALU64 | BPF_OR | BPF_K: + //r[d] |=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off | imm; + break; + case BPF_ALU | BPF_ADD | BPF_X: + case BPF_ALU64 | BPF_ADD | BPF_X: + //r[d] +=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_ADD | BPF_K: + case BPF_ALU64 | BPF_ADD | BPF_K: + //r[d] +=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off + imm; + break; + case BPF_ALU | BPF_SUB | BPF_X: + case BPF_ALU64 | BPF_SUB | BPF_X: + //r[d] -=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_SUB | BPF_K: + case BPF_ALU64 | BPF_SUB | BPF_K: + //r[d] -=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off - imm; + break; + case BPF_ALU | BPF_MUL | BPF_X: + case BPF_ALU64 | BPF_MUL | BPF_X: + //r[d] *=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_MUL | BPF_K: + case BPF_ALU64 | BPF_MUL | BPF_K: + //r[d] *=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off * imm; + break; + case BPF_ALU | BPF_DIV | BPF_X: + case BPF_ALU64 | BPF_DIV | BPF_X: + //r[d] /=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_DIV | BPF_K: + case BPF_ALU64 | BPF_DIV | BPF_K: + //r[d] /=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off / imm; + break; + case BPF_ALU | BPF_NEG: + case BPF_ALU64 | BPF_NEG: + //r[d] =3D -r[d]; + break; + case BPF_ALU | BPF_LSH | BPF_X: + case BPF_ALU64 | BPF_LSH | BPF_X: + //r[d] <<=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_LSH | BPF_K: + case BPF_ALU64 | BPF_LSH | BPF_K: + //r[d] <<=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off << imm; + break; + case BPF_ALU | BPF_RSH | BPF_X: + case BPF_ALU64 | BPF_RSH | BPF_X: + //r[d] >>=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_RSH | BPF_K: + case BPF_ALU64 | BPF_RSH | BPF_K: + //r[d] >>=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off >> imm; + break; + case BPF_ALU | BPF_ARSH | BPF_X: + case BPF_ALU64 | BPF_ARSH | BPF_X: + //r[d] >>=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_ARSH | BPF_K: + case BPF_ALU64 | BPF_ARSH | BPF_K: + //r[d] >>=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->stack_off =3D prev_meta->dreg.stack_off >> imm; + break; + } + knod_jit_dbg(" %d: dreg->stack_off =3D %d\n", meta->bpf_insn_idx, + kdreg->stack_off); + } + + if (dreg->type =3D=3D PTR_TO_PACKET) { + imm =3D meta->insn.imm; + + switch (meta->insn.code) { + /* ALU + * If a destination register contains a pointer of STACK, + * offset should not be minus. + */ + case BPF_ALU | BPF_MOV | BPF_X: + case BPF_ALU64 | BPF_MOV | BPF_X: + //r[d] =3D r[s]; + kdreg->packet_off =3D ksreg->packet_off; + break; + case BPF_ALU | BPF_MOV | BPF_K: + case BPF_ALU64 | BPF_MOV | BPF_K: + //r[d] =3D imm; + kdreg->packet_off =3D ksreg->packet_off; + break; + case BPF_ALU | BPF_XOR | BPF_X: + case BPF_ALU64 | BPF_XOR | BPF_X: + //r[d] ^=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_XOR | BPF_K: + case BPF_ALU64 | BPF_XOR | BPF_K: + //r[d] ^=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off ^ imm; + break; + case BPF_ALU | BPF_MOD | BPF_X: + case BPF_ALU64 | BPF_MOD | BPF_X: + //r[d] %=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_MOD | BPF_K: + case BPF_ALU64 | BPF_MOD | BPF_K: + //r[d] %=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off % imm; + break; + case BPF_ALU | BPF_AND | BPF_X: + case BPF_ALU64 | BPF_AND | BPF_X: + //r[d] &=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_AND | BPF_K: + case BPF_ALU64 | BPF_AND | BPF_K: + //r[d] &=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off & imm; + break; + case BPF_ALU | BPF_OR | BPF_X: + case BPF_ALU64 | BPF_OR | BPF_X: + //r[d] |=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_OR | BPF_K: + case BPF_ALU64 | BPF_OR | BPF_K: + //r[d] |=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off | imm; + break; + case BPF_ALU | BPF_ADD | BPF_X: + case BPF_ALU64 | BPF_ADD | BPF_X: + //r[d] +=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_ADD | BPF_K: + case BPF_ALU64 | BPF_ADD | BPF_K: + //r[d] +=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off + imm; + break; + case BPF_ALU | BPF_SUB | BPF_X: + case BPF_ALU64 | BPF_SUB | BPF_X: + //r[d] -=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_SUB | BPF_K: + case BPF_ALU64 | BPF_SUB | BPF_K: + //r[d] -=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off - imm; + break; + case BPF_ALU | BPF_MUL | BPF_X: + case BPF_ALU64 | BPF_MUL | BPF_X: + //r[d] *=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_MUL | BPF_K: + case BPF_ALU64 | BPF_MUL | BPF_K: + //r[d] *=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off * imm; + break; + case BPF_ALU | BPF_DIV | BPF_X: + case BPF_ALU64 | BPF_DIV | BPF_X: + //r[d] /=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_DIV | BPF_K: + case BPF_ALU64 | BPF_DIV | BPF_K: + //r[d] /=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off / imm; + break; + case BPF_ALU | BPF_NEG: + case BPF_ALU64 | BPF_NEG: + //r[d] =3D -r[d]; + break; + case BPF_ALU | BPF_LSH | BPF_X: + case BPF_ALU64 | BPF_LSH | BPF_X: + //r[d] <<=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_LSH | BPF_K: + case BPF_ALU64 | BPF_LSH | BPF_K: + //r[d] <<=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off << imm; + break; + case BPF_ALU | BPF_RSH | BPF_X: + case BPF_ALU64 | BPF_RSH | BPF_X: + //r[d] >>=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_RSH | BPF_K: + case BPF_ALU64 | BPF_RSH | BPF_K: + //r[d] >>=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off >> imm; + break; + case BPF_ALU | BPF_ARSH | BPF_X: + case BPF_ALU64 | BPF_ARSH | BPF_X: + //r[d] >>=3D r[s]; + knod_jit_dbg(" PTR_TO_STACK with BPF_X is not supported\n"); + return -EINVAL; + case BPF_ALU | BPF_ARSH | BPF_K: + case BPF_ALU64 | BPF_ARSH | BPF_K: + //r[d] >>=3D imm; + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg( + knod_prog, meta, meta->insn.dst_reg); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + return -EINVAL; + } + kdreg->packet_off =3D prev_meta->dreg.packet_off >> imm; + break; + } + knod_jit_dbg(" %d: dreg->packet_off =3D %d\n", meta->bpf_insn_idx, + kdreg->packet_off); + } + return 0; +} + +static int knod_bpf_verify_insn(struct bpf_verifier_env *env, + int insn_idx, int prev_insn) +{ + struct knod_prog *knod_prog =3D env->prog->aux->offload->dev_priv; + const struct bpf_reg_state *sreg, *dreg, *kreg, *vreg; + struct knod_insn_meta *meta =3D knod_prog->meta; + struct knod_insn_meta *prev_meta; + int err =3D 0; + + meta =3D knod_bpf_goto_meta(knod_prog, meta, insn_idx); + sreg =3D cur_regs(env) + meta->insn.src_reg; + dreg =3D cur_regs(env) + meta->insn.dst_reg; + knod_prog->meta =3D meta; + meta->sreg.reg =3D *sreg; + meta->dreg.reg =3D *dreg; + + knod_bpf_update_ptr_off(knod_prog, meta, env); + + if (meta->insn.src_reg >=3D MAX_BPF_REG || + meta->insn.dst_reg >=3D MAX_BPF_REG) { + knod_jit_dbg(" program uses extended registers - jit hardening?\n"); + err =3D -EINVAL; + goto out; + } + + if (is_mbpf_load(meta)) { + err =3D knod_bpf_check_ptr(knod_prog, meta, env, + meta->insn.src_reg); + goto out; + } + if (is_mbpf_store(meta)) { + err =3D knod_bpf_check_store(knod_prog, meta, env); + goto out; + } + + if (is_mbpf_map_call(meta)) { + kreg =3D cur_regs(env) + 2; + meta->kreg.reg =3D *kreg; + + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg(knod_prog, + meta, + 2); + if (!prev_meta) { + knod_jit_dbg(" Invalid\n"); + err =3D -EINVAL; + goto out; + } + if (kreg->type =3D=3D PTR_TO_PACKET) + meta->kreg.packet_off =3D prev_meta->dreg.packet_off; + else + meta->kreg.stack_off =3D prev_meta->dreg.stack_off; + if (knod_prog->max_stack_off > meta->kreg.stack_off) + knod_prog->max_stack_off =3D meta->kreg.stack_off; + if (knod_prog->max_packet_off < meta->kreg.packet_off) + knod_prog->max_packet_off =3D meta->kreg.packet_off; + + /* bpf_map_update_elem: track r3 (value pointer) */ + if (meta->insn.imm =3D=3D 2) { + vreg =3D cur_regs(env) + 3; + meta->vreg.reg =3D *vreg; + + prev_meta =3D knod_bpf_lookup_prev_meta_by_dreg(knod_prog, + meta, + 3); + if (!prev_meta) { + knod_jit_dbg(" Invalid vreg\n"); + err =3D -EINVAL; + goto out; + } + if (vreg->type =3D=3D PTR_TO_PACKET) + meta->vreg.packet_off =3D + prev_meta->dreg.packet_off; + else + meta->vreg.stack_off =3D + prev_meta->dreg.stack_off; + if (knod_prog->max_stack_off > meta->vreg.stack_off) + knod_prog->max_stack_off =3D meta->vreg.stack_off; + if (knod_prog->max_packet_off < meta->vreg.packet_off) + knod_prog->max_packet_off =3D + meta->vreg.packet_off; + } + } + + if (is_mbpf_alu(meta)) + err =3D knod_bpf_check_alu(knod_prog, meta, env); + + /* less stack offset is bigger */ + if (knod_prog->max_stack_off > meta->sreg.stack_off) + knod_prog->max_stack_off =3D meta->sreg.stack_off; + if (knod_prog->max_stack_off > meta->dreg.stack_off) + knod_prog->max_stack_off =3D meta->dreg.stack_off; + if (knod_prog->max_packet_off < meta->sreg.packet_off) + knod_prog->max_packet_off =3D meta->sreg.packet_off; + if (knod_prog->max_packet_off < meta->dreg.packet_off) + knod_prog->max_packet_off =3D meta->dreg.packet_off; + +out: + if (err) + pr_warn("knod_bpf: verifier rejected bpf insn %d (code 0x%02x off %d imm= %d): %d\n", + insn_idx, meta->insn.code, meta->insn.off, + meta->insn.imm, err); + return err; +} + +static int knod_bpf_finalize(struct bpf_verifier_env *env) +{ + return 0; +} + +static int knod_bpf_offload(struct knod_dev *knodev, + struct bpf_prog *prog, bool oldprog) +{ + struct knod_bpf_priv *priv =3D knodev->accel->xdp.priv; + + WARN(!!knod_dev_offloaded(knodev) !=3D oldprog, + "bad offload state, expected offload %sto be active", + oldprog ? "" : "not "); + + WRITE_ONCE(priv->prog, prog); + knod_dev_offload(knodev, prog); + + /* + * Uninstalling the prog: reload the pass kernel now, while the prog's + * maps are still valid, so the worker stops dispatching prog code that + * is about to reference freed maps. + */ + if (!prog) + knod_bpf_reload_pass(knodev); + + return 0; +} + +static int knod_bpf_xdp_offload_prog(struct knod_dev *knodev, + struct netdev_bpf *bpf) +{ + if (!knod_dev_active(knodev) && !bpf->prog) + return 0; + + if (!knod_dev_active(knodev) && bpf->prog && + knodev->accel->xdp.bpf_offloaded) { + return -EBUSY; + } + + return knod_bpf_offload(knodev, bpf->prog, knod_dev_active(knodev)); +} + +static int knod_bpf_xdp_set_prog(struct knod_dev *knodev, + struct netdev_bpf *bpf) +{ + int err; + + if (bpf->command =3D=3D XDP_SETUP_PROG_HW) { + err =3D knod_bpf_xdp_offload_prog(knodev, bpf); + if (err) + return err; + } + + xdp_attachment_setup(&knodev->accel->xdp.xdp_hw, bpf); + + return 0; +} + +static void knod_wait_vmcnt(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + knod_emit(priv, meta, s_waitcnt_vmcnt); +} + +static void knod_global_load_size_cache(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 *d, + struct amdgcn_param32 s, + int dst_idx, int start_off, int length) +{ + int off =3D start_off; + + /* length is 4B aligned */ + while (length) { + if (length >=3D 16) { + knod_emit(priv, meta, global_load_dwordx4, d[dst_idx], + s, off); + length -=3D 16; + off +=3D 16; + dst_idx +=3D 4; + } else if (length >=3D 8) { + knod_emit(priv, meta, global_load_dwordx2, d[dst_idx], + s, off); + length -=3D 8; + off +=3D 8; + dst_idx +=3D 2; + } else if (length >=3D 4) { + knod_emit(priv, meta, global_load_dword, d[dst_idx], + s, off); + length -=3D 4; + off +=3D 4; + dst_idx +=3D 1; + } + } + + knod_wait_vmcnt(priv, meta); +} + +static void knod_global_store_size_cache(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 *d, + struct amdgcn_param32 s, + int dst_idx, int start_off, int length) +{ + int off =3D start_off; + + /* length is 4B aligned */ + while (length) { + if (length >=3D 16) { + knod_emit(priv, meta, global_store_dwordx4, d[dst_idx], + s, off); + length -=3D 16; + off +=3D 16; + dst_idx +=3D 4; + } else if (length >=3D 8) { + knod_emit(priv, meta, global_store_dwordx2, d[dst_idx], + s, off); + length -=3D 8; + off +=3D 8; + dst_idx +=3D 2; + } else if (length >=3D 4) { + knod_emit(priv, meta, global_store_dword, d[dst_idx], + s, off); + length -=3D 4; + off +=3D 4; + dst_idx +=3D 1; + } + } + + knod_wait_vmcnt(priv, meta); +} + +static int knod_prog_prepare_insns(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog) +{ + struct amdgcn_param64 param64[3]; + struct amdgcn_param32 param[10]; + struct knod_insn_meta *meta; + int bs_shift; + + meta =3D kzalloc_obj(*meta, GFP_KERNEL); + if (!meta) + return -ENOMEM; + + meta->amdgpu_insn_idx =3D 0; + + /* Invalidate SQC instruction cache so that a re-uploaded shader + * at the same VRAM address is fetched from memory, not from the + * stale I-cache. Must be the very first instruction at the entry + * point so that every shader version has s_icache_inv at the same + * offset - the cached old version executes s_icache_inv too, + * which flushes the cache before divergent code is reached. + */ + knod_emit(priv, meta, s_icache_inv); + knod_emit(priv, meta, s_waitcnt_vmcnt_lgkmcnt); + + knod_sset32(¶m[0], KNOD_AMDGPU_PARAM_SREG_LO); + knod_sset32(¶m[1], KNOD_AMDGPU_ARG_SREG); + /* param =3D (__global struct _knod_bpf_param *)pkt.kernarg_address; */ + knod_emit(priv, meta, s_load_dwordx2, param[0], param[1], + offsetof(struct hsa_kernel_dispatch_packet, kernarg_address)); + + knod_vset32(¶m[0], KNOD_AMDGPU_IDX_VREG); + knod_vset32(¶m[1], KNOD_AMDGPU_VREG0_LO); + knod_iset32(¶m[2], 0); + /* 10bits, lidx can up to 1024, Do not edit */ + knod_iset32(¶m[3], 10); + /* extract workitem ID to reserved vgpr register. + * In the 2D-dispatch layout, IDX_VREG holds the per-workgroup tid + * (0..workgroup_size_x-1). queue_id =3D workgroup_id_y (s15). The + * flat index (queue_id * batch_size + local_idx) is computed later, + * after queue_desc has been loaded and the v_cmpx bounds check has + * narrowed EXEC to lanes with local_idx < count. + */ + knod_emit(priv, meta, v_bfe_i32, param[0], param[1], param[2], + param[3]); + if (priv->batch_size > knod_bpf_workgroups) { + /* local_idx =3D workgroup_id_x * workgroup_size_x + * + workitem_id. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG5_LO); + knod_sset32(¶m[1], KNOD_AMDGPU_WORKGROUP_ID_X_SREG); + knod_iset32(¶m[2], knod_bpf_workgroups); + knod_emit(priv, meta, + v_mul_lo_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_IDX_VREG); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG5_LO); + knod_vset32(¶m[2], KNOD_AMDGPU_IDX_VREG); + knod_emit(priv, meta, v_add_u32, param[0], param[1], param[2]); + } + /* set frame pointer to 0 */ + knod_sset32(¶m[0], KNOD_AMDGPU_FRAME_POINTER_SREG); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, s_mov_b32, param[0], param[1]); + /* wait for s_load_dwordx2 (kernarg_address) */ + knod_emit(priv, meta, s_waitcnt_vmcnt_lgkmcnt); + + /* load {nr_backlogs, _pad} from param (offset 0, 8-byte aligned) */ + knod_sset32(¶m[0], KNOD_AMDGPU_TMP_SREG1_LO); + knod_sset32(¶m[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, s_load_dwordx2, param[0], param[1], 0); + knod_emit(priv, meta, s_waitcnt_vmcnt_lgkmcnt); + + /* NOTE: the bounds check `EXEC &=3D (tid < count)` is deferred until + * after the queue descriptor load (queue<->workgroup binding). + * nr_backlogs is no longer the right upper bound because lanes with + * tid > this queue's count must be masked, not just the ones past + * the aggregate backlog total. + */ + + /* Initialize done_mask to 0 for structurized CFG */ + knod_emit(priv, meta, s_mov_b64, knod_prog->done_mask_sreg, + AMDGCN_SREG_INTEGER_0); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D + * Queue-descriptor prep (2D dispatch: queue_id =3D workgroup_id_y) + * + * Loads this WG's queue descriptor from VRAM, performs the + * per-lane bounds check (local_idx < queues[queue_id].count), and + * converts IDX_VREG from local_idx to flat_IDX (queue_id * + * batch_size + local_idx) which the rest of the prologue/BPF body + * expects. TMP_VREG9_LO is repurposed to hold the saved local + * tid for use as local_idx in the slot-address step. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + bs_shift =3D ilog2(priv->batch_size); + + /* a. queue_id =3D workgroup_id_y (broadcast scalar to VGPR LO). */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_sset32(¶m[1], KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + /* b. Copy queue_id into TMP_VREG5_LO - separate scratch used as + * the v_mad src-multiplicand. Avoids dst/src1 overlap on the + * following v_mad_u64_u32 (dst=3DTMP_VREG0_LO:HI). + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG5_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + /* c. TMP_VREG0 =3D PARAM + queue_id * sizeof(queue_desc). */ + knod_vset64(¶m64[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_sset32(¶m[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_iset32(¶m[1], + sizeof(struct knod_bpf_queue_desc)); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG5_LO); + knod_sset64(¶m64[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, v_mad_u64_u32, param64[0], param[0], + param[1], param[2], param64[1]); + + /* d. TMP_VREG0 +=3D offsetof(queues). */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], + offsetof(struct knod_bpf_param, queues)); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[0]); + + /* e. Load pool_gaddr + base_gaddr (offset 0, 16 bytes) into + * TMP_VREG1_LO..TMP_VREG2_HI (v24..v27 - must be consecutive). + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG1_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, global_load_dwordx4, param[0], param[1], 0); + + /* f. Load count + _pad + ring_start + ring_mask (offset 16, 16 + * bytes) into TMP_VREG3_LO..TMP_VREG4_HI. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG3_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, global_load_dwordx4, param[0], param[1], + offsetof(struct knod_bpf_queue_desc, count)); + knod_emit(priv, meta, s_waitcnt_vmcnt); + + /* g. Bounds check: EXEC &=3D (workitem_id < count). */ + knod_vset32(¶m[0], KNOD_AMDGPU_IDX_VREG); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG3_LO); + knod_emit(priv, meta, v_cmpx_lt_u32, param[0], param[1]); + + /* Snapshot the in-bounds lane mask. The unified epilogue uses this + * to publish one verdict for every lane the dispatch claimed, even if + * a malformed or newly added CFG path fails to join done_mask. + */ + knod_emit(priv, meta, s_mov_b64, knod_prog->initial_exec_sreg, + AMDGCN_SREG_EXEC_LO); + + /* h. Save per-queue local_idx to TMP_VREG9_LO. + * The slot-address step consumes this value; keeping it in a + * dedicated VGPR lets us overwrite IDX_VREG with flat_IDX for + * the CTX address computation that immediately follows. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_IDX_VREG); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + /* i. IDX_VREG =3D (queue_id << ilog2(batch_size)) + local_idx + * -> flat_IDX into the sub[] / sqw->bds[] arrays, matching the + * CPU-side layout `sqw->bds[queue_id * batch_size + local_idx]`. + * batch_size is rounded down to a power of two at start so the + * shift is exact. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_IDX_VREG); + knod_sset32(¶m[1], KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_iset32(¶m[2], bs_shift); + knod_vset32(¶m[3], KNOD_AMDGPU_IDX_VREG); + knod_emit(priv, meta, v_lshl_add_u32, param[0], param[1], + param[2], param[3]); + + /* ctx =3D ¶m->sub[flat_IDX].ctx; + * v_mad: VREG1 =3D sizeof(sub_obj) * flat_IDX + PARAM_SREG + * then add offsetof(sub) =3D 8 to account for nr_backlogs/_pad + */ + knod_vset64(¶m64[0], KNOD_AMDGPU_VREG1_LO); + knod_sset32(¶m[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_iset32(¶m[1], sizeof(struct knod_bpf_subparam_obj)); + knod_vset32(¶m[2], KNOD_AMDGPU_IDX_VREG); + knod_sset64(¶m64[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, v_mad_u64_u32, param64[0], param[0], + param[1], param[2], param64[1]); + /* + offsetof(struct knod_bpf_param, sub) */ + knod_vset32(¶m[0], KNOD_AMDGPU_VREG1_LO); + knod_iset32(¶m[1], offsetof(struct knod_bpf_param, sub)); + knod_vset32(¶m[2], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_VREG1_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[0]); + knod_vset32(¶m[0], KNOD_AMDGPU_CTX_VREG_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_CTX_VREG_HI); + knod_vset32(¶m[1], KNOD_AMDGPU_VREG1_HI); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_FRAME_POINTER_VREG_LO); + knod_iset32(¶m[1], 0x200); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_FRAME_POINTER_VREG_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + /* 5. slot =3D (ring_start + local_idx) & ring_mask. + * local_idx =3D saved per-queue local_idx in TMP_VREG9_LO. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG5_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG4_LO); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, meta, v_add_u32, param[0], param[1], param[2]); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG4_HI); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG5_LO); + knod_emit(priv, meta, v_and_b32_e32, param[0], param[1], param[2]); + + /* Save backlog index before step 6 overwrites IDX_VREG -> SLOT_VREG. + * v_mov_b32 BACKLOG_IDX_VREG(v58), IDX_VREG(v62) + * Used in epilogue for XDP_PASS pass_indices[] write. + */ + knod_vset32(¶m[0], KNOD_AMDGPU_BACKLOG_IDX_VREG); + knod_vset32(¶m[1], KNOD_AMDGPU_IDX_VREG); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + /* 6. slot_addr =3D pool_gaddr + slot * spsc_stride, where + * spsc_stride =3D ALIGN(sizeof(spsc_bd), SMP_CACHE_BYTES) + * Compute directly into SLOT_VREG (v62:v63). + */ + knod_vset32(¶m[0], KNOD_AMDGPU_SLOT_VREG_LO); + knod_iset32(¶m[1], + ilog2(ALIGN(sizeof(struct spsc_bd), SMP_CACHE_BYTES))); + knod_emit(priv, meta, v_lshlrev_b32, param[0], param[1], param[2]); + /* slot_addr =3D pool_gaddr + slot_offset */ + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG1_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_SLOT_VREG_HI); + knod_iset32(¶m[1], 0); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG1_HI); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[2]); + + /* 7. Load spsc_bd: {off(u16)|len(u16), page_idx} via single dwordx2 + * TMP_VREG6_LO (v34) =3D off|len, TMP_VREG6_HI (v35) =3D page_idx + */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG6_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_SLOT_VREG_LO); + knod_emit(priv, meta, global_load_dwordx2, param[0], param[1], + offsetof(struct spsc_bd, off)); + knod_emit(priv, meta, s_waitcnt_vmcnt); + + /* 8. data =3D base_gaddr + (page_idx << PAGE_SHIFT) + off + * Compute directly into DATA_VREG (v64:v65). + */ + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_LO); + knod_iset32(¶m[1], PAGE_SHIFT); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG6_HI); + knod_emit(priv, meta, v_lshlrev_b32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_HI); + knod_iset32(¶m[1], 32 - PAGE_SHIFT); + knod_emit(priv, meta, v_lshrrev_b32, param[0], param[1], param[2]); + + /* data =3D base_gaddr + page_gaddr */ + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG2_LO); + knod_vset32(¶m[2], KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_HI); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG2_HI); + knod_vset32(¶m[2], KNOD_AMDGPU_DATA_VREG_HI); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[2]); + + if (knod_prog->uses_adjust) { + /* Save page_base to PAGE_BASE_VREG before adding off */ + knod_vset32(¶m[0], KNOD_AMDGPU_PAGE_BASE_VREG_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_PAGE_BASE_VREG_HI); + knod_vset32(¶m[1], KNOD_AMDGPU_DATA_VREG_HI); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + } + + /* extract off (lower 16 bits of TMP_VREG6_LO) */ + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG8_LO); + knod_iset32(¶m[1], 0xffff); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG6_LO); + knod_emit(priv, meta, v_and_b32_e32, param[0], param[1], param[2]); + + /* data +=3D off */ + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG8_LO); + knod_vset32(¶m[2], KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_VREG_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[0]); + + /* 9. data_end =3D data + len (upper 16 bits of TMP_VREG6_LO) + * Compute directly into DATA_END_VREG (v66:v67). + */ + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_END_VREG_LO); + knod_iset32(¶m[1], 16); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG6_LO); + knod_emit(priv, meta, v_lshrrev_b32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_END_VREG_LO); + knod_vset32(¶m[1], KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(¶m[2], KNOD_AMDGPU_DATA_END_VREG_LO); + knod_emit(priv, meta, v_add_co_u32, param[0], param[1], param[2]); + knod_vset32(¶m[0], KNOD_AMDGPU_DATA_END_VREG_HI); + knod_iset32(¶m[1], 0); + knod_vset32(¶m[2], KNOD_AMDGPU_DATA_VREG_HI); + knod_emit(priv, meta, v_add_co_ci_u32_e32, param[0], param[1], + param[2]); + + /* Step 10 eliminated: data->DATA_VREG, data_end->DATA_END_VREG, + * slot_addr->SLOT_VREG computed directly in steps 6/8/9 above. + */ + + pr_debug("knod_bpf DEBUG: prologue emitted idx=3D%u (KNOD_META_INSNS=3D%d= )\n", + meta->amdgpu_insns, KNOD_META_INSNS); + if (WARN_ON(meta->amdgpu_insns > KNOD_META_INSNS)) + return -ENOSPC; + list_add_tail(&meta->l, &knod_prog->pre_insns); + + return 0; +} + +static int knod_prog_prepare(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog, + const struct bpf_insn *prog, + unsigned int cnt) +{ + struct knod_insn_meta *meta; + unsigned int i; + + /* Pre-scan: detect helper 44/65 to set uses_adjust early */ + for (i =3D 0; i < cnt; i++) { + if (prog[i].code =3D=3D (BPF_JMP | BPF_CALL) && + (prog[i].imm =3D=3D 44 || prog[i].imm =3D=3D 65)) { + knod_prog->uses_adjust =3D true; + break; + } + } + + knod_vset64(&r64[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_vset64(&r64[1], KNOD_AMDGPU_TMP_VREG1_LO); + knod_vset64(&r64[2], KNOD_AMDGPU_TMP_VREG2_LO); + knod_vset64(&r64[3], KNOD_AMDGPU_TMP_VREG3_LO); + knod_vset64(&r64[4], KNOD_AMDGPU_TMP_VREG4_LO); + knod_vset64(&r64[5], KNOD_AMDGPU_TMP_VREG5_LO); + knod_vset64(&r64[6], KNOD_AMDGPU_TMP_VREG6_LO); + knod_vset64(&r64[7], KNOD_AMDGPU_TMP_VREG7_LO); + knod_vset64(&r64[8], KNOD_AMDGPU_TMP_VREG8_LO); + knod_vset64(&r64[9], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset64(&r64[10], KNOD_AMDGPU_TMP_VREG10_LO); + knod_vset64(&r64[11], KNOD_AMDGPU_TMP_VREG11_LO); + knod_vset64(&r64[12], KNOD_AMDGPU_TMP_VREG12_LO); + knod_vset64(&r64[13], KNOD_AMDGPU_TMP_VREG13_LO); + knod_vset64(&r64[14], KNOD_AMDGPU_TMP_VREG14_LO); + knod_vset64(&r64[15], KNOD_AMDGPU_TMP_VREG15_LO); + knod_vset64(&r64[16], KNOD_AMDGPU_TMP_VREG16_LO); + knod_vset64(&r64[17], KNOD_AMDGPU_TMP_VREG17_LO); + knod_vset64(&r64[18], KNOD_AMDGPU_TMP_VREG18_LO); + knod_vset64(&r64[19], KNOD_AMDGPU_CTX_VREG_LO); + + knod_sset64(&sr64[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_sset64(&sr64[1], KNOD_AMDGPU_TMP_SREG1_LO); + knod_sset64(&sr64[2], KNOD_AMDGPU_TMP_SREG2_LO); + knod_sset64(&sr64[3], KNOD_AMDGPU_TMP_SREG3_LO); + knod_sset64(&sr64[4], KNOD_AMDGPU_TMP_SREG4_LO); + knod_sset64(&sr64[5], KNOD_AMDGPU_TMP_SREG5_LO); + + knod_vset64(&bpf_reg64[0], KNOD_AMDGPU_VREG0_LO); + knod_vset64(&bpf_reg64[1], KNOD_AMDGPU_VREG1_LO); + knod_vset64(&bpf_reg64[2], KNOD_AMDGPU_VREG2_LO); + knod_vset64(&bpf_reg64[3], KNOD_AMDGPU_VREG3_LO); + knod_vset64(&bpf_reg64[4], KNOD_AMDGPU_VREG4_LO); + knod_vset64(&bpf_reg64[5], KNOD_AMDGPU_VREG5_LO); + knod_vset64(&bpf_reg64[6], KNOD_AMDGPU_VREG6_LO); + knod_vset64(&bpf_reg64[7], KNOD_AMDGPU_VREG7_LO); + knod_vset64(&bpf_reg64[8], KNOD_AMDGPU_VREG8_LO); + knod_vset64(&bpf_reg64[9], KNOD_AMDGPU_VREG9_LO); + knod_vset64(&bpf_reg64[10], KNOD_AMDGPU_FRAME_POINTER_VREG_LO); + + knod_vset32(&r32[0], KNOD_AMDGPU_TMP_VREG0_LO); + for (i =3D 1; i < 40; i++) + knod_vset32(&r32[i], r32[i - 1].v + 1); + + if (knod_bpf_pkt_cache) { + int pkt_cache_start =3D knod_prog->uses_adjust ? + KNOD_AMDGPU_PKT_CACHE_VREG0 : + KNOD_AMDGPU_PAGE_BASE_VREG_LO; + + knod_vset32(&pkt_cache[0], pkt_cache_start); + for (i =3D 1; i < 64; i++) + knod_vset32(&pkt_cache[i], + pkt_cache[i - 1].v + 1); + } + + knod_vset32(&stack[0], KNOD_AMDGPU_STACK_VREG0); + for (i =3D 1; i < 128; i++) + knod_vset32(&stack[i], stack[i - 1].v + 1); + + for (i =3D 0; i < cnt; i++) { + meta =3D kzalloc_obj(*meta, GFP_KERNEL); + if (!meta) + return -ENOMEM; + + meta->insn =3D prog[i]; + meta->bpf_insn_idx =3D i; + + list_add_tail(&meta->l, &knod_prog->insns); + } + knod_prog->n_insns =3D cnt; + + return 0; +} + +static void knod_prog_free(struct knod_prog *knod_prog) +{ + struct knod_insn_meta *meta, *tmp; + + //kfree(knod_prog->subprog); + + list_for_each_entry_safe(meta, tmp, &knod_prog->pre_insns, l) { + list_del(&meta->l); + kfree(meta); + } + list_for_each_entry_safe(meta, tmp, &knod_prog->insns, l) { + list_del(&meta->l); + kfree(meta); + } + list_for_each_entry_safe(meta, tmp, &knod_prog->post_insns, l) { + list_del(&meta->l); + kfree(meta); + } + kfree(knod_prog); +} + +static int knod_bpf_verifier_prep(struct bpf_prog *prog) +{ + struct knod_prog *knod_prog; + struct knod_bpf_priv *priv; + int err; + + knod_prog =3D kzalloc_obj(struct knod_prog, GFP_KERNEL); + if (!knod_prog) + return -ENOMEM; + + INIT_LIST_HEAD(&knod_prog->insns); + INIT_LIST_HEAD(&knod_prog->pre_insns); + INIT_LIST_HEAD(&knod_prog->post_insns); + prog->aux->offload->dev_priv =3D knod_prog; + priv =3D bpf_offload_dev_priv(prog->aux->offload->offdev); + knod_prog->knodev =3D priv->knodev; + WRITE_ONCE(priv->knod_prog, knod_prog); + knod_prog->knod =3D priv->knod; + knod_prog->insn_idx =3D 0; + + if (priv->isa_version =3D=3D 10) { + knod_prog->done_mask_sreg =3D 32; + knod_prog->exec_save_base =3D 34; + knod_prog->initial_exec_sreg =3D + KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX10; + } else { + knod_prog->done_mask_sreg =3D KNOD_AMDGPU_DONE_MASK_SREG; + knod_prog->exec_save_base =3D KNOD_AMDGPU_EXEC_SAVE_SREG_BASE; + knod_prog->initial_exec_sreg =3D + KNOD_AMDGPU_INITIAL_EXEC_SREG_GFX9; + } + + err =3D knod_prog_prepare(priv, knod_prog, prog->insnsi, prog->len); + if (err) + goto err_free; + + knod_prog->meta =3D knod_prog_first_meta(knod_prog); + + return 0; + +err_free: + knod_prog_free(knod_prog); + + return err; +} + +static struct knod_insn_meta *knod_bpf_lookup_meta(struct knod_prog *knod_= prog, + short idx) +{ + struct knod_insn_meta *meta; + + list_for_each_entry(meta, &knod_prog->insns, l) { + if (meta->amdgpu_insn_idx =3D=3D AMDGPU_INSN_SKIP) + continue; + if (meta->bpf_insn_idx =3D=3D idx) + return meta; + } + + return NULL; +} + +static void knod_mov64_imm(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int d, u64 imm64) +{ + struct amdgcn_param32 param[2]; + + knod_vset32(¶m[0], d); + knod_iset32(¶m[1], imm64 & ~0U); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], d + 1); + knod_iset32(¶m[1], imm64 >> 32); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); +} + +static void knod_mov32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src) +{ + knod_emit(priv, meta, v_mov_b32_e32, dst, src); +} + +static void knod_mov64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src) +{ + knod_mov32(priv, meta, dst.lo, src.lo); + knod_mov32(priv, meta, dst.hi, src.hi); +} + +static void knod_add64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_add_co_u32, dst.lo, src0.lo, src1.lo); + knod_emit(priv, meta, v_add_co_ci_u32_e32, dst.hi, src0.hi, + src1.hi); +} + +/* No carry out/in */ +static void knod_add32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_add_u32, dst, src0, src1); +} + +static void knod_xor32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_xor_b32_e32, dst, src0, src1); +} + +static void knod_alignbit32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + knod_emit(priv, meta, v_alignbit_b32, dst, src0, src1, src2); +} + +static void knod_bfe32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + knod_emit(priv, meta, v_bfe_u32, dst, src0, src1, src2); +} + +static void knod_bfi32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + knod_emit(priv, meta, v_bfi_b32, dst, src0, src1, src2); +} + +static void knod_lshrrev32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_lshrrev_b32, dst, src0, src1); +} + +static void knod_lshrrev64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_lshrrev_b64, dst, src0, src1); +} + +static void knod_ashrrev32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_ashrrev_i32, dst, src0, src1); +} + +static void knod_ashrrev64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_ashrrev_i64, dst, src0, src1); +} + +static void knod_lshlrev32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_lshlrev_b32, dst, src0, src1); +} + +static void knod_lshlrev64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_lshlrev_b64, dst, src0, src1); +} + +/* No carry out/in */ +static void knod_sub32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_sub_u32, dst, src0, src1); +} + +static void knod_and32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_and_b32_e32, dst, src0, src1); +} + +static void knod_and64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_and32(priv, meta, dst.lo, src0.lo, src1.lo); + knod_and32(priv, meta, dst.hi, src0.hi, src1.hi); +} + +static void knod_or32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src0, + struct amdgcn_param32 src1) +{ + knod_emit(priv, meta, v_or_b32_e32, dst, src0, src1); +} + +static void knod_sub64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_sub_co_u32, dst.lo, src0.lo, src1.lo); + knod_emit(priv, meta, v_sub_co_ci_u32_e32, dst.hi, src0.hi, + src1.hi); +} + +static void knod_subrev64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src0, + struct amdgcn_param64 src1) +{ + knod_emit(priv, meta, v_subrev_co_u32, dst.lo, src0.lo, src1.lo); + knod_emit(priv, meta, v_subrev_co_ci_u32_e32, dst.hi, src0.hi, + src1.hi); +} + +static void knod_mul_lo32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + knod_emit(priv, meta, v_mul_lo_u32, dst, src1, src2); +} + +static void knod_mul_hi32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param32 dst, + struct amdgcn_param32 src1, + struct amdgcn_param32 src2) +{ + knod_emit(priv, meta, v_mul_hi_u32, dst, src1, src2); +} + +static void knod_mul64(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 src1, + struct amdgcn_param64 src2, + struct amdgcn_param64 tmp) +{ + /* + * v_mul_lo_u32 v1, v2, v1 + * v_mul_hi_u32 v5, v2, v0 + * v_mul_lo_u32 v3, v3, v0 + * v_mul_lo_u32 v0, v2, v0 + * v_add_u32_e32 v1, v5, v1 + * v_add_u32_e32 v1, v1, v3 + * + * v[0:1] =3D src1, dst + * v[2:3] =3D src2 + * v5 =3D tmp + */ + + /* v_mul_lo_u32 v1, v2, v1 */ + knod_mul_lo32(priv, meta, src2.hi, src1.lo, src2.hi); + /* v_mul_hi_u32 v5, v2, v0 */ + knod_mul_hi32(priv, meta, tmp.lo, src1.lo, src2.lo); + /* v_mul_lo_u32 v3, v3, v0 */ + knod_mul_lo32(priv, meta, src1.hi, src1.hi, src2.lo); + /* v_mul_lo_u32 v0, v2, v0 */ + knod_mul_lo32(priv, meta, src1.lo, src1.lo, src2.lo); + /* v_add_u32_e32 v1, v5, v1 */ + knod_add32(priv, meta, src2.lo, tmp.lo, src2.hi); + /* v_add_u32_e32 v1, v1, v3 */ + knod_add32(priv, meta, src1.hi, src2.lo, src1.hi); + knod_mov32(priv, meta, dst.lo, src1.lo); + knod_mov32(priv, meta, dst.hi, src1.hi); +} + +static void knod_div(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 imm, + struct amdgcn_param64 tmp_reg0, + struct amdgcn_param64 tmp_reg1, + struct amdgcn_param64 tmp_reg2, + struct amdgcn_param64 tmp_reg3) +{ + struct reciprocal_value_adv rvalue; + struct amdgcn_param64 p64[4]; + u8 pre_shift, exp; + + WARN_ON((imm.lo.type !=3D AMDGCN_PARAM_TYPE_INTEGER_0) && + (imm.lo.type !=3D AMDGCN_PARAM_TYPE_LITERAL_CONST)); + WARN_ON((imm.hi.type !=3D AMDGCN_PARAM_TYPE_INTEGER_0) && + (imm.hi.type !=3D AMDGCN_PARAM_TYPE_LITERAL_CONST)); + knod_iset64(&p64[0], 0); + knod_iset64(&p64[1], 0); + knod_iset64(&p64[2], 0); + knod_iset64(&p64[3], 0); + /* + * dst :=3D imm + * n :=3D dst_reg + */ + if (imm.imm > U32_MAX) { + knod_mov64(priv, meta, dst, p64[0]); + return; + } + + if (imm.imm >=3D 1U << 31) { + /* result =3D n >=3D dst; */ + knod_mov64(priv, meta, tmp_reg0, imm); + knod_emit(priv, meta, v_cmp_ge_u64, dst, tmp_reg0); + return; + } + + rvalue =3D reciprocal_value_adv(imm.lo.v, 32); + exp =3D rvalue.exp; + if (rvalue.is_wide_m && !(imm.lo.v & 1)) { + pre_shift =3D fls(imm.lo.v & -imm.lo.v) - 1; + rvalue =3D reciprocal_value_adv(imm.lo.v >> pre_shift, + 32 - pre_shift); + } else { + pre_shift =3D 0; + } + + if (imm.lo.v =3D=3D 1U << exp) { + knod_iset64(&p64[0], exp); + /* n =3D n >> exp */ + knod_lshrrev64(priv, meta, dst, p64[0], dst); + return; + } else if (rvalue.is_wide_m) { + /* + * pre_shift must be zero when reached here. + * t =3D (n * rvalue.m) >> 32; + * result =3D n - t; + * result >>=3D 1; + * result +=3D t; + * result >>=3D rvalue.sh - 1; + */ + + /* + * n :=3D VREG0 + * t :=3D VREG1 + * rvalue.m :=3D VREG2 + * tmp :=3D VREG3 + */ + + /* n :=3D TMP_VREG0 */ + knod_mov64(priv, meta, tmp_reg0, dst); + + knod_iset64(&p64[0], rvalue.m); + /* rvalue.m :=3D TMP_VREG2 */ + knod_mov64(priv, meta, tmp_reg2, p64[0]); + + /* t =3D n * rvalue.m; */ + knod_mul64(priv, meta, + tmp_reg1, /* t */ + tmp_reg0, /* n */ + tmp_reg2, /* rvalue.m */ + tmp_reg3); /* tmp */ + + /* t >>=3D 32; */ + knod_iset64(&p64[0], 0); + knod_mov32(priv, meta, tmp_reg1.lo, tmp_reg1.hi); + knod_mov32(priv, meta, tmp_reg1.hi, p64[0].lo); + + /* result =3D n - t */ + knod_sub64(priv, meta, dst, dst, tmp_reg1); + + /* result >>=3D 1 */ + knod_iset64(&p64[0], 1); + knod_lshrrev64(priv, meta, dst, p64[0], dst); + + /* result +=3D t; */ + knod_add64(priv, meta, + dst, + dst, /* result */ + tmp_reg1); /* t */ + + /* result >>=3D rvalue.sh - 1; */ + knod_iset64(&p64[0], rvalue.sh - 1); + WARN_ON(rvalue.sh - 1 > 31); + knod_lshrrev64(priv, meta, dst, p64[0], dst); + return; + } + + /* + * if (pre_shift) + * result =3D n >> pre_shift; + * result =3D ((u64)result * rvalue.m) >> 32; + * result >>=3D rvalue.sh; + */ + + /* + * n :=3D VREG0 + * :=3D VREG1 + * rvalue.m :=3D VREG2 + * tmp :=3D VREG3 + * result :=3D dst * 2 + */ + + /* n :=3D TMP_VREG0 */ + knod_mov64(priv, meta, tmp_reg0, dst); + + /* rvalue.m :=3D TMP_VREG2 */ + knod_iset64(&p64[0], rvalue.m); + knod_mov64(priv, meta, tmp_reg2, p64[0]); + + if (pre_shift) { + /* result =3D n >> pre_shift; */ + knod_iset64(&p64[0], pre_shift); + knod_lshrrev64(priv, meta, dst, p64[0], + tmp_reg0); /* n */ + } else { + /* tmp =3D 0 */ + knod_iset64(&p64[0], 0); + knod_mov64(priv, meta, tmp_reg0, p64[0]); + } + + /* result =3D result * rvalue.m; */ + knod_mul64(priv, meta, + dst, /* result */ + dst, /* result */ + tmp_reg2, /* rvalue.m */ + tmp_reg3); /* tmp */ + + /* result >>=3D (32 + rvalue.sh); */ + knod_iset64(&p64[0], 32 + rvalue.sh); + knod_lshrrev64(priv, meta, dst, p64[0], dst); +} + +static void knod_mod(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, + struct amdgcn_param64 imm, + struct amdgcn_param64 tmp_reg0, + struct amdgcn_param64 tmp_reg1, + struct amdgcn_param64 tmp_reg2, + struct amdgcn_param64 tmp_reg3, + struct amdgcn_param64 tmp_reg4) +{ + WARN_ON((imm.lo.type !=3D AMDGCN_PARAM_TYPE_INTEGER_0) && + (imm.lo.type !=3D AMDGCN_PARAM_TYPE_LITERAL_CONST)); + WARN_ON((imm.hi.type !=3D AMDGCN_PARAM_TYPE_INTEGER_0) && + (imm.hi.type !=3D AMDGCN_PARAM_TYPE_LITERAL_CONST)); + /* q :=3D tmp_reg0 */ + knod_mov64(priv, meta, tmp_reg0, dst); + /* q =3D n / imm */ + knod_div(priv, meta, tmp_reg0, imm, + tmp_reg1, tmp_reg2, tmp_reg3, tmp_reg4); + + /* tmp_reg1 :=3D imm_reg */ + knod_mov64(priv, meta, tmp_reg1, imm); + + /* imm * q :=3D tmp_reg3 */ + knod_mul64(priv, meta, + tmp_reg3, /* imm * q */ + tmp_reg0, /* q */ + tmp_reg1, /* imm_reg */ + tmp_reg2); /* tmp */ + + knod_sub64(priv, meta, dst, dst, tmp_reg3); +} + +/* + * Fast constant modulo on the 32-bit value in @dst.lo for divisors of a + * special form, avoiding knod_mod's reciprocal divide + 64-bit multiply: + * 2^k -> dst & (2^k-1) (mask) + * 2^k + 1 -> lo - hi (+C if lo lo + hi (-C while >=3DC) (Mersenne: 2^k =3D 1 m= od C) + * lo/hi are the low/high k-bit halves. One fold is exact for a 32-bit + * dividend when 2^k covers the high half (true for e.g. 65537 =3D 2^16+1, + * kondor's per-packet `hash % RING_SIZE`). Returns false for other + * divisors (caller falls back to knod_mod). Scratch: r64[0], r64[1]. + */ +static bool knod_mod_k32(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 dst, u32 imm) +{ + struct amdgcn_param32 p; + int i; + + if (is_power_of_2(imm)) { + knod_iset32(&p, imm - 1); + knod_and32(priv, meta, dst.lo, p, dst.lo); + } else if (is_power_of_2(imm - 1) && (imm - 1) >=3D (1u << 16)) { + knod_iset32(&p, imm - 2); /* mask 2^k-1 */ + knod_and32(priv, meta, r64[0].lo, p, dst.lo); /* lo */ + knod_iset32(&p, ilog2(imm - 1)); /* k */ + knod_emit(priv, meta, v_lshrrev_b32, r64[1].lo, p, dst.lo); + /* lo-hi */ + knod_sub32(priv, meta, dst.lo, r64[0].lo, r64[1].lo); + knod_emit(priv, meta, v_cmp_lt_u32, r64[0].lo, r64[1].lo); + knod_iset32(&p, imm); + knod_add32(priv, meta, r64[1].lo, p, dst.lo); /* +C */ + knod_emit(priv, meta, v_cndmask_b32_e32, dst.lo, dst.lo, + r64[1].lo); + } else if (is_power_of_2(imm + 1) && (imm + 1) >=3D (1u << 16)) { + knod_iset32(&p, imm); /* mask 2^k-1 */ + knod_and32(priv, meta, r64[0].lo, p, dst.lo); /* lo */ + knod_iset32(&p, ilog2(imm + 1)); /* k */ + knod_emit(priv, meta, v_lshrrev_b32, r64[1].lo, p, dst.lo); + /* lo+hi */ + knod_add32(priv, meta, dst.lo, r64[0].lo, r64[1].lo); + knod_iset32(&p, imm); + knod_mov32(priv, meta, r64[0].lo, p); /* C in VGPR */ + for (i =3D 0; i < 2; i++) { /* r < 2C */ + knod_emit(priv, meta, v_cmp_le_u32, r64[0].lo, dst.lo); + knod_sub32(priv, meta, r64[1].lo, dst.lo, r64[0].lo); + knod_emit(priv, meta, v_cndmask_b32_e32, dst.lo, + dst.lo, r64[1].lo); + } + } else { + return false; + } + + knod_iset32(&p, 0); + knod_mov32(priv, meta, dst.hi, p); + return true; +} + +/* Considered to be able to use all temporary vregisters + * Also, key is stack pointer, not global + */ +static void knod_jhash(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + u32 dst_idx, u32 length, u32 initval) +{ + u32 a_reg =3D TREG32_MAX - 3, b_reg =3D TREG32_MAX - 2; + u32 c_reg =3D TREG32_MAX - 1, d_reg =3D TREG32_MAX; + u32 key_in_pkt =3D KEY_IN_PKT_32; + struct amdgcn_param32 p32; + + knod_iset32(&p32, JHASH_INITVAL + length + initval); + knod_mov32(priv, meta, r32[a_reg], p32); + knod_mov32(priv, meta, r32[b_reg], p32); + knod_mov32(priv, meta, r32[c_reg], p32); + knod_iset32(&p32, 0); + + while (length > 12) { + /* a +=3D *key; */ + knod_add32(priv, meta, r32[a_reg], r32[a_reg], + r32[key_in_pkt]); + /* b +=3D *(key + 4); */ + knod_add32(priv, meta, r32[b_reg], r32[b_reg], + r32[key_in_pkt + 1]); + /* c +=3D *(key + 8); */ + knod_add32(priv, meta, r32[c_reg], r32[c_reg], + r32[key_in_pkt + 2]); + /* a -=3D c; */ + knod_sub32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* a ^=3D rol32(c, 4); */ + knod_iset32(&p32, 32 - 4); + knod_alignbit32(priv, meta, r32[d_reg], r32[c_reg], + r32[c_reg], p32); + knod_xor32(priv, meta, + r32[a_reg], r32[a_reg], r32[d_reg]); + /* c +=3D b; */ + knod_add32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* b -=3D a; */ + knod_sub32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + /* b ^=3D rol32(a, 6); */ + knod_iset32(&p32, 32 - 6); + knod_alignbit32(priv, meta, r32[d_reg], r32[a_reg], + r32[a_reg], p32); + knod_xor32(priv, meta, + r32[b_reg], r32[b_reg], r32[d_reg]); + /*a +=3D c; */ + knod_add32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* c -=3D b; */ + knod_sub32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* c ^=3D rol32(b, 8); */ + knod_iset32(&p32, 32 - 8); + knod_alignbit32(priv, meta, r32[d_reg], r32[b_reg], + r32[b_reg], p32); + knod_xor32(priv, meta, + r32[c_reg], r32[c_reg], r32[d_reg]); + /* b +=3D a; */ + knod_add32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + /* a -=3D c; */ + knod_sub32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* a ^=3D rol32(c, 16); */ + knod_iset32(&p32, 32 - 16); + knod_alignbit32(priv, meta, r32[d_reg], r32[c_reg], + r32[c_reg], p32); + knod_xor32(priv, meta, r32[a_reg], r32[a_reg], r32[d_reg]); + /* c +=3D b; */ + knod_add32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* b -=3D a; */ + knod_sub32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + /* b ^=3D rol32(a, 19); */ + knod_iset32(&p32, 32 - 19); + knod_alignbit32(priv, meta, r32[d_reg], r32[a_reg], + r32[a_reg], p32); + knod_xor32(priv, meta, r32[b_reg], r32[b_reg], r32[d_reg]); + /* a +=3D c; */ + knod_add32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* c -=3D b; */ + knod_sub32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* c ^=3D rol32(b, 4); */ + knod_iset32(&p32, 32 - 4); + knod_alignbit32(priv, meta, r32[d_reg], r32[b_reg], + r32[b_reg], p32); + knod_xor32(priv, meta, r32[c_reg], r32[c_reg], r32[d_reg]); + /* b +=3D a; */ + knod_add32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + length -=3D 12; + key_in_pkt +=3D 3; + } + + switch (length) { + case 12: + /* c +=3D (unsigned int)k[11]<<24; */ + fallthrough; + case 11: + /* c +=3D (unsigned int)k[10]<<16; */ + fallthrough; + case 10: + /* c +=3D (unsigned int)k[9]<<8; */ + fallthrough; + case 9: + /* c +=3D k[8]; */ + knod_add32(priv, meta, r32[c_reg], r32[c_reg], + r32[key_in_pkt + 2]); + fallthrough; + case 8: + /* b +=3D (unsigned int)k[7]<<24; */ + fallthrough; + case 7: + /* b +=3D (unsigned int)k[6]<<16; */ + fallthrough; + case 6: + /* b +=3D (unsigned int)k[5]<<8; */ + fallthrough; + case 5: + /* b +=3D k[4]; */ + knod_add32(priv, meta, r32[b_reg], r32[b_reg], + r32[key_in_pkt + 1]); + fallthrough; + case 4: + /* a +=3D (unsigned int)k[3]<<24; */ + fallthrough; + case 3: + /* a +=3D (unsigned int)k[2]<<16; */ + fallthrough; + case 2: + /* a +=3D (unsigned int)k[1]<<8; */ + fallthrough; + case 1: + /* a +=3D k[0]; */ + knod_add32(priv, meta, r32[a_reg], r32[a_reg], + r32[key_in_pkt]); + /* c ^=3D b; */ + knod_xor32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* c -=3D rol32(b, 14); */ + knod_iset32(&p32, 32 - 14); + knod_alignbit32(priv, meta, r32[d_reg], r32[b_reg], + r32[b_reg], p32); + knod_sub32(priv, meta, r32[c_reg], r32[c_reg], r32[d_reg]); + /* a ^=3D c; */ + knod_xor32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* a -=3D rol32(c, 11); */ + knod_iset32(&p32, 32 - 11); + knod_alignbit32(priv, meta, r32[d_reg], r32[c_reg], + r32[c_reg], p32); + knod_sub32(priv, meta, r32[a_reg], r32[a_reg], r32[d_reg]); + /* b ^=3D a; */ + knod_xor32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + /* b -=3D rol32(a, 25); */ + knod_iset32(&p32, 32 - 25); + knod_alignbit32(priv, meta, r32[d_reg], r32[a_reg], + r32[a_reg], p32); + knod_sub32(priv, meta, r32[b_reg], r32[b_reg], r32[d_reg]); + /* c ^=3D b; */ + knod_xor32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* c -=3D rol32(b, 16); */ + knod_iset32(&p32, 32 - 16); + knod_alignbit32(priv, meta, r32[d_reg], r32[b_reg], + r32[b_reg], p32); + knod_sub32(priv, meta, r32[c_reg], r32[c_reg], r32[d_reg]); + /* a ^=3D c; */ + knod_xor32(priv, meta, r32[a_reg], r32[a_reg], r32[c_reg]); + /* a -=3D rol32(c, 4); */ + knod_iset32(&p32, 32 - 4); + knod_alignbit32(priv, meta, r32[d_reg], r32[c_reg], + r32[c_reg], p32); + knod_sub32(priv, meta, r32[a_reg], r32[a_reg], r32[d_reg]); + /* b ^=3D a; */ + knod_xor32(priv, meta, r32[b_reg], r32[b_reg], r32[a_reg]); + /* b -=3D rol32(a, 14); */ + knod_iset32(&p32, 32 - 14); + knod_alignbit32(priv, meta, r32[d_reg], r32[a_reg], + r32[a_reg], p32); + knod_sub32(priv, meta, r32[b_reg], r32[b_reg], r32[d_reg]); + /* c ^=3D b; */ + knod_xor32(priv, meta, r32[c_reg], r32[c_reg], r32[b_reg]); + /* c -=3D rol32(b, 24); */ + knod_iset32(&p32, 32 - 24); + knod_alignbit32(priv, meta, r32[d_reg], r32[b_reg], + r32[b_reg], p32); + knod_sub32(priv, meta, r32[c_reg], r32[c_reg], r32[d_reg]); + break; + case 0: /* Nothing left to add */ + break; + } + + knod_mov32(priv, meta, r64[dst_idx].lo, r32[c_reg]); +} + +static u64 knod_bpf_map_gaddr(struct knod_bpf_priv *priv, int id) +{ + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_map *knod_map; + struct knod_mem *mem; + + mutex_lock(&knodev->lock); + list_for_each_entry(knod_map, &knodev->accel->xdp.bound_maps, list) { + if (knod_map->offmap->map.id =3D=3D id) { + mem =3D knod_map->mem; + mutex_unlock(&knodev->lock); + return (u64)mem->gaddr; + } + } + mutex_unlock(&knodev->lock); + + return 0; +} + +static void *knod_bpf_map_kaddr(struct knod_bpf_priv *priv, int id) +{ + struct knod_dev *knodev =3D priv->knodev; + struct knod_bpf_map *knod_map; + struct knod_mem *mem; + + mutex_lock(&knodev->lock); + list_for_each_entry(knod_map, &knodev->accel->xdp.bound_maps, list) { + if (knod_map->offmap->map.id =3D=3D id) { + mem =3D knod_map->mem; + + /* GPUVM */ + mutex_unlock(&knodev->lock); + return (void *)mem->kaddr; + } + } + mutex_unlock(&knodev->lock); + + return NULL; +} + +static u64 knod_bpf_get_map_gaddr(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta1, + struct knod_insn_meta *meta2) +{ + struct bpf_map *map; + + map =3D (void *)(unsigned long)((u32)meta1->insn.imm | + (u64)meta2->insn.imm << 32); + + return knod_bpf_map_gaddr(priv, map->id); +} + +static int knod_bpf_get_map_id(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta1, + struct knod_insn_meta *meta2) +{ + struct bpf_map *map; + + map =3D (void *)(unsigned long)((u32)meta1->insn.imm | + (u64)meta2->insn.imm << 32); + + return map->id; +} + +static int knod_bpf_get_amdgpu_insn_idx(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int t) +{ + int i, insn_idx =3D meta->amdgpu_insn_idx; + + for (i =3D 0; i < t; i++) + insn_idx +=3D meta->amdgpu_insn[i].size / 4; + + return insn_idx; +} + +static void knod_bpf_fixup_branch(struct knod_bpf_priv *priv, + struct amdgcn_branch_fixup *fixup) +{ + int target_off =3D knod_bpf_get_amdgpu_insn_idx(priv, + fixup->target_label->meta, + fixup->target_label->insn_idx); + int cur_off =3D knod_bpf_get_amdgpu_insn_idx(priv, + fixup->meta, + fixup->insn_idx); + cur_off++; + + target_off -=3D cur_off; + emit_branch_fixup(priv->isa_version, + &fixup->meta->amdgpu_insn[fixup->insn_idx], + target_off); + knod_jit_dbg(" target_off was updated to %d\n", target_off); +} + +static void knod_bpf_set_fixup(struct knod_insn_meta *meta, + struct amdgcn_branch_fixup *fixup, + struct amdgcn_label *target_label, + int insn_idx) +{ + fixup->meta =3D meta; + fixup->insn_idx =3D insn_idx; + fixup->target_label =3D target_label; +} + +static void knod_bpf_set_label(struct knod_insn_meta *meta, + struct amdgcn_label *label, + int insn_idx) +{ + label->meta =3D meta; + label->insn_idx =3D insn_idx; +} + +/* + * knod_bpf_emit_offlen_writeback - Write updated off/len to spsc_bd. + * + * Uses PAGE_BASE_VREG (set once in prologue), computes + * new_off =3D DATA_VREG_LO - PAGE_BASE_VREG_LO and + * new_len =3D DATA_END_VREG_LO - DATA_VREG_LO, packs them as (len<<16)|of= f, + * and stores the result at spsc_bd.off via SLOT_VREG. + * + * Clobbers: TMP_VREG2 (v26:v27). + */ +static void knod_bpf_emit_offlen_writeback(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + struct amdgcn_param32 s0, s1, data_lo, data_end_lo, pbase_lo, slot_lo; + struct amdgcn_param32 imm; + + knod_vset32(&s0, KNOD_AMDGPU_TMP_VREG2_LO); + knod_vset32(&s1, KNOD_AMDGPU_TMP_VREG2_HI); + knod_vset32(&data_lo, KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(&data_end_lo, KNOD_AMDGPU_DATA_END_VREG_LO); + knod_vset32(&pbase_lo, KNOD_AMDGPU_PAGE_BASE_VREG_LO); + knod_vset32(&slot_lo, KNOD_AMDGPU_SLOT_VREG_LO); + + /* s0 =3D len =3D DATA_END_LO - DATA_LO */ + knod_sub32(priv, meta, s0, data_end_lo, data_lo); + + /* s0 =3D len << 16 */ + knod_iset32(&imm, 16); + knod_lshlrev32(priv, meta, s0, imm, s0); + + /* s1 =3D off =3D DATA_LO - page_base_lo */ + knod_sub32(priv, meta, s1, data_lo, pbase_lo); + + /* s0 =3D (len << 16) | off */ + knod_or32(priv, meta, s0, s0, s1); + + /* Store packed {off, len} to spsc_bd */ + knod_emit(priv, meta, global_store_dword, s0, slot_lo, + offsetof(struct spsc_bd, off)); +} + +/* + * knod_bpf_xdp_adjust_head - JIT bpf_xdp_adjust_head (helper 44). + * + * R2 =3D delta (signed 32-bit). Adjusts DATA_VREG by delta. + * Bounds: page_base <=3D DATA_VREG <=3D DATA_END_VREG - ETH_HLEN. + * Each bound is checked with its own VOPC, but VCC is captured into + * VGPRs via v_cndmask (VALU) rather than SGPRs via s_mov_b64 (SALU). + * VALU reads VCC correctly after VOPC; only SALU suffers the GFX10 + * dual-VOPC stale-read hazard. + * page_base is reloaded on demand from param + spsc_bd. + * On failure, DATA_VREG is restored and R0 =3D -EINVAL. + * On success, R0 =3D 0. + * + * Clobbers: TMP_VREG0 (v22:v23), TMP_VREG1 (v24:v25), TMP_VREG2 (v26:v27), + * TMP_SREG0 (s16), TMP_SREG2 (s20:s21). + */ +static void knod_bpf_xdp_adjust_head(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + struct amdgcn_param32 ub_lo, ub_hi, dend_lo, dend_hi, sext_dst; + struct amdgcn_param32 shift_amt; + struct amdgcn_param32 tmp0_lo, tmp0_hi, data_lo, data_hi, fail_lo; + struct amdgcn_param32 fail_hi; + struct amdgcn_param64 data_vreg, pbase_vreg, ub; + struct amdgcn_param32 r0_lo, r0_hi, imm, delta; + + knod_vset64(&data_vreg, KNOD_AMDGPU_DATA_VREG_LO); + + knod_vset32(&tmp0_lo, KNOD_AMDGPU_TMP_VREG0_LO); + knod_vset32(&tmp0_hi, KNOD_AMDGPU_TMP_VREG0_HI); + knod_vset32(&data_lo, KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(&data_hi, KNOD_AMDGPU_DATA_VREG_HI); + knod_vset32(&r0_lo, KNOD_AMDGPU_VREG0_LO); + knod_vset32(&r0_hi, KNOD_AMDGPU_VREG0_HI); + knod_vset32(&delta, bpf_reg64[2].lo.v); + knod_vset32(&fail_lo, KNOD_AMDGPU_TMP_VREG2_LO); + knod_vset32(&fail_hi, KNOD_AMDGPU_TMP_VREG2_HI); + + /* 1. Save original DATA_VREG -> TMP_VREG0 */ + knod_mov32(priv, meta, tmp0_lo, data_lo); + knod_mov32(priv, meta, tmp0_hi, data_hi); + + /* 2. DATA_VREG +=3D delta (R2.lo, sign-extended to 64-bit) */ + knod_emit(priv, meta, v_add_co_u32, data_lo, delta, data_lo); + + knod_vset32(&sext_dst, KNOD_AMDGPU_TMP_VREG1_LO); + knod_iset32(&shift_amt, 31); + knod_emit(priv, meta, v_ashrrev_i32, sext_dst, shift_amt, delta); + + knod_emit(priv, meta, v_add_co_ci_u32_e32, data_hi, sext_dst, + data_hi); + + /* 3. Lower bound: DATA_VREG < page_base -> VCC =3D fail */ + knod_vset64(&pbase_vreg, KNOD_AMDGPU_PAGE_BASE_VREG_LO); + knod_emit(priv, meta, v_cmp_lt_u64, data_vreg, pbase_vreg); + + /* + * Capture VCC -> VGPR via v_cndmask (VALU reads VCC correctly, + * unlike SALU which suffers the dual-VOPC stale-read hazard). + */ + knod_iset32(&imm, 1); + knod_mov32(priv, meta, fail_hi, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, fail_lo, imm, fail_hi); + + /* 5. Upper bound: DATA_VREG > DATA_END_VREG - ETH_HLEN */ + knod_vset32(&ub_lo, KNOD_AMDGPU_TMP_VREG1_LO); + knod_vset32(&ub_hi, KNOD_AMDGPU_TMP_VREG1_HI); + knod_vset32(&dend_lo, KNOD_AMDGPU_DATA_END_VREG_LO); + knod_vset32(&dend_hi, KNOD_AMDGPU_DATA_END_VREG_HI); + + knod_iset32(&imm, ETH_HLEN); + /* + * v_sub_co_u32 is VOP2 on GFX9, whose vsrc1 must be a VGPR (a literal + * there reads v0). Subtraction is not commutative, so materialise + * ETH_HLEN into a scratch VGPR (ub_hi, overwritten by the high half + * below) and use it as src1 instead of an immediate. + */ + knod_mov32(priv, meta, ub_hi, imm); + knod_emit(priv, meta, v_sub_co_u32, ub_lo, dend_lo, ub_hi); + knod_iset32(&imm, 0); + knod_mov32(priv, meta, delta, imm); + knod_emit(priv, meta, v_sub_co_ci_u32_e32, ub_hi, dend_hi, delta); + + knod_vset64(&ub, KNOD_AMDGPU_TMP_VREG1_LO); + knod_emit(priv, meta, v_cmp_gt_u64, data_vreg, ub); + + /* Capture upper_fail via v_cndmask, combine, convert to VCC */ + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, fail_hi, imm, fail_hi); + + knod_emit(priv, meta, v_or_b32_e32, fail_lo, fail_lo, fail_hi); + + knod_emit(priv, meta, v_cmp_lt_u32, imm, fail_lo); + + /* 6. Conditional restore: VCC=3D1(fail) -> original, + * VCC=3D0(pass) -> adjusted + */ + knod_emit(priv, meta, v_cndmask_b32_e32, data_lo, data_lo, + tmp0_lo); + knod_emit(priv, meta, v_cndmask_b32_e32, data_hi, data_hi, + tmp0_hi); + + /* 7. R0 =3D VCC ? -EINVAL : 0 */ + knod_iset32(&imm, -EINVAL); + knod_mov32(priv, meta, tmp0_lo, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, r0_lo, imm, tmp0_lo); + + knod_iset32(&imm, -1); + knod_mov32(priv, meta, tmp0_hi, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, r0_hi, imm, tmp0_hi); +} + +/* + * knod_bpf_xdp_adjust_tail - JIT bpf_xdp_adjust_tail (helper 65). + * + * R2 =3D delta (signed 32-bit). Adjusts DATA_END_VREG by delta. + * Bounds: DATA_VREG + ETH_HLEN <=3D DATA_END_VREG <=3D page_base + PAGE_S= IZE. + * Each bound is checked with its own VOPC, but VCC is captured into + * VGPRs via v_cndmask (VALU) rather than SGPRs via s_mov_b64 (SALU). + * VALU reads VCC correctly after VOPC; only SALU suffers the GFX10 + * dual-VOPC stale-read hazard. + * page_base is reloaded on demand from param + spsc_bd. + * On failure, DATA_END_VREG is restored and R0 =3D -EINVAL. + * On success, R0 =3D 0. + * + * Clobbers: TMP_VREG0 (v22:v23), TMP_VREG1 (v24:v25), TMP_VREG2 (v26:v27), + * TMP_SREG0 (s16), TMP_SREG2 (s20:s21). + */ +static void knod_bpf_xdp_adjust_tail(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + struct amdgcn_param32 tmp0_lo, tmp0_hi, dend_lo, dend_hi, fail_lo; + struct amdgcn_param32 fail_hi; + struct amdgcn_param32 lb_lo, lb_hi, d_lo, d_hi, sext_dst, shift_amt; + struct amdgcn_param32 pb_src_lo, pb_src_hi; + struct amdgcn_param32 r0_lo, r0_hi; + struct amdgcn_param32 imm, delta; + struct amdgcn_param64 dend_vreg, lb; + + knod_vset32(&tmp0_lo, KNOD_AMDGPU_TMP_VREG0_LO); + knod_vset32(&tmp0_hi, KNOD_AMDGPU_TMP_VREG0_HI); + knod_vset32(&dend_lo, KNOD_AMDGPU_DATA_END_VREG_LO); + knod_vset32(&dend_hi, KNOD_AMDGPU_DATA_END_VREG_HI); + knod_vset32(&r0_lo, KNOD_AMDGPU_VREG0_LO); + knod_vset32(&r0_hi, KNOD_AMDGPU_VREG0_HI); + knod_vset32(&delta, bpf_reg64[2].lo.v); + knod_vset32(&fail_lo, KNOD_AMDGPU_TMP_VREG2_LO); + knod_vset32(&fail_hi, KNOD_AMDGPU_TMP_VREG2_HI); + knod_vset64(&dend_vreg, KNOD_AMDGPU_DATA_END_VREG_LO); + + /* 1. Save original DATA_END_VREG -> TMP_VREG0 */ + knod_mov32(priv, meta, tmp0_lo, dend_lo); + knod_mov32(priv, meta, tmp0_hi, dend_hi); + + /* 2. DATA_END_VREG +=3D delta (R2.lo, sign-extended) */ + knod_emit(priv, meta, v_add_co_u32, dend_lo, delta, dend_lo); + + knod_vset32(&sext_dst, KNOD_AMDGPU_TMP_VREG1_LO); + knod_iset32(&shift_amt, 31); + knod_emit(priv, meta, v_ashrrev_i32, sext_dst, shift_amt, delta); + + knod_emit(priv, meta, v_add_co_ci_u32_e32, dend_hi, sext_dst, + dend_hi); + + /* 3. Lower bound: lb =3D DATA + ETH_HLEN -> TMP_VREG1 */ + knod_vset32(&lb_lo, KNOD_AMDGPU_TMP_VREG1_LO); + knod_vset32(&lb_hi, KNOD_AMDGPU_TMP_VREG1_HI); + knod_vset32(&d_lo, KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(&d_hi, KNOD_AMDGPU_DATA_VREG_HI); + + knod_iset32(&imm, ETH_HLEN); + knod_emit(priv, meta, v_add_co_u32, lb_lo, imm, d_lo); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_add_co_ci_u32_e32, lb_hi, imm, d_hi); + + /* VOPC#1: DATA_END < lb -> VCC =3D lower_fail */ + knod_vset64(&lb, KNOD_AMDGPU_TMP_VREG1_LO); + knod_emit(priv, meta, v_cmp_lt_u64, dend_vreg, lb); + + /* + * Capture VCC -> VGPR via v_cndmask (VALU reads VCC correctly, + * unlike SALU which suffers the GFX10 dual-VOPC stale-read hazard). + */ + knod_iset32(&imm, 1); + knod_mov32(priv, meta, fail_hi, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, fail_lo, imm, fail_hi); + + /* 4. Upper bound: ub =3D page_base + PAGE_SIZE -> TMP_VREG1 */ + knod_vset32(&pb_src_lo, KNOD_AMDGPU_PAGE_BASE_VREG_LO); + knod_vset32(&pb_src_hi, KNOD_AMDGPU_PAGE_BASE_VREG_HI); + knod_mov32(priv, meta, lb_lo, pb_src_lo); + knod_mov32(priv, meta, lb_hi, pb_src_hi); + + knod_iset32(&imm, PAGE_SIZE); + knod_emit(priv, meta, v_add_co_u32, lb_lo, imm, lb_lo); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_add_co_ci_u32_e32, lb_hi, imm, lb_hi); + + /* VOPC#2: DATA_END > ub -> VCC =3D upper_fail */ + knod_emit(priv, meta, v_cmp_gt_u64, dend_vreg, lb); + + /* Capture upper_fail via v_cndmask, combine, convert to VCC */ + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, fail_hi, imm, fail_hi); + + knod_emit(priv, meta, v_or_b32_e32, fail_lo, fail_lo, fail_hi); + + knod_emit(priv, meta, v_cmp_lt_u32, imm, fail_lo); + + /* 5. Conditional restore: VCC=3D1(fail) -> original, + * VCC=3D0(pass) -> adjusted + */ + knod_emit(priv, meta, v_cndmask_b32_e32, dend_lo, dend_lo, + tmp0_lo); + knod_emit(priv, meta, v_cndmask_b32_e32, dend_hi, dend_hi, + tmp0_hi); + + /* 6. R0 =3D VCC ? -EINVAL : 0 */ + knod_iset32(&imm, -EINVAL); + knod_mov32(priv, meta, tmp0_lo, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, r0_lo, imm, tmp0_lo); + + knod_iset32(&imm, -1); + knod_mov32(priv, meta, tmp0_hi, imm); + knod_iset32(&imm, 0); + knod_emit(priv, meta, v_cndmask_b32_e32, r0_hi, imm, tmp0_hi); +} + +static void knod_bpf_load_size(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 *dst, + /* packet or stack */ + struct amdgcn_param32 *cache, + int size, int off) +{ + struct amdgcn_param32 p32[2]; + + knod_jit_dbg(" %d: off =3D %d off_4 =3D %d size =3D %d\n", meta->bpf_insn= _idx, + off, off%4, size); + switch (size) { + case sizeof(unsigned long): + if ((off % 4) =3D=3D 0) { + knod_mov32(priv, meta, dst->lo, cache[off / 4]); + knod_mov32(priv, meta, dst->hi, + cache[(off / 4) + 1]); + } else if ((off % 4) =3D=3D 1) { + WARN_ON_ONCE(1); + } else if ((off % 4) =3D=3D 2) { + WARN_ON_ONCE(1); + } else { + WARN_ON_ONCE(1); + } + break; + case sizeof(unsigned int): + if ((off % 4) =3D=3D 0) { + knod_mov32(priv, meta, dst->lo, cache[off / 4]); + } else if ((off % 4) =3D=3D 1) { + knod_iset32(&p32[0], 8); + knod_lshrrev32(priv, meta, r32[0], p32[0], + cache[off / 4]); + knod_iset32(&p32[0], 24); + knod_lshlrev32(priv, meta, dst->lo, p32[0], + cache[(off / 4) + 1]); + knod_or32(priv, meta, dst->lo, dst->lo, r32[0]); + } else if ((off % 4) =3D=3D 2) { + knod_iset32(&p32[0], 16); + knod_lshrrev32(priv, meta, r32[0], p32[0], + cache[off / 4]); + knod_lshlrev32(priv, meta, dst->lo, p32[0], + cache[(off / 4) + 1]); + knod_or32(priv, meta, dst->lo, dst->lo, r32[0]); + } else { + knod_iset32(&p32[0], 24); + knod_lshrrev32(priv, meta, r32[0], p32[0], + cache[off / 4]); + knod_iset32(&p32[0], 8); + knod_lshlrev32(priv, meta, dst->lo, p32[0], + cache[(off / 4) + 1]); + knod_or32(priv, meta, dst->lo, dst->lo, r32[0]); + } + break; + case sizeof(unsigned short): + if ((off % 4) =3D=3D 3) { + knod_iset32(&p32[0], 24); + knod_iset32(&p32[1], 8); + knod_bfe32(priv, meta, r64[0].lo, cache[off / 4], + p32[0], p32[1]); + knod_iset32(&p32[0], 0); + knod_bfe32(priv, meta, r64[0].hi, + cache[(off / 4) + 1], p32[0], p32[1]); + /* bpf_reg64[d].lo =3D (r64[0].hi << 8) | r64[0].lo. */ + knod_emit(priv, meta, v_lshl_or_b32, dst->lo, + r64[0].hi, p32[1], r64[0].lo); + } else { + if (!(off % 4)) + knod_iset32(&p32[0], 0); + else if ((off % 4) =3D=3D 1) + knod_iset32(&p32[0], 8); + else if ((off % 4) =3D=3D 2) + knod_iset32(&p32[0], 16); + knod_iset32(&p32[1], 16); + knod_bfe32(priv, meta, dst->lo, cache[off / 4], + p32[0], p32[1]); + } + break; + case sizeof(unsigned char): + if ((off % 4) =3D=3D 0) + knod_iset32(&p32[0], 0); + else if ((off % 4) =3D=3D 1) + knod_iset32(&p32[0], 8); + else if ((off % 4) =3D=3D 2) + knod_iset32(&p32[0], 16); + else + knod_iset32(&p32[0], 24); + knod_iset32(&p32[1], 8); + knod_bfe32(priv, meta, dst->lo, cache[off / 4], p32[0], + p32[1]); + break; + default: + WARN_ON_ONCE(1); + break; + } + + if (size !=3D sizeof(unsigned long)) { + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, dst->hi, p32[0]); + } +} + +/* + * GFX10 (RDNA2) quirk: global_load_{dword,dwordx2,dwordx4} silently + * clear the low 2 bits of the effective address, forcing Dword + * alignment. For PTR_TO_PACKET loads at a byte offset that is not + * Dword-aligned, round the offset down to the nearest 4-byte boundary, + * load enough contiguous dwords to cover the requested range, then use + * v_alignbit_b32 to extract the byte-aligned result. For size < 4 a + * final v_and_b32 masks the result to the correct width. + * + * Caller is responsible for zeroing dst.hi for size < 8; this helper + * only writes dst.lo (and dst.hi when size =3D=3D 8). + * + * Scratch: up to 4 contiguous VGPRs at v32..v35 + * (TMP_VREG5_LO..TMP_VREG6_HI). + */ +static void knod_bpf_emit_gfx10_unaligned_load(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int size, + struct amdgcn_param64 dst, + struct amdgcn_param32 src_lo, + int off) +{ + int off_a =3D off & ~3; + int shift_bits =3D (off - off_a) * 8; + int needed =3D DIV_ROUND_UP((off & 3) + size, 4); + struct amdgcn_param32 tmp[4]; + struct amdgcn_param32 shift_imm, mask_imm; + + knod_vset32(&tmp[0], KNOD_AMDGPU_TMP_VREG5_LO); + knod_vset32(&tmp[1], KNOD_AMDGPU_TMP_VREG5_HI); + knod_vset32(&tmp[2], KNOD_AMDGPU_TMP_VREG6_LO); + knod_vset32(&tmp[3], KNOD_AMDGPU_TMP_VREG6_HI); + knod_iset32(&shift_imm, shift_bits); + + if (needed <=3D 1) { + knod_emit(priv, meta, global_load_dword, tmp[0], src_lo, + off_a); + } else if (needed =3D=3D 2) { + knod_emit(priv, meta, global_load_dwordx2, tmp[0], src_lo, + off_a); + } else { + /* needed =3D=3D 3: no dwordx3, widen to dwordx4. */ + knod_emit(priv, meta, global_load_dwordx4, tmp[0], src_lo, + off_a); + } + knod_wait_vmcnt(priv, meta); + + if (size <=3D 4) { + /* v_alignbit_b32 D, S0, S1, S2: + * D =3D ({S0, S1} >> S2)[31:0] + * S0 is HIGH, S1 is LOW. tmp[0] holds + * bytes[off_a..+4) (memory-low) and tmp[1] holds + * bytes[off_a+4..+8) (memory-high), so + * src0=3Dtmp[1], src1=3Dtmp[0]. + */ + if (shift_bits =3D=3D 0) + knod_mov32(priv, meta, dst.lo, tmp[0]); + else + knod_alignbit32(priv, meta, dst.lo, + tmp[1], tmp[0], shift_imm); + + if (size =3D=3D 1) { + knod_iset32(&mask_imm, 0xff); + knod_and32(priv, meta, dst.lo, dst.lo, + mask_imm); + } else if (size =3D=3D 2) { + knod_iset32(&mask_imm, 0xffff); + knod_and32(priv, meta, dst.lo, dst.lo, + mask_imm); + } + } else { + /* size =3D=3D 8: two alignbits for low / high output dwords. */ + if (shift_bits =3D=3D 0) { + knod_mov32(priv, meta, dst.lo, tmp[0]); + knod_mov32(priv, meta, dst.hi, tmp[1]); + } else { + knod_alignbit32(priv, meta, dst.lo, + tmp[1], tmp[0], shift_imm); + knod_alignbit32(priv, meta, dst.hi, + tmp[2], tmp[1], shift_imm); + } + } +} + +#define LABEL_NEXT 8 +#define LABEL_OUT 9 +static void knod_bpf_ktime_get_ns(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + struct amdgcn_param32 p[2]; + + knod_sset32(&p[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, s_load_dwordx2, p[0], p[1], + offsetof(struct knod_bpf_param, ktime_ns)); + + knod_emit(priv, meta, s_waitcnt_lgkmcnt); + + knod_sset32(&p[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_mov32(priv, meta, bpf_reg64[0].lo, p[0]); + knod_sset32(&p[0], KNOD_AMDGPU_TMP_SREG0_HI); + knod_mov32(priv, meta, bpf_reg64[0].hi, p[0]); +} + +static void knod_bpf_map_lookup(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int map_id) +{ + struct knod_bpf_map_obj *knod_map_obj_k, *knod_map_obj_g; + int off, len, _len, idx, key_in_pkt, key_in_map; + bool first_cmp; + struct amdgcn_branch_fixup fixups[12] =3D {0,}; + struct amdgcn_label labels[10] =3D {0,}; + u32 stack_off =3D meta->kreg.stack_off; + unsigned long bucket_gaddr; + struct amdgcn_param32 p32; + int fixup_idx =3D 0; + + knod_map_obj_k =3D + (struct knod_bpf_map_obj *)knod_bpf_map_kaddr(priv, map_id); + knod_map_obj_g =3D + (struct knod_bpf_map_obj *)knod_bpf_map_gaddr(priv, map_id); + bucket_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, bucket); + + knod_jit_dbg(" stack_off =3D %d map_id =3D %d\n", stack_off, map_id); + if (!knod_map_obj_g || !knod_map_obj_k) + WARN_ON_ONCE(1); + + knod_bpf_load_size(priv, meta, + &r64[2], + &stack[0], + sizeof(unsigned int), + 512 + stack_off); + /* reg1 :=3D bucket + * NOTE: bucket_gaddr is greater than X + */ + knod_iset64(&p64[0], bucket_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[1], 0); + knod_mov32(priv, meta, r64[2].hi, p64[1].lo); + + knod_iset64(&p64[1], 0); + knod_mov64(priv, meta, bpf_reg64[0], p64[1]); + + /* BPF_REG0 =3D 0 + * TMP_REG1 =3D bucket_gaddr + * TMP_REG2 =3D key + */ + + if (knod_map_obj_k->map_type =3D=3D BPF_MAP_TYPE_ARRAY || + knod_map_obj_k->map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + /* if (key > knod_map_obj_k.max_entries) + * NOTE: integer + */ + knod_iset64(&p64[1], knod_map_obj_k->max_entries); + knod_mov64(priv, meta, r64[3], p64[1]); + knod_emit(priv, meta, v_cmp_ge_u64, r64[2], r64[3]); + /* structurized CFG: save OOB lanes, narrow exec */ + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); /* update required */ + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_OUT], meta->amdgpu_insns); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + /* PERCPU_ARRAY: bucket +=3D workgroup_id_y * per_instance_size so + * each RX queue addresses its own instance and the atomic + * update after the lookup has no cross-CU contention. Auto + * xgroups keeps PERCPU programs at one workgroup per queue, so + * the queue id (workgroup_id_y) is the instance index. + */ + if (knod_map_obj_k->map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + /* r64[3] is scratch after the bounds check: .lo =3D + * workgroup_id_y, .hi =3D per_instance_size (too large + * for an inline constant, so stage both in VGPRs + * first). + */ + knod_sset32(&p32, KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_emit(priv, meta, v_mov_b32_e32, r64[3].lo, p32); + knod_iset64(&p64[1], + knod_map_obj_k->meta.ameta + .per_instance_size); + knod_mov32(priv, meta, r64[3].hi, p64[1].lo); + emit_v_mad_u64_u32(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + r64[1], + sr64[0].lo, + r64[3].hi, /* per_instance_size */ + r64[3].lo, /* workgroup_id_y */ + r64[1]); /* bucket */ + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + } + /* elem_id =3D &bucket[key]; */ + knod_iset64(&p64[1], knod_map_obj_k->value_size); + emit_v_mad_u64_u32(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + bpf_reg64[0], + sr64[0].lo, + p64[1].lo, /* value_size */ + r64[2].lo, /* key */ + r64[1]); /* bucket */ + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + /* structurized CFG: restore OOB lanes */ + knod_bpf_set_label(meta, &labels[LABEL_OUT], + meta->amdgpu_insns); + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); + } else if (knod_map_obj_k->map_type =3D=3D BPF_MAP_TYPE_HASH) { + key_in_pkt =3D KEY_IN_PKT_64; + len =3D knod_map_obj_k->key_size; + off =3D stack_off; + + /* TMP_VREGs(vgpr-pair) + * |0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18| + * | | | |K|K|K|K|K|K|K|K |K |K |K |K |K |K |K |K | + */ + while (len) { + if (len >=3D sizeof(unsigned long)) + _len =3D sizeof(unsigned long); + else + _len =3D len; + knod_bpf_load_size(priv, meta, + &r64[key_in_pkt], + &stack[0], + _len, + 512 + off); + key_in_pkt++; + len -=3D _len; + off +=3D _len; + } + + knod_jhash(priv, meta, + 2, + knod_map_obj_k->key_size, + knod_map_obj_k->meta.hmeta.hashrnd); + /* clear hi register of r64[2] because hash is 32bit */ + knod_iset64(&p64[0], 0); + knod_mov32(priv, meta, r64[2].hi, p64[0].lo); + + /* hash =3D hash & (n_buckets - 1) before indexing the bucket + * array -- jhash returns the full 32-bit hash and the update + * and delete emitters mask it too; without this + * bucket_gaddr[hash] runs off the end of the bucket array. + */ + knod_iset32(&p64[0].lo, + knod_map_obj_k->meta.hmeta.n_buckets - 1); + knod_and32(priv, meta, r64[2].lo, p64[0].lo, r64[2].lo); + + /* elem_id =3D bucket_gaddr[hash]; */ + knod_iset64(&p64[1], sizeof(int)); + emit_v_mad_u64_u32(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + r64[2], + sr64[0].lo, + p64[1].lo, /* sizeof(int) */ + r64[2].lo, /* hash */ + r64[1]); /* bucket */ + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + /* bpf_reg64[0] =3D 0 (default return for not-found lanes) */ + knod_iset64(&p64[0], 0); + knod_mov64(priv, meta, bpf_reg64[0], p64[0]); + + /* structurized CFG: save initial exec for restoring at end */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO); + + knod_bpf_set_label(meta, &labels[LABEL_NEXT], + meta->amdgpu_insns); + /* load elem_id from elem structure */ + knod_emit(priv, meta, global_load_dword, r64[2].lo, + r64[2].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* mask out DELETED bit from next field */ + knod_iset32(&p32, KNOD_BPF_HASH_NEXT_MASK); + knod_emit(priv, meta, v_and_b32_e32, r64[2].lo, p32, + r64[2].lo); + + /* if (r64[2].lo =3D=3D KNOD_BPF_HASH_NEXT_END) + * goto out; + * VOPC cannot encode literal constants - move to VGPR first. + * NEXT_MASK =3D=3D NEXT_END (0x7FFFFFFF), reuse p32 from v_and + * above. + */ + knod_emit(priv, meta, v_mov_b32_e32, r64[0].hi, p32); + knod_emit(priv, meta, v_cmp_eq_u32, r64[0].hi, r64[2].lo); + /* structurized CFG: remove end-of-chain lanes */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); /* update required */ + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_OUT], meta->amdgpu_insns); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + knod_iset64(&p64[0], + (unsigned long)knod_map_obj_k->meta.hmeta.elems); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[0], knod_map_obj_k->meta.hmeta.elem_size); + knod_mov64(priv, meta, r64[0], p64[0]); + + key_in_map =3D KEY_IN_MAP_32; + len =3D knod_map_obj_k->key_size; + off =3D offsetof(struct knod_bpf_hash_elem_obj, kv); + + /* elem =3D &elems[elem_id]; */ + emit_v_mad_u64_u32(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + r64[2], /* elem */ + sr64[0].lo, + r64[0].lo, /* elem_size */ + r64[2].lo, /* elem_id */ + r64[1]); /* elem_gaddr */ + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + /* load elem.next for DELETED check (parallel with key loads) */ + knod_emit(priv, meta, global_load_dword, r64[0].hi, + r64[2].lo, 0); + while (len >=3D 16) { + knod_emit(priv, meta, global_load_dwordx4, + r32[key_in_map], + r64[2].lo, /* elem ptr */ off); + off +=3D 16; + len -=3D 16; + key_in_map +=3D 4; + } + + if (len >=3D 8) { + knod_emit(priv, meta, global_load_dwordx2, + r32[key_in_map], + /* elem_id */ r64[2].lo, /* elem ptr */ off); + off +=3D 8; + len -=3D 8; + key_in_map +=3D 2; + } + + if (len >=3D 4) { + knod_emit(priv, meta, global_load_dword, + r32[key_in_map], + /* elem_id */ r64[2].lo, /* elem ptr */ off); + off +=3D 4; + len -=3D 4; + key_in_map +=3D 1; + } + + /* map key padding was inited to zero, no AND is required */ + if (len) { + knod_emit(priv, meta, global_load_dword, + r32[key_in_map], + /* elem_id */ r64[2].lo, /* elem ptr */ off); + } + + knod_wait_vmcnt(priv, meta); + + /* structurized CFG: accumulate key match into TMP_SREG4 + * instead of early-exit branching per key part + */ + key_in_map =3D KEY_IN_MAP_32; + key_in_pkt =3D KEY_IN_PKT_32; + len =3D knod_map_obj_k->key_size; + first_cmp =3D true; + + while (len >=3D 8) { + knod_emit(priv, meta, v_cmp_eq_u64, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_VCC_LO); + } + + key_in_map +=3D 2; + key_in_pkt +=3D 2; + len -=3D 8; + } + + if (len >=3D 4) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_VCC_LO); + } + + key_in_map +=3D 1; + key_in_pkt +=3D 1; + len -=3D 4; + } + + if (len) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG4_LO, + KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_VCC_LO); + } + } + + /* DELETED check: remove deleted lanes from match result. + * r64[0].hi =3D elem.next (loaded in parallel with key). + * Deleted elems have bit 31 set - exclude them from SREG4. + */ + knod_iset32(&p32, KNOD_BPF_HASH_NEXT_DELETED); + knod_emit(priv, meta, v_and_b32_e32, r64[0].hi, p32, + r64[0].hi); + knod_iset32(&p32, 0); + knod_emit(priv, meta, v_cmp_eq_u32, p32, r64[0].hi); + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG4_LO, + KNOD_AMDGPU_TMP_SREG4_LO, AMDGCN_SREG_VCC_LO); + + /* TMP_SREG4 =3D lanes where key matched AND not deleted. + * Save current exec, narrow to matched lanes for value + * computation. + */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG5_LO, + AMDGCN_SREG_EXEC_LO); + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG4_LO); + + /* bpf_reg64[0] =3D value address (only for matched lanes) */ + knod_iset64(&p64[1], + offsetof(struct knod_bpf_hash_elem_obj, kv) + + knod_map_obj_k->key_size); + knod_add64(priv, meta, bpf_reg64[0], p64[1], r64[2]); + + /* set exec to unmatched lanes for next loop iteration */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG5_LO, KNOD_AMDGPU_TMP_SREG4_LO); + /* loop back if any unmatched lanes remain */ + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); /* update required */ + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_NEXT], meta->amdgpu_insns); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + /* structurized CFG: restore all original lanes */ + knod_bpf_set_label(meta, &labels[LABEL_OUT], + meta->amdgpu_insns); + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); + + } else { + WARN_ON_ONCE(1); + } +} + +static void knod_bpf_map_update_array(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int map_id) +{ + struct knod_bpf_map_obj *knod_map_obj_k, *knod_map_obj_g; + struct amdgcn_branch_fixup fixups[4] =3D {0,}; + u32 key_stack_off =3D meta->kreg.stack_off; + u32 val_stack_off =3D meta->vreg.stack_off; + struct amdgcn_label labels[10] =3D {0,}; + int idx, val_off, val_len; + unsigned long bucket_gaddr; + int fixup_idx =3D 0; + + knod_map_obj_k =3D + (struct knod_bpf_map_obj *)knod_bpf_map_kaddr(priv, map_id); + knod_map_obj_g =3D + (struct knod_bpf_map_obj *)knod_bpf_map_gaddr(priv, map_id); + bucket_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, bucket); + + if (!knod_map_obj_g || !knod_map_obj_k) + WARN_ON_ONCE(1); + + /* load key from stack -> r64[2] */ + knod_bpf_load_size(priv, meta, + &r64[2], + &stack[0], + sizeof(unsigned int), + 512 + key_stack_off); + + /* r64[1] =3D bucket_gaddr */ + knod_iset64(&p64[0], bucket_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + /* clear r64[2].hi (key is 32-bit) */ + knod_iset64(&p64[1], 0); + knod_mov32(priv, meta, r64[2].hi, p64[1].lo); + + /* bpf_reg64[0] =3D 0 (return value) */ + knod_mov64(priv, meta, bpf_reg64[0], p64[1]); + + /* bounds check: if (key >=3D max_entries) -> skip */ + knod_iset64(&p64[1], knod_map_obj_k->max_entries); + knod_mov64(priv, meta, r64[3], p64[1]); + knod_emit(priv, meta, v_cmp_ge_u64, r64[2], r64[3]); + + /* structurized CFG: save OOB lanes, narrow exec */ + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_OUT], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* dest =3D bucket_gaddr + key * value_size -> r64[0] */ + knod_iset64(&p64[1], knod_map_obj_k->value_size); + knod_emit(priv, meta, v_mad_u64_u32, r64[0], sr64[0].lo, + p64[1].lo, r64[2].lo, r64[1]); + + /* load value from stack and store to dest */ + val_off =3D 0; + val_len =3D knod_map_obj_k->value_size; + + while (val_len >=3D 16) { + knod_bpf_load_size(priv, meta, + &r64[3], + &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_bpf_load_size(priv, meta, + &r64[4], + &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off + 8); + knod_emit(priv, meta, global_store_dwordx4, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 16; + val_len -=3D 16; + } + + if (val_len >=3D 8) { + knod_bpf_load_size(priv, meta, + &r64[3], + &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dwordx2, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 8; + val_len -=3D 8; + } + + if (val_len >=3D 4) { + knod_bpf_load_size(priv, meta, + &r64[3], + &stack[0], + sizeof(unsigned int), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dword, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 4; + val_len -=3D 4; + } + + if (val_len >=3D 2) { + knod_bpf_load_size(priv, meta, + &r64[3], + &stack[0], + sizeof(unsigned short), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_short, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 2; + val_len -=3D 2; + } + + if (val_len >=3D 1) { + knod_bpf_load_size(priv, meta, + &r64[3], + &stack[0], + sizeof(unsigned char), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_byte, r64[3].lo, + r64[0].lo, val_off); + } + + /* structurized CFG: restore OOB lanes */ + knod_bpf_set_label(meta, &labels[LABEL_OUT], meta->amdgpu_insns); + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); +} + +static void knod_bpf_map_update_hash(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int map_id) +{ +#define LABEL_BUCKET_LOOP 0 +#define LABEL_LOCK_RETRY 1 +#define LABEL_INSERT_LANE 2 +#define LABEL_CHAIN_NEXT 3 +#define LABEL_ALLOC_INSERT 4 +#define LABEL_LANE_DONE 5 +#define LABEL_UNLOCK 6 + struct knod_bpf_map_obj *knod_map_obj_k, *knod_map_obj_g; + struct amdgcn_param32 s_bucket_lo, v_tmp, v_zero, v_one; + int off, len, _len, idx, key_in_pkt, key_in_map; + struct amdgcn_param32 s_exec_lo, s_exec_hi, s_elem_id; + struct amdgcn_branch_fixup fixups[12] =3D {0,}; + u32 key_stack_off =3D meta->kreg.stack_off; + u32 val_stack_off =3D meta->vreg.stack_off; + unsigned long queue_gaddr, elems_gaddr; + unsigned long bucket_gaddr, cur_gaddr; + struct amdgcn_label labels[12] =3D {0,}; + struct amdgcn_param32 v_minus_one; + struct amdgcn_param64 sr64_carry; + struct amdgcn_param32 p32; + unsigned long lock_offset; + unsigned int elem_size; + int val_off, val_len; + int fixup_idx =3D 0; + bool first_cmp; + int koff, voff; + + knod_map_obj_k =3D + (struct knod_bpf_map_obj *)knod_bpf_map_kaddr(priv, map_id); + knod_map_obj_g =3D + (struct knod_bpf_map_obj *)knod_bpf_map_gaddr(priv, map_id); + bucket_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, bucket); + cur_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, meta.hmeta.cur); + queue_gaddr =3D (unsigned long)knod_map_obj_k->meta.hmeta.q; + elems_gaddr =3D (unsigned long)knod_map_obj_k->meta.hmeta.elems; + elem_size =3D knod_map_obj_k->meta.hmeta.elem_size; + + if (!knod_map_obj_g || !knod_map_obj_k) + WARN_ON_ONCE(1); + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 1: Setup =3D=3D=3D=3D=3D=3D=3D=3D */ + + /* Load key from stack -> r64[3..9] (KEY_IN_PKT) */ + key_in_pkt =3D KEY_IN_PKT_64; + len =3D knod_map_obj_k->key_size; + off =3D key_stack_off; + while (len) { + if (len >=3D sizeof(unsigned long)) + _len =3D sizeof(unsigned long); + else + _len =3D len; + knod_bpf_load_size(priv, meta, + &r64[key_in_pkt], + &stack[0], + _len, + 512 + off); + key_in_pkt++; + len -=3D _len; + off +=3D _len; + } + + /* jhash -> r64[2].lo =3D hash */ + knod_jhash(priv, meta, + 2, + knod_map_obj_k->key_size, + knod_map_obj_k->meta.hmeta.hashrnd); + knod_iset64(&p64[0], 0); + knod_mov32(priv, meta, r64[2].hi, p64[0].lo); + + /* hash =3D hash & (n_buckets - 1) */ + knod_iset32(&p64[0].lo, + knod_map_obj_k->meta.hmeta.n_buckets - 1); + knod_and32(priv, meta, r64[2].lo, p64[0].lo, r64[2].lo); + + /* r64[1] =3D bucket_gaddr */ + knod_iset64(&p64[0], bucket_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + /* bucket_addr =3D bucket_gaddr + hash * sizeof(int) -> r64[2] */ + knod_iset64(&p64[1], sizeof(int)); + knod_emit(priv, meta, v_mad_u64_u32, r64[2], sr64[0].lo, + p64[1].lo, r64[2].lo, r64[1]); + + /* Save bucket_addr to r64[15] for CAS insert */ + knod_mov64(priv, meta, r64[15], r64[2]); + + /* SREG3 =3D initial exec */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO); + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 2: Sequential per-lane processing =3D= =3D=3D=3D=3D=3D=3D=3D */ + + /* SREG5 =3D exec (all lanes to process, for BUCKET_LOOP) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG5_LO, + AMDGCN_SREG_EXEC_LO); + + /* ---- BUCKET_LOOP: process one unique bucket per iteration ---- */ + knod_bpf_set_label(meta, &labels[LABEL_BUCKET_LOOP], + meta->amdgpu_insns); + + lock_offset =3D + (unsigned long)knod_map_obj_k->meta.hmeta.n_buckets * + sizeof(unsigned int); + + knod_sset32(&s_bucket_lo, + KNOD_AMDGPU_TMP_SREG1_HI); + knod_vset32(&v_tmp, KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(&v_zero, 0); + knod_iset32(&v_one, 1); + knod_sset32(&s_exec_lo, AMDGCN_SREG_EXEC_LO); + knod_sset32(&s_exec_hi, + AMDGCN_SREG_EXEC_LO + 1); + knod_sset32(&s_elem_id, + KNOD_AMDGPU_TMP_SREG1_LO); + knod_sset64(&sr64_carry, + KNOD_AMDGPU_TMP_SREG1_LO); + + /* Pick first active lane's bucket addr */ + knod_emit(priv, meta, v_readfirstlane_b32, KNOD_AMDGPU_TMP_SREG1_HI, + r64[15].lo.v); + + /* vcc =3D lanes with same bucket */ + knod_emit(priv, meta, v_cmp_eq_u32, s_bucket_lo, r64[15].lo); + + /* SREG5 =3D remaining lanes; exec =3D same-bucket lanes */ + knod_emit(priv, meta, s_and_saveexec_b64, KNOD_AMDGPU_TMP_SREG5_LO, + AMDGCN_SREG_VCC_LO); + + /* SREG0 =3D same-bucket lanes */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_EXEC_LO); + + /* ---- Lock acquire ---- */ + /* r64[10] =3D r64[15] + lock_offset */ + knod_iset64(&p64[0], lock_offset); + knod_add64(priv, meta, r64[10], p64[0], r64[15]); + + /* r64[11].lo =3D 1 (swap data) */ + knod_emit(priv, meta, v_mov_b32_e32, r64[11].lo, v_one); + + /* First-lane isolation via mbcnt */ + knod_emit(priv, meta, v_mbcnt_lo_u32_b32, v_tmp, s_exec_lo, + v_zero); + knod_emit(priv, meta, v_mbcnt_hi_u32_b32, v_tmp, s_exec_hi, v_tmp); + knod_emit(priv, meta, v_cmp_eq_u32, v_zero, v_tmp); + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* LOCK_RETRY: spin until lock acquired */ + knod_bpf_set_label(meta, &labels[LABEL_LOCK_RETRY], meta->amdgpu_insns); + + knod_emit(priv, meta, global_atomic_swap, r64[11].hi, r64[10].lo, + r64[11].lo, 0, 1); + knod_wait_vmcnt(priv, meta); + + meta->amdgpu_insn[meta->amdgpu_insns].size =3D + emit_gfx10_v_cmp_ne_u32( + &meta->amdgpu_insn[meta->amdgpu_insns].gfx10, + v_zero, r64[11].hi); + meta->amdgpu_insn[meta->amdgpu_insns].type =3D AMDGCN_INSN_TYPE_VOPC; + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + emit_s_cbranch_vccnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LOCK_RETRY], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* Lock acquired - restore same-bucket lanes */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + + /* ---- INSERT_LANE: process one lane at a time ---- */ + knod_bpf_set_label(meta, &labels[LABEL_INSERT_LANE], + meta->amdgpu_insns); + + /* SREG4 =3D exec (remaining same-bucket lanes) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_EXEC_LO); + + /* Pick first active lane via mbcnt */ + knod_emit(priv, meta, v_mbcnt_lo_u32_b32, v_tmp, s_exec_lo, + v_zero); + knod_emit(priv, meta, v_mbcnt_hi_u32_b32, v_tmp, s_exec_hi, v_tmp); + knod_emit(priv, meta, v_cmp_eq_u32, v_zero, v_tmp); + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* SREG2 =3D exec (single-lane mask) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG2_LO, + AMDGCN_SREG_EXEC_LO); + + /* r64[2] =3D r64[15] (bucket_addr for chain walk start) */ + knod_mov64(priv, meta, r64[2], r64[15]); + + /* ---- CHAIN_NEXT: walk chain ---- */ + knod_bpf_set_label(meta, &labels[LABEL_CHAIN_NEXT], meta->amdgpu_insns); + + knod_emit(priv, meta, global_load_dword, r64[0].lo, r64[2].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* Mask out DELETED bit */ + knod_iset32(&p32, KNOD_BPF_HASH_NEXT_MASK); + knod_emit(priv, meta, v_and_b32_e32, r64[0].lo, p32, r64[0].lo); + + /* End-of-chain check (VOPC literal workaround) */ + knod_emit(priv, meta, v_mov_b32_e32, r64[0].hi, p32); + knod_emit(priv, meta, v_cmp_eq_u32, r64[0].hi, r64[0].lo); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_ALLOC_INSERT], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* elem_addr =3D elems + elem_id * elem_size -> r64[2] */ + knod_iset64(&p64[0], elems_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[0], elem_size); + knod_mov32(priv, meta, r64[10].lo, p64[0].lo); + knod_emit(priv, meta, v_mad_u64_u32, r64[2], sr64_carry.lo, + r64[10].lo, r64[0].lo, r64[1]); + + /* Load elem.next for DELETED check */ + knod_emit(priv, meta, global_load_dword, r64[0].hi, r64[2].lo, 0); + + /* Load key from map element -> KEY_IN_MAP */ + key_in_map =3D KEY_IN_MAP_32; + len =3D knod_map_obj_k->key_size; + off =3D offsetof(struct knod_bpf_hash_elem_obj, kv); + + while (len >=3D 16) { + knod_emit(priv, meta, global_load_dwordx4, r32[key_in_map], + r64[2].lo, off); + off +=3D 16; + len -=3D 16; + key_in_map +=3D 4; + } + + if (len >=3D 8) { + knod_emit(priv, meta, global_load_dwordx2, r32[key_in_map], + r64[2].lo, off); + off +=3D 8; + len -=3D 8; + key_in_map +=3D 2; + } + + if (len >=3D 4) { + knod_emit(priv, meta, global_load_dword, r32[key_in_map], + r64[2].lo, off); + off +=3D 4; + len -=3D 4; + key_in_map +=3D 1; + } + + if (len) { + knod_emit(priv, meta, global_load_dword, r32[key_in_map], + r64[2].lo, off); + } + + knod_wait_vmcnt(priv, meta); + + /* Key comparison -> SREG1 */ + key_in_map =3D KEY_IN_MAP_32; + key_in_pkt =3D KEY_IN_PKT_32; + len =3D knod_map_obj_k->key_size; + first_cmp =3D true; + + while (len >=3D 8) { + knod_emit(priv, meta, v_cmp_eq_u64, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + key_in_map +=3D 2; + key_in_pkt +=3D 2; + len -=3D 8; + } + + if (len >=3D 4) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + key_in_map +=3D 1; + key_in_pkt +=3D 1; + len -=3D 4; + } + + if (len) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + } + + /* DELETED check: SREG1 &=3D not_deleted */ + knod_iset32(&p32, + KNOD_BPF_HASH_NEXT_DELETED); + knod_emit(priv, meta, v_and_b32_e32, r64[0].hi, p32, r64[0].hi); + knod_iset32(&p32, 0); + knod_emit(priv, meta, v_cmp_eq_u32, p32, r64[0].hi); + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, AMDGCN_SREG_VCC_LO); + + /* Narrow exec to matched lane */ + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG1_LO); + + /* Value overwrite for matched lane (exec-masked, skipped if no + * match) + */ + knod_iset64(&p64[1], + offsetof(struct knod_bpf_hash_elem_obj, + kv) + + knod_map_obj_k->key_size); + knod_add64(priv, meta, r64[0], p64[1], r64[2]); + + val_off =3D 0; + val_len =3D knod_map_obj_k->value_size; + + while (val_len >=3D 16) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_bpf_load_size(priv, meta, + &r64[11], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off + 8); + knod_emit(priv, meta, global_store_dwordx4, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 16; + val_len -=3D 16; + } + + if (val_len >=3D 8) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dwordx2, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 8; + val_len -=3D 8; + } + + if (val_len >=3D 4) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned int), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dword, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 4; + val_len -=3D 4; + } + + if (val_len >=3D 2) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned short), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_short, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 2; + val_len -=3D 2; + } + + if (val_len >=3D 1) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned char), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_byte, r64[10].lo, + r64[0].lo, val_off); + } + + /* If matched, done with this lane */ + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* No match: restore lane, continue chain walk */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG2_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_CHAIN_NEXT], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* ---- ALLOC_INSERT: key not found, insert new elem ---- */ + knod_bpf_set_label(meta, &labels[LABEL_ALLOC_INSERT], + meta->amdgpu_insns); + + /* Restore single-lane exec */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG2_LO); + + /* Alloc from free pool: atomic_add(cur, -1) */ + knod_iset32(&v_minus_one, -1); + knod_emit(priv, meta, v_mov_b32_e32, r64[11].lo, v_minus_one); + + knod_iset64(&p64[0], cur_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + knod_emit(priv, meta, global_atomic_add, r64[11].lo, r64[1].lo, + r64[11].lo, 0, 1); + knod_wait_vmcnt(priv, meta); + + /* my_cur =3D old_cur - 1 -> r64[0].lo */ + knod_emit(priv, meta, v_mov_b32_e32, r64[0].lo, v_one); + knod_emit(priv, meta, v_sub_u32, r64[0].lo, r64[11].lo, r64[0].lo); + + /* OOM check: if (my_cur < 0) -> skip insert */ + knod_emit(priv, meta, v_cmp_gt_i32, v_zero, r64[0].lo); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* queue_addr =3D queue_gaddr + my_cur * 4 -> r64[1] */ + knod_iset64(&p64[0], queue_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[1], sizeof(unsigned int)); + knod_emit(priv, meta, v_mad_u64_u32, r64[1], sr64_carry.lo, + p64[1].lo, r64[0].lo, r64[1]); + + /* elem_id =3D queue[my_cur] -> r64[0].lo */ + knod_emit(priv, meta, global_load_dword, r64[0].lo, r64[1].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* new_elem_addr =3D elems + elem_id * elem_size -> r64[2] */ + knod_iset64(&p64[0], elems_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[0], elem_size); + knod_mov32(priv, meta, r64[10].lo, p64[0].lo); + knod_emit(priv, meta, v_mad_u64_u32, r64[2], sr64_carry.lo, + r64[10].lo, r64[0].lo, r64[1]); + + /* Save elem_id to SGPR (v_mad carry already done) */ + knod_emit(priv, meta, v_readfirstlane_b32, KNOD_AMDGPU_TMP_SREG1_LO, + r64[0].lo.v); + + /* Load current bucket head -> r64[1].lo */ + knod_emit(priv, meta, global_load_dword, r64[1].lo, r64[15].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* new_elem.next =3D old_head */ + knod_emit(priv, meta, global_store_dword, r64[1].lo, r64[2].lo, 0); + + /* Write key to new element */ + koff =3D offsetof(struct knod_bpf_hash_elem_obj, kv); + + key_in_pkt =3D KEY_IN_PKT_32; + len =3D knod_map_obj_k->key_size; + + while (len >=3D 8) { + knod_emit(priv, meta, global_store_dwordx2, r32[key_in_pkt], + r64[2].lo, koff); + koff +=3D 8; + len -=3D 8; + key_in_pkt +=3D 2; + } + + if (len >=3D 4) { + knod_emit(priv, meta, global_store_dword, r32[key_in_pkt], + r64[2].lo, koff); + koff +=3D 4; + len -=3D 4; + key_in_pkt +=3D 1; + } + + if (len >=3D 2) { + knod_emit(priv, meta, global_store_short, r32[key_in_pkt], + r64[2].lo, koff); + koff +=3D 2; + len -=3D 2; + } + + if (len >=3D 1) { + knod_emit(priv, meta, global_store_byte, r32[key_in_pkt], + r64[2].lo, koff); + } + + /* Write value to new element */ + voff =3D offsetof(struct knod_bpf_hash_elem_obj, kv) + + knod_map_obj_k->key_size; + + val_off =3D 0; + val_len =3D knod_map_obj_k->value_size; + + knod_iset64(&p64[1], voff); + knod_add64(priv, meta, r64[0], p64[1], r64[2]); + + while (val_len >=3D 16) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_bpf_load_size(priv, meta, + &r64[11], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off + 8); + knod_emit(priv, meta, global_store_dwordx4, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 16; + val_len -=3D 16; + } + + if (val_len >=3D 8) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned long), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dwordx2, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 8; + val_len -=3D 8; + } + + if (val_len >=3D 4) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned int), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_dword, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 4; + val_len -=3D 4; + } + + if (val_len >=3D 2) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned short), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_short, r64[10].lo, + r64[0].lo, val_off); + val_off +=3D 2; + val_len -=3D 2; + } + + if (val_len >=3D 1) { + knod_bpf_load_size(priv, meta, + &r64[10], &stack[0], + sizeof(unsigned char), + 512 + val_stack_off + val_off); + knod_emit(priv, meta, global_store_byte, r64[10].lo, + r64[0].lo, val_off); + } + + knod_wait_vmcnt(priv, meta); + + /* Update bucket[hash] =3D new elem_id */ + knod_emit(priv, meta, v_mov_b32_e32, r64[1].lo, s_elem_id); + knod_emit(priv, meta, global_store_dword, r64[1].lo, r64[15].lo, + 0); + knod_wait_vmcnt(priv, meta); + + /* ---- LANE_DONE: remove this lane, next lane ---- */ + knod_bpf_set_label(meta, &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG4_LO, KNOD_AMDGPU_TMP_SREG2_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_INSERT_LANE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* ---- UNLOCK: release lock + next bucket ---- */ + knod_bpf_set_label(meta, &labels[LABEL_UNLOCK], meta->amdgpu_insns); + + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + + /* Recompute lock_addr (r64[10] clobbered) */ + knod_iset64(&p64[0], lock_offset); + knod_add64(priv, meta, r64[10], p64[0], r64[15]); + + knod_emit(priv, meta, v_mov_b32_e32, r64[1].lo, v_zero); + knod_emit(priv, meta, global_store_dword, r64[1].lo, r64[10].lo, + 0); + knod_wait_vmcnt(priv, meta); + + /* Next bucket */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG5_LO, KNOD_AMDGPU_TMP_SREG0_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_BUCKET_LOOP], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 5: Restore =3D=3D=3D=3D=3D=3D=3D=3D */ + + knod_bpf_set_label(meta, &labels[LABEL_OUT], meta->amdgpu_insns); + /* exec =3D SREG3 (restore all original lanes) */ + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + + /* bpf_reg64[0] =3D 0 (return value for all lanes) */ + knod_iset64(&p64[0], 0); + knod_mov64(priv, meta, bpf_reg64[0], p64[0]); + + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); + + return; +#undef LABEL_BUCKET_LOOP +#undef LABEL_LOCK_RETRY +#undef LABEL_INSERT_LANE +#undef LABEL_CHAIN_NEXT +#undef LABEL_ALLOC_INSERT +#undef LABEL_LANE_DONE +#undef LABEL_UNLOCK +} + +static void knod_bpf_map_delete_hash(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int map_id) +{ +#define LABEL_BUCKET_LOOP 0 +#define LABEL_LOCK_RETRY 1 +#define LABEL_DELETE_LANE 2 +#define LABEL_CHAIN_NEXT 3 +#define LABEL_LANE_DONE 4 +#define LABEL_UNLOCK 5 + struct knod_bpf_map_obj *knod_map_obj_k, *knod_map_obj_g; + struct amdgcn_param32 s_bucket_lo, v_tmp, v_zero, v_one; + int off, len, _len, idx, key_in_pkt, key_in_map; + struct amdgcn_branch_fixup fixups[12] =3D {0,}; + unsigned long bucket_gaddr, gc_count_gaddr; + struct amdgcn_param32 s_exec_lo, s_exec_hi; + unsigned long gc_list_gaddr, elems_gaddr; + u32 key_stack_off =3D meta->kreg.stack_off; + struct amdgcn_label labels[12] =3D {0,}; + struct amdgcn_param64 sr64_carry; + struct amdgcn_param32 v_del; + struct amdgcn_param32 p32; + unsigned long lock_offset; + unsigned int elem_size; + int fixup_idx =3D 0; + bool first_cmp; + + knod_map_obj_k =3D + (struct knod_bpf_map_obj *)knod_bpf_map_kaddr(priv, map_id); + knod_map_obj_g =3D + (struct knod_bpf_map_obj *)knod_bpf_map_gaddr(priv, map_id); + bucket_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, bucket); + gc_count_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, meta.hmeta.gc_count); + gc_list_gaddr =3D (unsigned long)knod_map_obj_k->meta.hmeta.gc_list; + elems_gaddr =3D (unsigned long)knod_map_obj_k->meta.hmeta.elems; + elem_size =3D knod_map_obj_k->meta.hmeta.elem_size; + + if (!knod_map_obj_g || !knod_map_obj_k) + WARN_ON_ONCE(1); + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 1: Setup =3D=3D=3D=3D=3D=3D=3D=3D */ + + /* Load key from stack -> r64[3..9] (KEY_IN_PKT) */ + key_in_pkt =3D KEY_IN_PKT_64; + len =3D knod_map_obj_k->key_size; + off =3D key_stack_off; + while (len) { + if (len >=3D sizeof(unsigned long)) + _len =3D sizeof(unsigned long); + else + _len =3D len; + knod_bpf_load_size(priv, meta, + &r64[key_in_pkt], + &stack[0], + _len, + 512 + off); + key_in_pkt++; + len -=3D _len; + off +=3D _len; + } + + /* jhash -> r64[2].lo =3D hash */ + knod_jhash(priv, meta, + 2, + knod_map_obj_k->key_size, + knod_map_obj_k->meta.hmeta.hashrnd); + knod_iset64(&p64[0], 0); + knod_mov32(priv, meta, r64[2].hi, p64[0].lo); + + /* hash =3D hash & (n_buckets - 1) */ + knod_iset32(&p64[0].lo, + knod_map_obj_k->meta.hmeta.n_buckets - 1); + knod_and32(priv, meta, r64[2].lo, p64[0].lo, r64[2].lo); + + /* r64[1] =3D bucket_gaddr */ + knod_iset64(&p64[0], bucket_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + /* bucket_addr =3D bucket_gaddr + hash * sizeof(int) -> r64[2] */ + knod_iset64(&p64[1], sizeof(int)); + knod_emit(priv, meta, v_mad_u64_u32, r64[2], sr64[0].lo, + p64[1].lo, r64[2].lo, r64[1]); + + /* Save bucket_addr to r64[15] */ + knod_mov64(priv, meta, r64[15], r64[2]); + + /* SREG3 =3D initial exec */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO); + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 2: Sequential per-lane processing =3D= =3D=3D=3D=3D=3D=3D=3D */ + + /* SREG5 =3D exec (all lanes to process, for BUCKET_LOOP) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG5_LO, + AMDGCN_SREG_EXEC_LO); + + /* ---- BUCKET_LOOP: process one unique bucket per iteration ---- */ + knod_bpf_set_label(meta, &labels[LABEL_BUCKET_LOOP], + meta->amdgpu_insns); + + lock_offset =3D + (unsigned long)knod_map_obj_k->meta.hmeta.n_buckets * + sizeof(unsigned int); + + knod_sset32(&s_bucket_lo, + KNOD_AMDGPU_TMP_SREG1_HI); + knod_vset32(&v_tmp, KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(&v_zero, 0); + knod_iset32(&v_one, 1); + knod_sset32(&s_exec_lo, AMDGCN_SREG_EXEC_LO); + knod_sset32(&s_exec_hi, + AMDGCN_SREG_EXEC_LO + 1); + knod_sset64(&sr64_carry, + KNOD_AMDGPU_TMP_SREG1_LO); + + /* Pick first active lane's bucket addr */ + knod_emit(priv, meta, v_readfirstlane_b32, KNOD_AMDGPU_TMP_SREG1_HI, + r64[15].lo.v); + + /* vcc =3D lanes with same bucket */ + knod_emit(priv, meta, v_cmp_eq_u32, s_bucket_lo, r64[15].lo); + + /* SREG5 =3D remaining lanes; exec =3D same-bucket lanes */ + knod_emit(priv, meta, s_and_saveexec_b64, KNOD_AMDGPU_TMP_SREG5_LO, + AMDGCN_SREG_VCC_LO); + + /* SREG0 =3D same-bucket lanes */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_EXEC_LO); + + /* ---- Lock acquire ---- */ + /* r64[10] =3D r64[15] + lock_offset */ + knod_iset64(&p64[0], lock_offset); + knod_add64(priv, meta, r64[10], p64[0], r64[15]); + + /* r64[11].lo =3D 1 (swap data) */ + knod_emit(priv, meta, v_mov_b32_e32, r64[11].lo, v_one); + + /* First-lane isolation via mbcnt */ + knod_emit(priv, meta, v_mbcnt_lo_u32_b32, v_tmp, s_exec_lo, + v_zero); + knod_emit(priv, meta, v_mbcnt_hi_u32_b32, v_tmp, s_exec_hi, v_tmp); + knod_emit(priv, meta, v_cmp_eq_u32, v_zero, v_tmp); + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* LOCK_RETRY: spin until lock acquired */ + knod_bpf_set_label(meta, &labels[LABEL_LOCK_RETRY], meta->amdgpu_insns); + + knod_emit(priv, meta, global_atomic_swap, r64[11].hi, r64[10].lo, + r64[11].lo, 0, 1); + knod_wait_vmcnt(priv, meta); + + meta->amdgpu_insn[meta->amdgpu_insns].size =3D + emit_gfx10_v_cmp_ne_u32( + &meta->amdgpu_insn[meta->amdgpu_insns].gfx10, + v_zero, r64[11].hi); + meta->amdgpu_insn[meta->amdgpu_insns].type =3D AMDGCN_INSN_TYPE_VOPC; + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + emit_s_cbranch_vccnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LOCK_RETRY], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* Lock acquired - restore same-bucket lanes */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + + /* ---- DELETE_LANE: process one lane at a time ---- */ + knod_bpf_set_label(meta, &labels[LABEL_DELETE_LANE], + meta->amdgpu_insns); + + /* SREG4 =3D exec (remaining same-bucket lanes) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG4_LO, + AMDGCN_SREG_EXEC_LO); + + /* Pick first active lane via mbcnt */ + knod_emit(priv, meta, v_mbcnt_lo_u32_b32, v_tmp, s_exec_lo, + v_zero); + knod_emit(priv, meta, v_mbcnt_hi_u32_b32, v_tmp, s_exec_hi, v_tmp); + knod_emit(priv, meta, v_cmp_eq_u32, v_zero, v_tmp); + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* SREG2 =3D exec (single-lane mask) */ + knod_emit(priv, meta, s_mov_b64, KNOD_AMDGPU_TMP_SREG2_LO, + AMDGCN_SREG_EXEC_LO); + + /* r64[2] =3D r64[15] (bucket_addr for chain walk start) */ + knod_mov64(priv, meta, r64[2], r64[15]); + + /* ---- CHAIN_NEXT: walk chain ---- */ + knod_bpf_set_label(meta, &labels[LABEL_CHAIN_NEXT], meta->amdgpu_insns); + + knod_emit(priv, meta, global_load_dword, r64[0].lo, r64[2].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* Mask out DELETED bit */ + knod_iset32(&p32, KNOD_BPF_HASH_NEXT_MASK); + knod_emit(priv, meta, v_and_b32_e32, r64[0].lo, p32, r64[0].lo); + + /* End-of-chain check (VOPC literal workaround) */ + knod_emit(priv, meta, v_mov_b32_e32, r64[0].hi, p32); + knod_emit(priv, meta, v_cmp_eq_u32, r64[0].hi, r64[0].lo); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* elem_addr =3D elems + elem_id * elem_size -> r64[2] */ + knod_iset64(&p64[0], elems_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_iset64(&p64[0], elem_size); + knod_mov32(priv, meta, r64[10].lo, p64[0].lo); + knod_emit(priv, meta, v_mad_u64_u32, r64[2], sr64_carry.lo, + r64[10].lo, r64[0].lo, r64[1]); + + /* Load elem.next for DELETED check */ + knod_emit(priv, meta, global_load_dword, r64[0].hi, r64[2].lo, 0); + + /* Load key from map element -> KEY_IN_MAP */ + key_in_map =3D KEY_IN_MAP_32; + len =3D knod_map_obj_k->key_size; + off =3D offsetof(struct knod_bpf_hash_elem_obj, kv); + + while (len >=3D 16) { + knod_emit(priv, meta, global_load_dwordx4, r32[key_in_map], + r64[2].lo, off); + off +=3D 16; + len -=3D 16; + key_in_map +=3D 4; + } + + if (len >=3D 8) { + knod_emit(priv, meta, global_load_dwordx2, r32[key_in_map], + r64[2].lo, off); + off +=3D 8; + len -=3D 8; + key_in_map +=3D 2; + } + + if (len >=3D 4) { + knod_emit(priv, meta, global_load_dword, r32[key_in_map], + r64[2].lo, off); + off +=3D 4; + len -=3D 4; + key_in_map +=3D 1; + } + + if (len) { + knod_emit(priv, meta, global_load_dword, r32[key_in_map], + r64[2].lo, off); + } + + knod_wait_vmcnt(priv, meta); + + /* Key comparison -> SREG1 */ + key_in_map =3D KEY_IN_MAP_32; + key_in_pkt =3D KEY_IN_PKT_32; + len =3D knod_map_obj_k->key_size; + first_cmp =3D true; + + while (len >=3D 8) { + knod_emit(priv, meta, v_cmp_eq_u64, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + key_in_map +=3D 2; + key_in_pkt +=3D 2; + len -=3D 8; + } + + if (len >=3D 4) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + + key_in_map +=3D 1; + key_in_pkt +=3D 1; + len -=3D 4; + } + + if (len) { + knod_emit(priv, meta, v_cmp_eq_u32, r32[key_in_map], + r32[key_in_pkt]); + + if (first_cmp) { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + first_cmp =3D false; + } else { + emit_s_and_b64(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_VCC_LO); + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + } + + /* DELETED check: SREG1 &=3D not_deleted */ + knod_iset32(&p32, + KNOD_BPF_HASH_NEXT_DELETED); + knod_emit(priv, meta, v_and_b32_e32, r64[0].hi, p32, r64[0].hi); + knod_iset32(&p32, 0); + knod_emit(priv, meta, v_cmp_eq_u32, p32, r64[0].hi); + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG1_LO, + KNOD_AMDGPU_TMP_SREG1_LO, AMDGCN_SREG_VCC_LO); + + /* Narrow exec to matched lane */ + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG1_LO); + + /* ---- Match path: set DELETED + append to gc_list ---- */ + + /* atomic_or(elem.next, DELETED_BIT) - mark deleted */ + knod_lset32(&v_del, + KNOD_BPF_HASH_NEXT_DELETED); + knod_emit(priv, meta, v_mov_b32_e32, r64[11].lo, v_del); + knod_emit(priv, meta, global_atomic_or, r64[11].hi, r64[2].lo, + r64[11].lo, 0, 0); + + /* atomic_add(gc_count, 1) -> old_count in r64[11].lo */ + knod_emit(priv, meta, v_mov_b32_e32, r64[11].lo, v_one); + + knod_iset64(&p64[0], gc_count_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + knod_emit(priv, meta, global_atomic_add, r64[11].lo, r64[1].lo, + r64[11].lo, 0, 1); + knod_wait_vmcnt(priv, meta); + + /* Store elem_id to gc_list[old_count] */ + knod_iset64(&p64[0], gc_list_gaddr); + knod_mov64(priv, meta, r64[10], p64[0]); + + knod_iset64(&p64[0], sizeof(unsigned int)); + knod_emit(priv, meta, v_mad_u64_u32, r64[1], sr64_carry.lo, + p64[0].lo, r64[11].lo, r64[10]); + + knod_emit(priv, meta, global_store_dword, r64[0].lo, r64[1].lo, 0); + knod_wait_vmcnt(priv, meta); + + /* If matched, done with this lane */ + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* No match: restore lane, continue chain walk */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG2_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_CHAIN_NEXT], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* ---- LANE_DONE: remove this lane, next lane ---- */ + knod_bpf_set_label(meta, &labels[LABEL_LANE_DONE], meta->amdgpu_insns); + + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG4_LO, KNOD_AMDGPU_TMP_SREG2_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_DELETE_LANE], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* ---- UNLOCK: release lock + next bucket ---- */ + knod_bpf_set_label(meta, &labels[LABEL_UNLOCK], meta->amdgpu_insns); + + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + + /* Recompute lock_addr (r64[10] clobbered) */ + knod_iset64(&p64[0], lock_offset); + knod_add64(priv, meta, r64[10], p64[0], r64[15]); + + knod_emit(priv, meta, v_mov_b32_e32, r64[1].lo, v_zero); + knod_emit(priv, meta, global_store_dword, r64[1].lo, r64[10].lo, + 0); + knod_wait_vmcnt(priv, meta); + + /* Next bucket */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG5_LO, KNOD_AMDGPU_TMP_SREG0_LO); + + emit_s_cbranch_execnz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_BUCKET_LOOP], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* =3D=3D=3D=3D=3D=3D=3D=3D Phase 3: Restore =3D=3D=3D=3D=3D=3D=3D=3D */ + + /* exec =3D SREG3 (restore all original lanes) */ + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + + /* bpf_reg64[0] =3D 0 (return value) */ + knod_iset64(&p64[0], 0); + knod_mov64(priv, meta, bpf_reg64[0], p64[0]); + + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); + + return; +#undef LABEL_BUCKET_LOOP +#undef LABEL_LOCK_RETRY +#undef LABEL_DELETE_LANE +#undef LABEL_CHAIN_NEXT +#undef LABEL_LANE_DONE +#undef LABEL_UNLOCK +} + +static void knod_bpf_map_delete_array(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + int map_id) +{ + struct knod_bpf_map_obj *knod_map_obj_k, *knod_map_obj_g; + struct amdgcn_branch_fixup fixups[4] =3D {0,}; + u32 key_stack_off =3D meta->kreg.stack_off; + struct amdgcn_label labels[10] =3D {0,}; + int idx, val_off, val_len; + struct amdgcn_param32 v_zero; + unsigned long bucket_gaddr; + int fixup_idx =3D 0; + + knod_map_obj_k =3D + (struct knod_bpf_map_obj *)knod_bpf_map_kaddr(priv, map_id); + knod_map_obj_g =3D + (struct knod_bpf_map_obj *)knod_bpf_map_gaddr(priv, map_id); + bucket_gaddr =3D (unsigned long)knod_map_obj_g + + offsetof(struct knod_bpf_map_obj, bucket); + + if (!knod_map_obj_g || !knod_map_obj_k) + WARN_ON_ONCE(1); + + /* load key from stack -> r64[2] */ + knod_bpf_load_size(priv, meta, + &r64[2], + &stack[0], + sizeof(unsigned int), + 512 + key_stack_off); + + /* r64[1] =3D bucket_gaddr */ + knod_iset64(&p64[0], bucket_gaddr); + knod_mov64(priv, meta, r64[1], p64[0]); + + /* clear r64[2].hi (key is 32-bit) */ + knod_iset64(&p64[1], 0); + knod_mov32(priv, meta, r64[2].hi, p64[1].lo); + + /* bpf_reg64[0] =3D 0 (return value) */ + knod_mov64(priv, meta, bpf_reg64[0], p64[1]); + + /* bounds check: if (key >=3D max_entries) -> skip */ + knod_iset64(&p64[1], knod_map_obj_k->max_entries); + knod_mov64(priv, meta, r64[3], p64[1]); + knod_emit(priv, meta, v_cmp_ge_u64, r64[2], r64[3]); + + /* structurized CFG: save OOB lanes, narrow exec */ + knod_emit(priv, meta, s_and_b64, KNOD_AMDGPU_TMP_SREG3_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + emit_s_cbranch_execz(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + 0); + knod_bpf_set_fixup(meta, &fixups[fixup_idx], + &labels[LABEL_OUT], meta->amdgpu_insns); + debug_insn(priv->isa_version, &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + fixup_idx++; + + /* dest =3D bucket_gaddr + key * value_size -> r64[0] */ + knod_iset64(&p64[1], knod_map_obj_k->value_size); + knod_emit(priv, meta, v_mad_u64_u32, r64[0], sr64[0].lo, + p64[1].lo, r64[2].lo, r64[1]); + + /* Zero out value at dest */ + knod_iset32(&v_zero, 0); + knod_emit(priv, meta, v_mov_b32_e32, r64[3].lo, v_zero); + knod_emit(priv, meta, v_mov_b32_e32, r64[3].hi, v_zero); + knod_emit(priv, meta, v_mov_b32_e32, r64[4].lo, v_zero); + knod_emit(priv, meta, v_mov_b32_e32, r64[4].hi, v_zero); + + val_off =3D 0; + val_len =3D knod_map_obj_k->value_size; + + while (val_len >=3D 16) { + knod_emit(priv, meta, global_store_dwordx4, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 16; + val_len -=3D 16; + } + + if (val_len >=3D 8) { + knod_emit(priv, meta, global_store_dwordx2, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 8; + val_len -=3D 8; + } + + if (val_len >=3D 4) { + knod_emit(priv, meta, global_store_dword, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 4; + val_len -=3D 4; + } + + if (val_len >=3D 2) { + knod_emit(priv, meta, global_store_short, r64[3].lo, + r64[0].lo, val_off); + val_off +=3D 2; + val_len -=3D 2; + } + + if (val_len >=3D 1) { + knod_emit(priv, meta, global_store_byte, r64[3].lo, + r64[0].lo, val_off); + } + + knod_wait_vmcnt(priv, meta); + + /* structurized CFG: restore OOB lanes */ + knod_bpf_set_label(meta, &labels[LABEL_OUT], meta->amdgpu_insns); + knod_emit(priv, meta, s_or_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, KNOD_AMDGPU_TMP_SREG3_LO); + + for (idx =3D 0; idx < fixup_idx; idx++) + knod_bpf_fixup_branch(priv, &fixups[idx]); +} + +static void knod_bpf_store_cache_size(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct amdgcn_param64 *src, + /* packet or stack */ + struct amdgcn_param32 *cache, + int size, int off) +{ + struct amdgcn_param32 p32[2]; + + knod_jit_dbg(" %d: off =3D %d off_4 =3D %d size =3D %d\n", meta->bpf_insn= _idx, + off, off%4, size); + WARN_ON(knod_param_is_literal(src->lo) || + knod_param_is_literal(src->hi)); + switch (size) { + case sizeof(unsigned long): + if ((off % 4) =3D=3D 0) { + knod_mov32(priv, meta, + cache[off / 4], + src->lo); + knod_mov32(priv, meta, + cache[(off / 4) + 1], + src->hi); + } else if ((off % 4) =3D=3D 1) { + WARN_ON_ONCE(1); + } else if ((off % 4) =3D=3D 2) { + WARN_ON_ONCE(1); + } else { + WARN_ON_ONCE(1); + } + break; + case sizeof(unsigned int): + if ((off % 4) =3D=3D 0) { + knod_mov32(priv, meta, + cache[off / 4], + src->lo); + } else if ((off % 4) =3D=3D 1) { + knod_iset64(&p64[0], 8); + knod_lshlrev64(priv, meta, r64[0], p64[0], *src); + + knod_iset32(&p32[0], 0xffffff00); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], + r32[2], r64[0].lo, cache[off / 4]); + knod_iset32(&p32[0], 0x000000ff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[(off / 4) + 1], r32[2], + r64[0].hi, cache[(off / 4) + 1]); + } else if ((off % 4) =3D=3D 2) { + knod_iset64(&p64[0], 16); + knod_lshlrev64(priv, meta, r64[0], p64[0], *src); + + knod_iset32(&p32[0], 0xffff0000); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + r64[0].lo, cache[off / 4]); + knod_iset32(&p32[0], 0x0000ffff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[(off / 4) + 1], r32[2], + r64[0].hi, cache[(off / 4) + 1]); + } else { + knod_iset64(&p64[0], 24); + knod_lshlrev64(priv, meta, r64[0], p64[0], *src); + + knod_iset32(&p32[0], 0xffff0000); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + r64[0].lo, cache[off / 4]); + knod_iset32(&p32[0], 0x00ffffff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[(off / 4) + 1], r32[2], + r64[0].hi, cache[(off / 4) + 1]); + } + break; + case sizeof(unsigned short): + if ((off % 4) =3D=3D 0) { + knod_iset32(&p32[0], 0x0000ffff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + src->lo, cache[off / 4]); + } else if ((off % 4) =3D=3D 1) { + knod_iset32(&p32[0], 8); + knod_lshlrev32(priv, meta, r32[0], p32[0], + src->lo); + knod_iset32(&p32[0], 0x00ffff00); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + r32[0], cache[off / 4]); + } else if ((off % 4) =3D=3D 2) { + knod_iset32(&p32[0], 16); + knod_lshlrev32(priv, meta, r32[0], p32[0], + src->lo); + knod_iset32(&p32[0], 0xffff0000); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + r32[0], cache[off / 4]); + } else { + knod_iset64(&p64[0], 24); + knod_lshlrev64(priv, meta, r64[0], p64[0], *src); + + knod_iset32(&p32[0], 0xff000000); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + r64[0].lo, cache[off / 4]); + knod_iset32(&p32[0], 0x000000ff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[(off / 4) + 1], r32[2], + r64[0].hi, cache[(off / 4) + 1]); + } + break; + case sizeof(unsigned char): + if ((off % 4) =3D=3D 0) { + knod_iset32(&p32[0], 0x000000ff); + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], + src->lo, cache[off / 4]); + return; + } else if ((off % 4) =3D=3D 1) { + knod_iset32(&p32[0], 8); + knod_lshlrev32(priv, meta, r32[0], p32[0], + src->lo); + + knod_iset32(&p32[0], 0x0000ff00); + } else if ((off % 4) =3D=3D 2) { + knod_iset32(&p32[0], 16); + knod_lshlrev32(priv, meta, r32[0], p32[0], + src->lo); + + knod_iset32(&p32[0], 0x00ff0000); + } else { + knod_iset32(&p32[0], 24); + knod_lshlrev32(priv, meta, r32[0], p32[0], + src->lo); + knod_iset32(&p32[0], 0xff000000); + } + + knod_mov32(priv, meta, r32[2], p32[0]); + knod_bfi32(priv, meta, cache[off / 4], r32[2], r32[0], + cache[off / 4]); + break; + default: + WARN_ON_ONCE(1); + } +} + +static bool knod_meta_is_exit(const struct knod_insn_meta *meta); +static bool knod_bpf_is_retval_move_to_r0(const struct knod_insn_meta *met= a); + +/* + * knod_bpf_emit_branch_tail - Emit EXEC mask manipulation after v_cmp for + * structurized per-lane branching. Replaces the old s_cbranch_vccnz/vccz. + * + * For FORWARD_SKIP: + * Save jumping lanes -> narrow EXEC -> s_cbranch_execz + * (skip if no active lanes) + * + * For DIRECT_EXIT: + * Compute exit lanes -> update done_mask -> remove from EXEC (no branch) + * + * Emits the required EXEC mask manipulation in-place. + */ +static void knod_bpf_emit_direct_exit_retval(struct knod_bpf_priv *priv, + struct knod_insn_meta *emit_meta, + struct knod_insn_meta *target) +{ + struct amdgcn_param64 dst, src; + s64 imm; + + if (!target || knod_meta_is_exit(target)) + return; + + if (WARN_ON_ONCE(!knod_bpf_is_retval_move_to_r0(target))) + return; + + knod_vset64(&dst, KNOD_AMDGPU_VREG0_LO); + + switch (target->insn.code) { + case BPF_ALU | BPF_MOV | BPF_X: + case BPF_ALU64 | BPF_MOV | BPF_X: + knod_vset64(&src, target->insn.src_reg * 2); + knod_mov64(priv, emit_meta, dst, src); + break; + case BPF_ALU | BPF_MOV | BPF_K: + imm =3D (u32)target->insn.imm; + knod_iset64(&src, imm); + knod_mov64(priv, emit_meta, dst, src); + break; + case BPF_ALU64 | BPF_MOV | BPF_K: + imm =3D (s64)(s32)target->insn.imm; + knod_iset64(&src, imm); + knod_mov64(priv, emit_meta, dst, src); + break; + default: + WARN_ON_ONCE(1); + break; + } +} + +static void knod_bpf_emit_branch_tail(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct knod_prog *knod_prog, + short off) +{ + switch (meta->branch_type) { + case KNOD_BR_FORWARD_SKIP: + if (meta->jump_neg_op) { + /* JNE: VCC=3D0 -> jump, VCC=3D1 -> fall-through. + * Save jump lanes (VCC=3D0): s[n] =3D exec & ~vcc + */ + knod_emit(priv, meta, s_andn2_b64, + meta->exec_save_sreg, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + + /* Keep fall-through (VCC=3D1): exec =3D exec & vcc */ + knod_emit(priv, meta, s_and_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + } else { + /* Normal: VCC=3D1 -> jump, VCC=3D0 -> fall-through. + * Save jump lanes (VCC=3D1): s[n] =3D exec & vcc + */ + knod_emit(priv, meta, s_and_b64, + meta->exec_save_sreg, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + + /* Keep fall-through (VCC=3D0): exec =3D exec & ~vcc */ + knod_emit(priv, meta, s_andn2_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + } + + /* + * No GPU branch. After the RPO reorder, branch scopes + * interleave, so the jumping lanes must flow through every + * following block under the EXEC mask and rejoin at their merge + * point. An s_cbranch_execz skipping ahead to the merge would + * jump over other scopes' merge points and strand their saved + * lanes (EXEC never restored -> act=3D0). + */ + break; + + case KNOD_BR_DIRECT_EXIT: + if (meta->jump_neg_op) { + /* JNE: VCC=3D0 -> exit. exit_lanes =3D exec & ~vcc */ + knod_emit(priv, meta, s_andn2_b64, + KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + } else { + /* Normal: VCC=3D1 -> exit. exit_lanes =3D exec & vcc */ + knod_emit(priv, meta, s_and_b64, + KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_VCC_LO); + } + + /* Keep the lanes that did not take the exit path. */ + knod_emit(priv, meta, s_andn2_b64, + KNOD_AMDGPU_TMP_SREG1_LO, + AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + + /* Replay a shared "r0 =3D action; exit" target under the + * exiting lanes before marking them done. Otherwise a direct + * branch to the common exit can publish stale r0 scratch state. + */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + knod_bpf_emit_direct_exit_retval(priv, meta, meta->merge_point); + + /* done_mask |=3D exit_lanes */ + knod_emit(priv, meta, s_or_b64, + knod_prog->done_mask_sreg, + knod_prog->done_mask_sreg, + AMDGCN_SREG_EXEC_LO); + + /* Continue with the non-exit lanes. */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG1_LO); + + /* No branch - fall through with reduced EXEC. + * No fixup needed. + */ + meta->jmp_dst =3D NULL; + break; + + default: + WARN_ON_ONCE(1); + break; + } +} + +/* + * --- Basic-block CFG analysis (foundation for block reordering) --- + * + * The emitter is a linear SIMT machine: instructions run in list order un= der + * an EXEC mask. A *forward* jump is realized by masking off the jumping + * lanes and restoring them at the merge point. A *backward* jump has no = such + * realization unless it is a loop (real GPU branch + EXEC convergence, no= t yet + * implemented). + * + * LLVM tail-sharing and block placement routinely emit jumps that are + * backward in BPF byte order but are NOT loops - e.g. a UDP bounds check = that + * jumps back to a shared XDP_PASS tail. Classifying those as "exit" (the + * jmp_off < 0 heuristic in knod_bpf_analyze_cfg) silently miscompiles the= m: + * the jumping lanes exit carrying whatever R0 happened to hold instead of + * flowing to the real target. + * + * The fix is to classify by control-flow, not byte order: + * 1. partition the instruction stream into basic blocks, + * 2. build the control-flow graph (successor edges), + * 3. DFS for a reverse-postorder (RPO) and detect back-edges, + * 4. no back-edges (a DAG) -> reorder blocks into RPO so every edge po= ints + * forward, then classify by linear position, + * 5. a real loop is present -> bail (-EOPNOTSUPP) until loop emission l= ands. + * + * Loop emission (step 5) is not implemented yet, so programs containing a + * loop are rejected with -EOPNOTSUPP. + */ +struct knod_bb { + struct knod_insn_meta *leader; /* first instruction of the block */ + struct knod_insn_meta *last; /* last instruction of the block */ + /* successors: [0] not-taken, [1] taken */ + struct knod_bb *succ[2]; + int n_succ; + /* reverse-postorder rank, -1 if unreachable */ + int rpo; + /* DFS color: 0 white, 1 gray, 2 black */ + int dfs; + bool loop_header; /* target of a back-edge */ + /* scratch: member of the loop being walked */ + bool in_loop; + /* immediate dominator (self for entry) */ + struct knod_bb *idom; +}; + +static bool knod_meta_is_exit(const struct knod_insn_meta *meta) +{ + u8 code =3D meta->insn.code; + + return code =3D=3D (BPF_JMP | BPF_EXIT) || code =3D=3D (BPF_JMP32 | BPF_E= XIT); +} + +static struct knod_insn_meta * +knod_bpf_next_meta(struct knod_prog *knod_prog, struct knod_insn_meta *met= a) +{ + if (!meta || list_is_last(&meta->l, &knod_prog->insns)) + return NULL; + + return list_next_entry(meta, l); +} + +static bool knod_bpf_is_retval_move_to_r0(const struct knod_insn_meta *met= a) +{ + u8 code; + + if (!meta || meta->insn.dst_reg !=3D BPF_REG_0) + return false; + + code =3D meta->insn.code; + return code =3D=3D (BPF_ALU | BPF_MOV | BPF_X) || + code =3D=3D (BPF_ALU64 | BPF_MOV | BPF_X) || + code =3D=3D (BPF_ALU | BPF_MOV | BPF_K) || + code =3D=3D (BPF_ALU64 | BPF_MOV | BPF_K); +} + +static bool knod_bpf_is_direct_exit_target(struct knod_prog *knod_prog, + struct knod_insn_meta *target) +{ + if (knod_meta_is_exit(target)) + return true; + + if (!knod_bpf_is_retval_move_to_r0(target)) + return false; + + return knod_meta_is_exit(knod_bpf_next_meta(knod_prog, target)); +} + +static bool knod_meta_is_ja(const struct knod_insn_meta *meta) +{ + u8 code =3D meta->insn.code; + + return code =3D=3D (BPF_JMP | BPF_JA | BPF_K) || + code =3D=3D (BPF_JMP32 | BPF_JA | BPF_K); +} + +/* A block ends after a terminator; the next instruction starts a new bloc= k. */ +static bool knod_meta_is_terminator(const struct knod_insn_meta *meta) +{ + return is_mbpf_cond_jump(meta) || knod_meta_is_ja(meta) || + knod_meta_is_exit(meta); +} + +/* Target instruction index of a conditional jump or BPF_JA. */ +static short knod_meta_jump_target_idx(const struct knod_insn_meta *meta) +{ + if (meta->insn.code =3D=3D (BPF_JMP32 | BPF_JA | BPF_K)) + return meta->bpf_insn_idx + meta->insn.imm + 1; + return meta->bpf_insn_idx + meta->insn.off + 1; +} + +static struct knod_bb *knod_bb_of_leader(struct knod_bb *bbs, int n_bbs, + const struct knod_insn_meta *meta) +{ + int i; + + for (i =3D 0; i < n_bbs; i++) + if (bbs[i].leader =3D=3D meta) + return &bbs[i]; + return NULL; +} + +/* Resolve the block a conditional jump / BPF_JA at @jmp transfers to. */ +static struct knod_bb *knod_bb_jump_target(struct knod_prog *knod_prog, + struct knod_bb *bbs, int n_bbs, + const struct knod_insn_meta *jmp) +{ + struct knod_insn_meta *tgt; + + tgt =3D knod_bpf_lookup_meta(knod_prog, knod_meta_jump_target_idx(jmp)); + return tgt ? knod_bb_of_leader(bbs, n_bbs, tgt) : NULL; +} + +/* + * Partition knod_prog->insns into basic blocks. A leader is the first + * instruction, any jump target, or the instruction after a terminator. + * Returns the block count or a negative errno; @bbs holds >=3D n_insns bl= ocks. + */ +static int knod_bpf_build_bbs(struct knod_prog *knod_prog, struct knod_bb = *bbs) +{ + struct knod_insn_meta *meta, *tgt; + struct knod_bb *cur =3D NULL; + int n_bbs =3D 0; + short tgt_idx; + + /* Pass A: mark every jump target as a leader. */ + list_for_each_entry(meta, &knod_prog->insns, l) + meta->flags &=3D ~FLAG_INSN_IS_JUMP_DST; + + list_for_each_entry(meta, &knod_prog->insns, l) { + if (!is_mbpf_cond_jump(meta) && !knod_meta_is_ja(meta)) + continue; + tgt_idx =3D knod_meta_jump_target_idx(meta); + tgt =3D knod_bpf_lookup_meta(knod_prog, tgt_idx); + if (!tgt) { + pr_warn("knod_cfg: bpf#%d jump target %d unresolved\n", + meta->bpf_insn_idx, tgt_idx); + return -EINVAL; + } + tgt->flags |=3D FLAG_INSN_IS_JUMP_DST; + } + + /* Pass B: cut the list into blocks. */ + list_for_each_entry(meta, &knod_prog->insns, l) { + if (!cur || (meta->flags & FLAG_INSN_IS_JUMP_DST)) { + cur =3D &bbs[n_bbs++]; + cur->leader =3D meta; + cur->n_succ =3D 0; + } + cur->last =3D meta; + + if (knod_meta_is_terminator(meta)) + /* next instruction starts a new block */ + cur =3D NULL; + } + + return n_bbs; +} + +/* Build successor edges for every block from its terminator. */ +static int knod_bpf_build_edges(struct knod_prog *knod_prog, + struct knod_bb *bbs, int n_bbs) +{ + struct knod_bb *bb, *fall, *tgt_bb; + struct knod_insn_meta *last; + int i; + + for (i =3D 0; i < n_bbs; i++) { + bb =3D &bbs[i]; + last =3D bb->last; + bb->n_succ =3D 0; + + if (knod_meta_is_exit(last)) + continue; /* no successors */ + + /* Successor in list order: block led by the next + * instruction. + */ + fall =3D NULL; + if (!list_is_last(&last->l, &knod_prog->insns)) + fall =3D knod_bb_of_leader(bbs, n_bbs, + list_next_entry(last, l)); + + if (is_mbpf_cond_jump(last)) { + tgt_bb =3D knod_bb_jump_target(knod_prog, bbs, n_bbs, + last); + if (!fall || !tgt_bb) + return -EINVAL; + bb->succ[bb->n_succ++] =3D fall; /* not taken */ + bb->succ[bb->n_succ++] =3D tgt_bb; /* taken */ + } else if (knod_meta_is_ja(last)) { + tgt_bb =3D knod_bb_jump_target(knod_prog, bbs, n_bbs, + last); + if (!tgt_bb) + return -EINVAL; + bb->succ[bb->n_succ++] =3D tgt_bb; + } else { + if (!fall) /* fell off the end */ + return -EINVAL; + bb->succ[bb->n_succ++] =3D fall; + } + } + + return 0; +} + +/* + * Iterative DFS from the entry block. Computes a reverse-postorder rank = for + * every reachable block and flags back-edge targets as loop headers. Ret= urns + * the number of back-edges in *n_back, or a negative errno. + */ +static int knod_bpf_compute_rpo(struct knod_bb *bbs, int n_bbs, + struct knod_bb *entry, int *n_back) +{ + struct knod_bb **stack; + int *cursor; + int top =3D 0, post =3D 0, nb =3D 0, i; + + for (i =3D 0; i < n_bbs; i++) { + bbs[i].dfs =3D 0; + bbs[i].rpo =3D -1; + bbs[i].loop_header =3D false; + } + + stack =3D kcalloc(n_bbs, sizeof(*stack), GFP_KERNEL); + cursor =3D kcalloc(n_bbs, sizeof(*cursor), GFP_KERNEL); + if (!stack || !cursor) { + kfree(stack); + kfree(cursor); + return -ENOMEM; + } + + entry->dfs =3D 1; + stack[top] =3D entry; + cursor[top] =3D 0; + top++; + + while (top > 0) { + struct knod_bb *bb =3D stack[top - 1]; + + if (cursor[top - 1] < bb->n_succ) { + struct knod_bb *s =3D bb->succ[cursor[top - 1]++]; + + if (s->dfs =3D=3D 0) { /* tree edge */ + s->dfs =3D 1; + stack[top] =3D s; + cursor[top] =3D 0; + top++; + } else if (s->dfs =3D=3D 1) { /* gray -> back-edge */ + s->loop_header =3D true; + nb++; + } + /* s->dfs =3D=3D 2 -> forward/cross edge, nothing to do */ + } else { + /* finished: postorder */ + bb->dfs =3D 2; + bb->rpo =3D post++; + top--; + } + } + + /* postorder -> reverse-postorder rank */ + for (i =3D 0; i < n_bbs; i++) + if (bbs[i].rpo >=3D 0) + bbs[i].rpo =3D post - 1 - bbs[i].rpo; + + kfree(stack); + kfree(cursor); + *n_back =3D nb; + return 0; +} + +/* + * Cooper-Harvey-Kennedy dominator intersect: walk the two fingers up the = idom + * chain (toward the entry, which has the lowest RPO) until they meet. + */ +static struct knod_bb *knod_dom_intersect(struct knod_bb *a, struct knod_b= b *b) +{ + while (a !=3D b) { + while (a->rpo > b->rpo) + a =3D a->idom; + while (b->rpo > a->rpo) + b =3D b->idom; + } + return a; +} + +/* + * Compute the immediate dominator of every reachable block (Cooper, Harve= y, + * Kennedy, "A Simple, Fast Dominance Algorithm"). Iterates over RPO to a + * fixpoint; bb->idom is the block's immediate dominator, the entry domina= ting + * itself. Requires bb->rpo from knod_bpf_compute_rpo. + */ +static int knod_bpf_compute_dom(struct knod_bb *bbs, int n_bbs, + struct knod_bb *entry) +{ + struct knod_bb **order; + int i, k, n_order =3D 0; + bool changed; + + order =3D kcalloc(n_bbs, sizeof(*order), GFP_KERNEL); + if (!order) + return -ENOMEM; + + for (i =3D 0; i < n_bbs; i++) { + bbs[i].idom =3D NULL; + if (bbs[i].rpo >=3D 0) { + order[bbs[i].rpo] =3D &bbs[i]; + n_order++; + } + } + entry->idom =3D entry; + + do { + changed =3D false; + + /* process every reachable block but the entry, in RPO order */ + for (k =3D 1; k < n_order; k++) { + struct knod_bb *n =3D order[k]; + struct knod_bb *new_idom =3D NULL; + int b, s; + + /* intersect over already-processed predecessors */ + for (b =3D 0; b < n_bbs; b++) { + for (s =3D 0; s < bbs[b].n_succ; s++) { + if (bbs[b].succ[s] !=3D n || !bbs[b].idom) + continue; + new_idom =3D new_idom ? + knod_dom_intersect(&bbs[b], + new_idom) : + &bbs[b]; + } + } + + if (new_idom && n->idom !=3D new_idom) { + n->idom =3D new_idom; + changed =3D true; + } + } + } while (changed); + + kfree(order); + return 0; +} + +/* Does block @a dominate block @b? Walk @b up the idom chain to the entr= y. */ +static bool knod_dom_dominates(struct knod_bb *a, struct knod_bb *b) +{ + for (;;) { + if (b =3D=3D a) + return true; + if (b->idom =3D=3D b) /* reached the entry */ + return false; + b =3D b->idom; + } +} + +/* + * Mark the natural loop body of back-edge @latch->@hdr in bb->in_loop: the + * header plus every block that reaches the latch without passing through = the + * header, found by walking predecessors back from the latch. @stack is + * caller-provided scratch of at least @n_bbs entries. + */ +static void knod_loop_mark_body(struct knod_bb *bbs, int n_bbs, + struct knod_bb *latch, struct knod_bb *hdr, + struct knod_bb **stack) +{ + int b, sp, k, top =3D 0; + + for (k =3D 0; k < n_bbs; k++) + bbs[k].in_loop =3D false; + + hdr->in_loop =3D true; + if (latch !=3D hdr) { + latch->in_loop =3D true; + stack[top++] =3D latch; + } + + while (top > 0) { + struct knod_bb *d =3D stack[--top]; + + for (b =3D 0; b < n_bbs; b++) { + if (bbs[b].in_loop) + continue; + for (sp =3D 0; sp < bbs[b].n_succ; sp++) { + if (bbs[b].succ[sp] !=3D d) + continue; + bbs[b].in_loop =3D true; + stack[top++] =3D &bbs[b]; + break; + } + } + } +} + +/* + * Detect natural loops from the dominator tree and report their structure. + * + * A back-edge is an edge u->v whose target v dominates its source u - v is + * the loop header, u the latch. Its natural loop body is the header plus= the + * blocks that reach the latch without passing through the header; an exit= edge + * leaves a body block for a non-body block. + * + * Loops are still rejected by the reorder (-EOPNOTSUPP); this only report= s what + * was found (to dmesg, since a rejected program never attaches so /bpf/cf= g is + * unavailable) so the detection can be verified before emission is built. + */ +static int knod_bpf_detect_loops(struct knod_bb *bbs, int n_bbs) +{ + struct knod_bb **stack; + int u, s, k, n_be =3D 0; + + stack =3D kcalloc(n_bbs, sizeof(*stack), GFP_KERNEL); + if (!stack) + return -ENOMEM; + + for (u =3D 0; u < n_bbs; u++) { + for (s =3D 0; s < bbs[u].n_succ; s++) { + struct knod_bb *hdr =3D bbs[u].succ[s]; + int body =3D 0, exits =3D 0, sp; + + if (!knod_dom_dominates(hdr, &bbs[u])) + continue; /* not a back-edge */ + n_be++; + + knod_loop_mark_body(bbs, n_bbs, &bbs[u], hdr, stack); + + for (k =3D 0; k < n_bbs; k++) { + if (!bbs[k].in_loop) + continue; + body++; + for (sp =3D 0; sp < bbs[k].n_succ; sp++) + if (!bbs[k].succ[sp]->in_loop) + exits++; + } + + pr_info("knod_loop: back-edge bpf#%d -> bpf#%d (latch->header) body=3D%= d exits=3D%d\n", + bbs[u].leader->bpf_insn_idx, + hdr->leader->bpf_insn_idx, body, exits); + } + } + + kfree(stack); + + if (n_be) + pr_info("knod_loop: %d back-edge(s) - %s\n", n_be, + n_be =3D=3D 1 ? "single loop (simple-shape candidate)" : + "nested/multiple loops (complex)"); + return 0; +} + +/* + * Block that lanes fall into in list order when the terminator is not tak= en: + * the not-taken successor of a conditional jump, or the sole successor of= a + * block that ended only because the next instruction was a leader. BPF_J= A and + * EXIT have no such successor (control leaves explicitly). + */ +static struct knod_bb *knod_bb_fall_succ(struct knod_bb *bb) +{ + if (knod_meta_is_exit(bb->last) || knod_meta_is_ja(bb->last)) + return NULL; + return bb->n_succ ? bb->succ[0] : NULL; +} + +/* + * Reorder the instruction list into reverse-postorder so every control-fl= ow + * edge points forward, and splice in a synthetic BPF_JA wherever a block's + * not-taken successor no longer follows it in list order. After this the + * emitter's forward-only machinery (FORWARD_SKIP / FORWARD_GOTO) handles = the + * whole program - including the backward-in-byte-order, non-loop jumps th= at + * the old jmp_off < 0 heuristic miscompiled. + * + * Loops (back-edges) are rejected with -EOPNOTSUPP until loop emission la= nds. + */ +static int knod_bpf_reorder_rpo(struct knod_prog *knod_prog, + struct knod_bb *bbs, int n_bbs, int n_back) +{ + struct knod_insn_meta *m, *nx, *sj; + struct knod_bb **order; + int n_order =3D 0, r, i, k, idx =3D 0; + LIST_HEAD(new_list); + + if (n_back) { + pr_warn("knod_cfg: %d loop back-edge(s) - block reorder cannot lower loo= ps yet (-EOPNOTSUPP)\n", + n_back); + return -EOPNOTSUPP; + } + + order =3D kcalloc(n_bbs, sizeof(*order), GFP_KERNEL); + if (!order) + return -ENOMEM; + + /* Reachable blocks in RPO, then any unreachable ones so no instruction + * is dropped from the list. + */ + for (r =3D 0; r < n_bbs; r++) + for (i =3D 0; i < n_bbs; i++) + if (bbs[i].rpo =3D=3D r) { + order[n_order++] =3D &bbs[i]; + break; + } + for (i =3D 0; i < n_bbs; i++) + if (bbs[i].rpo < 0) + order[n_order++] =3D &bbs[i]; + + for (k =3D 0; k < n_order; k++) { + struct knod_bb *bb =3D order[k]; + struct knod_bb *next =3D (k + 1 < n_order) ? order[k + 1] : NULL; + struct knod_bb *fall; + + m =3D bb->leader; + while (true) { + nx =3D (m =3D=3D bb->last) ? NULL : knod_meta_next(m); + list_move_tail(&m->l, &new_list); + if (m =3D=3D bb->last) + break; + m =3D nx; + } + + fall =3D knod_bb_fall_succ(bb); + if (!fall || (next && next->leader =3D=3D fall->leader)) + continue; + + /* Not-taken successor no longer adjacent: route it + * explicitly. + */ + sj =3D kzalloc_obj(*sj, GFP_KERNEL); + if (!sj) { + list_splice(&new_list, &knod_prog->insns); + kfree(order); + return -ENOMEM; + } + sj->insn.code =3D BPF_JMP | BPF_JA | BPF_K; + /* synthetic, never a jump target */ + sj->bpf_insn_idx =3D -1; + /* consumed by classify_linear */ + sj->jmp_dst =3D fall->leader; + INIT_LIST_HEAD(&sj->l); + list_add_tail(&sj->l, &new_list); + } + + list_splice(&new_list, &knod_prog->insns); + + list_for_each_entry(m, &knod_prog->insns, l) + m->linear_idx =3D idx++; + + kfree(order); + return 0; +} + +/* + * Classify branches by linear position after the RPO reorder. Every edge= is + * now forward, so a conditional jump is FORWARD_SKIP (or DIRECT_EXIT when= it + * targets the exit), and every BPF_JA - real or synthetic - is FORWARD_GO= TO + * (or DIRECT_EXIT). + */ +static int knod_bpf_classify_linear(struct knod_prog *knod_prog) +{ + struct knod_insn_meta *meta, *target; + short ti; + + list_for_each_entry(meta, &knod_prog->insns, l) { + if (is_mbpf_cond_jump(meta)) { + meta->jump_neg_op =3D (mbpf_op(meta) =3D=3D BPF_JNE); + ti =3D knod_meta_jump_target_idx(meta); + target =3D knod_bpf_lookup_meta(knod_prog, ti); + } else if (knod_meta_is_ja(meta)) { + /* synthetic JA carries its destination in jmp_dst; + * a real BPF_JA is resolved from its offset. + */ + if (meta->jmp_dst) { + target =3D meta->jmp_dst; + } else { + ti =3D knod_meta_jump_target_idx(meta); + target =3D knod_bpf_lookup_meta(knod_prog, ti); + } + } else { + continue; + } + + if (!target) { + pr_err("knod_cfg: bpf#%d unresolved branch target\n", + meta->bpf_insn_idx); + return -EINVAL; + } + + if (target->linear_idx <=3D meta->linear_idx) + pr_warn("knod_cfg: bpf#%d -> #%d still backward after reorder (linear %= d -> %d)\n", + meta->bpf_insn_idx, target->bpf_insn_idx, + meta->linear_idx, target->linear_idx); + + if (knod_bpf_is_direct_exit_target(knod_prog, target)) { + meta->branch_type =3D KNOD_BR_DIRECT_EXIT; + meta->merge_point =3D target; + continue; + } + + meta->branch_type =3D is_mbpf_cond_jump(meta) ? + KNOD_BR_FORWARD_SKIP : KNOD_BR_FORWARD_GOTO; + meta->merge_point =3D target; + target->is_merge_point =3D true; + } + + return 0; +} + +/* + * Build the basic-block CFG, compute RPO, reorder the instruction list in= to + * RPO and insert synthetic jumps. Returns 0, or a negative errno (a loop + * yields -EOPNOTSUPP). + */ +static int knod_bpf_build_cfg(struct knod_prog *knod_prog) +{ + struct knod_insn_meta *meta; + int n_insns =3D 0, n_bbs, n_back =3D 0, ret; + struct knod_bb *bbs; + + list_for_each_entry(meta, &knod_prog->insns, l) + n_insns++; + if (!n_insns) + return 0; + + bbs =3D kcalloc(n_insns, sizeof(*bbs), GFP_KERNEL); + if (!bbs) + return -ENOMEM; + + n_bbs =3D knod_bpf_build_bbs(knod_prog, bbs); + if (n_bbs < 0) { + ret =3D n_bbs; + goto out_free; + } + + ret =3D knod_bpf_build_edges(knod_prog, bbs, n_bbs); + if (ret) + goto out_free; + + ret =3D knod_bpf_compute_rpo(bbs, n_bbs, &bbs[0], &n_back); + if (ret) + goto out_free; + + ret =3D knod_bpf_compute_dom(bbs, n_bbs, &bbs[0]); + if (ret) + goto out_free; + + if (n_back) { + ret =3D knod_bpf_detect_loops(bbs, n_bbs); + if (ret) + goto out_free; + } + + /* Hand the block array to the prog for the /bpf/cfg view (freed at + * teardown); kept even if the reorder below rejects a loop, so the + * rejection can be inspected. + */ + kfree(knod_prog->bbs); + knod_prog->bbs =3D bbs; + knod_prog->n_bbs =3D n_bbs; + knod_prog->n_back =3D n_back; + + return knod_bpf_reorder_rpo(knod_prog, bbs, n_bbs, n_back); + +out_free: + kfree(bbs); + return ret; +} + +/* + * Assign exec_save SGPR pairs to the forward branches, recycling a pair o= nce + * its merge point has been passed. The peak concurrent live count is the + * actual SGPR requirement - usually far less than the total branch count. + */ +static int knod_bpf_alloc_exec_sregs(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog) +{ + struct { + u8 sreg; + struct knod_insn_meta *merge; + } live[72]; + int exec_save_max, max_pairs, n_live, peak, j; + struct knod_insn_meta *meta; + u8 free_stack[72]; + int free_top; + + exec_save_max =3D (priv->isa_version =3D=3D 10) ? + KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX10 : + KNOD_AMDGPU_EXEC_SAVE_SREG_MAX_GFX9; + max_pairs =3D (exec_save_max - knod_prog->exec_save_base + 1) / 2; + + for (free_top =3D 0; free_top < max_pairs; free_top++) + free_stack[free_top] =3D knod_prog->exec_save_base + + (max_pairs - 1 - free_top) * 2; + + n_live =3D 0; + peak =3D 0; + + list_for_each_entry(meta, &knod_prog->insns, l) { + /* Reclaim pairs from scopes that merge at this insn */ + for (j =3D n_live - 1; j >=3D 0; j--) { + if (live[j].merge =3D=3D meta) { + free_stack[free_top++] =3D live[j].sreg; + live[j] =3D live[--n_live]; + } + } + + if (meta->branch_type !=3D KNOD_BR_FORWARD_SKIP && + meta->branch_type !=3D KNOD_BR_FORWARD_GOTO) + continue; + + if (free_top =3D=3D 0) { + pr_err("knod_cfg: exec_save exhausted, peak %d concurrent scopes (max %= d)\n", + peak, max_pairs); + return -ENOSPC; + } + + meta->exec_save_sreg =3D free_stack[--free_top]; + live[n_live].sreg =3D meta->exec_save_sreg; + live[n_live].merge =3D meta->merge_point; + n_live++; + + if (n_live > peak) + peak =3D n_live; + } + + knod_prog->exec_save_pairs_used =3D peak; + pr_debug("knod_cfg: done, peak %d concurrent scopes (total fwd jumps: %d+= %d)\n", + peak, peak, n_live); + return 0; +} + +/* + * knod_bpf_analyze_cfg - Classify branches and allocate SGPRs for + * structurized CFG. + * + * Runs before instruction emission. For each conditional branch: + * - Backward jump or jump to EXIT -> DIRECT_EXIT (no SGPR needed) + * - Forward jump to non-EXIT -> FORWARD_SKIP, allocate SGPR pair + * + * The "save jumping lanes" pattern handles crossing scopes correctly: + * branch: s_and_b64 s[n], exec, vcc; s_andn2_b64 exec, exec, vcc + * merge: s_or_b64 exec, exec, s[n] + * + * For JNE (jump_neg_op): VCC=3D0 -> jump, so lanes are swapped. + */ +static int knod_bpf_analyze_cfg(struct knod_bpf_priv *priv, + struct knod_prog *knod_prog) +{ + int ret; + + /* Build the basic-block CFG, reorder the instruction list into RPO so + * every branch is forward (inserting synthetic jumps where a not-taken + * successor would no longer be adjacent), then classify each branch by + * linear position. A loop in the program is rejected (-EOPNOTSUPP). + */ + ret =3D knod_bpf_build_cfg(knod_prog); + if (ret) + return ret; + ret =3D knod_bpf_classify_linear(knod_prog); + if (ret) + return ret; + + return knod_bpf_alloc_exec_sregs(priv, knod_prog); +} + +/* + * Shader stores packet address and length into pass_meta_buf slot header. + * Host-side SDMA engine does the actual copy to the delivery page. + * + * At entry: + * TMP_VREG10_LO (v42) =3D old_val * 2 (from pass_indices addressing) + * DATA_VREG (v64:v65) =3D packet source VRAM address + * DATA_END_VREG (v66:v67) =3D packet end address + * PARAM_SREG (s28:s29) =3D param GTT address + * + * Stores at slot header: + * +0: u32 len (DATA_END_LO - DATA_LO) + * +8: u64 src_addr (DATA_VREG) + */ +static void knod_emit_pass_addr_store(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta) +{ + struct amdgcn_param32 p[3]; + + /* s_lshl_b32 s18, s15, 3 - queue_idx * 8 for pass_meta_buf_gaddr + * stride + */ + knod_sset32(&p[0], KNOD_AMDGPU_TMP_SREG1_LO); + knod_sset32(&p[1], KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_iset32(&p[2], 3); + knod_emit(priv, meta, s_lshl_b32, p[0], p[1], p[2]); + + /* s_load_dwordx2 s[16:17], s[28:29], offsetof(pass_meta_buf_gaddr) + * soffset=3Ds18 + */ + knod_sset32(&p[0], KNOD_AMDGPU_TMP_SREG0_LO); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, s_load_dwordx2_soff, p[0], p[1], + offsetof(struct knod_bpf_param, pass_meta_buf_gaddr), + KNOD_AMDGPU_TMP_SREG1_LO); + + /* s_waitcnt lgkmcnt(0) */ + knod_emit(priv, meta, s_waitcnt_lgkmcnt); + + /* Compute slot offset: old_val << 12 =3D (old_val*2) << 11 + * v_lshlrev_b32 v44, 11, v42 + */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG11_LO); + knod_iset32(&p[1], 11); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG10_LO); + knod_emit(priv, meta, v_lshlrev_b32, p[0], p[1], p[2]); + + /* v_mov_b32 v45, 0 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG11_HI); + knod_iset32(&p[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* v_add_co_u32 v44, s16, v44 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG11_LO); + knod_sset32(&p[1], KNOD_AMDGPU_TMP_SREG0_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG11_LO); + knod_emit(priv, meta, v_add_co_u32, p[0], p[1], p[2]); + + /* + * slot_hi =3D base_hi (NOT base_hi + carry). The slot offset is at + * most (pass_pkts_per_queue-1)*KNOD_PASS_SLOT_SIZE and the whole + * pass_meta_buf is a single contiguous allocation that never straddles + * a 4GiB boundary, so base_lo + offset never wraps and the carry is + * always 0. Avoid the v_add_co/v_addc carry chain entirely: on GFX9 + * the v_addc here was picking up a stale VCC (from the preceding + * XDP_PASS v_cmp) instead of the v_add_co carry-out, setting slot_hi=3D1 + * and faulting at 0x1_xxxx. + * v_mov_b32 v45, s17 + */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG11_HI); + knod_sset32(&p[1], KNOD_AMDGPU_TMP_SREG0_HI); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* v44:v45 =3D slot_addr in pass_meta_buf */ + + /* Store len: v_sub_u32 v0, DATA_END_LO, DATA_LO */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_vset32(&p[1], KNOD_AMDGPU_DATA_END_VREG_LO); + knod_vset32(&p[2], KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_sub_u32, p[0], p[1], p[2]); + + /* global_store_dword [slot+0], len */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG11_LO); + knod_emit(priv, meta, global_store_dword, p[0], p[1], + offsetof(struct knod_pass_slot_hdr, len)); + + /* global_store_dwordx2 [slot+8], DATA_VREG (src_addr) */ + knod_vset32(&p[0], KNOD_AMDGPU_DATA_VREG_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG11_LO); + knod_emit(priv, meta, global_store_dwordx2, p[0], p[1], + offsetof(struct knod_pass_slot_hdr, src_addr)); +} + +static int knod_bpf_jit(struct knod_dev *knodev, + struct knod_prog *knod_prog) +{ + struct knod_bpf_priv *priv =3D + (struct knod_bpf_priv *)knodev->accel->xdp.priv; + short off, packet_off, stack_off; + struct knod_insn_meta *meta, *meta2; + struct amdgcn_param64 param64[2]; + u32 insn_idx =3D 0, i; + struct amdgcn_param32 param[3]; + struct amdgcn_param32 p32[2]; + struct amdgcn_param32 p[3]; + int s, d, imm, imm2; + int pass_branch_idx; + bool is_dw, fetch; + bool skip =3D false; + int pass_dwords; + int atomic_op; + int j; + int map_id; + u64 imm64; + int ret; + + /* Analyze CFG before instruction emission */ + ret =3D knod_bpf_analyze_cfg(priv, knod_prog); + + if (ret) + return ret; + + knod_bpf_layout_sregs(priv, knod_prog); + ret =3D knod_prog_prepare_insns(priv, knod_prog); + if (ret) + return ret; + + knod_prog->max_stack_off =3D -knod_prog->max_stack_off; + knod_prog->max_stack_off =3D ALIGN(knod_prog->max_stack_off, 4); + knod_prog->max_packet_off =3D ALIGN(knod_prog->max_packet_off, 4); + /* NOTE: + * packet is accessed with packet_off + size + * largest size of it is unsigned long + */ + knod_prog->max_packet_off +=3D sizeof(unsigned long); + if (knod_prog->max_packet_off > MAX_PACKET_CACHE) { + WARN_ON_ONCE(1); + knod_bpf_pkt_cache =3D 0; + } + + /* Initialize all exec_save SGPRs to 0. + * Without this, merge points that restore from exec_save SGPRs + * of branches that were skipped (by an outer s_cbranch_execz) + * would OR garbage into EXEC, enabling invalid lanes. + * In the old code, BPF_EXIT used s_endpgm so execution never + * reached those merge points; now it does. + */ + if (knod_prog->exec_save_pairs_used > 0) { + u8 sreg; + + meta =3D knod_prog_pre_last_meta(knod_prog); + + for (sreg =3D knod_prog->exec_save_base; + sreg < knod_prog->exec_save_base + + knod_prog->exec_save_pairs_used * 2; + sreg +=3D 2) + knod_emit(priv, meta, s_mov_b64, sreg, + AMDGCN_SREG_INTEGER_0); + } + + if (knod_bpf_pkt_cache) { + meta =3D knod_prog_pre_last_meta(knod_prog); + + /* ctx->data is in DATA_VREG -> copy to r32[0] via v_mov */ + knod_vset32(¶m[0], r32[0].v); + knod_vset32(¶m[1], KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + knod_vset32(¶m[0], r32[0].v + 1); + knod_vset32(¶m[1], KNOD_AMDGPU_DATA_VREG_HI); + knod_emit(priv, meta, v_mov_b32_e32, param[0], param[1]); + + knod_global_load_size_cache(priv, meta, + &pkt_cache[0], + r32[0], + 0, /* dst index */ + 0, /* start offset */ + knod_prog->max_packet_off); + } + + insn_idx =3D 0; + list_for_each_entry(meta, &knod_prog->pre_insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) + insn_idx +=3D (meta->amdgpu_insn[i].size / 4); + } + + list_for_each_entry(meta, &knod_prog->insns, l) { + if (skip) { + skip =3D false; + meta->amdgpu_insn_idx =3D AMDGPU_INSN_SKIP; + continue; + } + s =3D meta->insn.src_reg; + d =3D meta->insn.dst_reg; + imm =3D meta->insn.imm; + off =3D meta->insn.off; + + meta->amdgpu_insn_idx =3D insn_idx; + meta->amdgpu_insns =3D 0; + + /* Structurized CFG: restore EXEC at merge points */ + if (meta->is_merge_point) { + struct knod_insn_meta *br; + + list_for_each_entry(br, &knod_prog->insns, l) { + if ((br->branch_type =3D=3D KNOD_BR_FORWARD_SKIP || + br->branch_type =3D=3D KNOD_BR_FORWARD_GOTO) && + br->merge_point =3D=3D meta) { + knod_emit(priv, meta, s_or_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, + br->exec_save_sreg); + } + } + /* Remove done lanes from restored EXEC */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, + knod_prog->done_mask_sreg); + } + + switch (meta->insn.code) { + /* ALU + * If a destination register contains a pointer of STACK, + * offset should not be minus. + */ + case BPF_ALU | BPF_MOV | BPF_X: + case BPF_ALU64 | BPF_MOV | BPF_X: + //r[d] =3D r[s]; + knod_mov64(priv, meta, bpf_reg64[d], bpf_reg64[s]); + break; + case BPF_ALU | BPF_MOV | BPF_K: + case BPF_ALU64 | BPF_MOV | BPF_K: + //r[d] =3D imm; + knod_iset64(&p64[0], imm); + knod_mov64(priv, meta, bpf_reg64[d], p64[0]); + break; + case BPF_ALU | BPF_XOR | BPF_X: + knod_xor32(priv, meta, + bpf_reg64[d].lo, bpf_reg64[d].lo, + bpf_reg64[s].lo); + knod_iset64(&p64[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p64[0].lo); + break; + case BPF_ALU64 | BPF_XOR | BPF_X: + //r[d] ^=3D r[s]; + knod_xor32(priv, meta, + bpf_reg64[d].lo, bpf_reg64[d].lo, + bpf_reg64[s].lo); + knod_xor32(priv, meta, + bpf_reg64[d].hi, bpf_reg64[d].hi, + bpf_reg64[s].hi); + break; + case BPF_ALU | BPF_XOR | BPF_K: + case BPF_ALU64 | BPF_XOR | BPF_K: + knod_iset64(&p64[0], imm); + knod_mov64(priv, meta, bpf_reg64[d], p64[0]); + knod_xor32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, r64[0].lo); + break; + //r[d] ^=3D imm; + break; + case BPF_ALU | BPF_MOD | BPF_X: + case BPF_ALU64 | BPF_MOD | BPF_X: + //r[d] %=3D r[s]; + knod_iset64(&p64[0], meta->umin_src); + knod_mod(priv, meta, bpf_reg64[d], p64[0], + r64[0], r64[1], r64[2], r64[3], r64[4]); + break; + case BPF_ALU | BPF_MOD | BPF_K: + case BPF_ALU64 | BPF_MOD | BPF_K: + //r[d] %=3D imm; + /* The dividend fits 32 bits (verifier rejects wider + * div/mod), so the 32-bit fold is valid even when + * clang emitted this as a 64-bit ALU op (e.g. u32 + * hash % 65537 -> `r2 %=3D 65537`). + */ + if (meta->umax_dst <=3D U32_MAX && imm && + knod_mod_k32(priv, meta, bpf_reg64[d], imm)) + break; + knod_iset64(&p64[0], imm); + knod_mod(priv, meta, bpf_reg64[d], p64[0], + r64[0], r64[1], r64[2], r64[3], r64[4]); + break; + case BPF_ALU | BPF_AND | BPF_X: + knod_and32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_AND | BPF_X: + //r[d] &=3D r[s]; + knod_and64(priv, meta, bpf_reg64[d], + bpf_reg64[d], bpf_reg64[s]); + break; + case BPF_ALU | BPF_AND | BPF_K: + case BPF_ALU64 | BPF_AND | BPF_K: + //r[d] &=3D imm; + knod_iset32(&p32[0], imm); + knod_and32(priv, meta, bpf_reg64[d].lo, p32[0], + bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU | BPF_OR | BPF_X: + knod_or32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_OR | BPF_X: + //r[d] |=3D r[s]; + knod_or32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + knod_or32(priv, meta, bpf_reg64[d].hi, + bpf_reg64[d].hi, bpf_reg64[s].hi); + break; + case BPF_ALU | BPF_OR | BPF_K: + case BPF_ALU64 | BPF_OR | BPF_K: + //r[d] |=3D imm; + knod_iset32(&p32[0], imm); + knod_or32(priv, meta, + bpf_reg64[d].lo, p32[0], bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU | BPF_ADD | BPF_X: + knod_add32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_ADD | BPF_X: + knod_add64(priv, meta, bpf_reg64[d], + bpf_reg64[d], + bpf_reg64[s]); + + //r[d] +=3D r[s]; + break; + case BPF_ALU | BPF_ADD | BPF_K: + case BPF_ALU64 | BPF_ADD | BPF_K: + //r[d] +=3D imm; + knod_iset32(&p32[0], imm); + knod_add32(priv, meta, bpf_reg64[d].lo, + p32[0], bpf_reg64[d].lo); + /* NOTE: + * imm is 24bit. + * But should we set hi to 0? + */ + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU | BPF_SUB | BPF_X: + //r[d] -=3D r[s]; + knod_sub32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_SUB | BPF_X: + //r[d] -=3D r[s]; + + knod_sub64(priv, meta, bpf_reg64[d], bpf_reg64[d], + bpf_reg64[s]); + break; + case BPF_ALU | BPF_SUB | BPF_K: + case BPF_ALU64 | BPF_SUB | BPF_K: + //r[d] -=3D imm; + knod_iset64(&p64[0], imm); + knod_subrev64(priv, meta, bpf_reg64[d], p64[0], + bpf_reg64[s]); + break; + case BPF_ALU | BPF_MUL | BPF_X: + knod_mul_lo32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[d].lo, bpf_reg64[s].lo); + break; + case BPF_ALU64 | BPF_MUL | BPF_X: + //r[d] *=3D r[s]; + knod_mov64(priv, meta, r64[0], bpf_reg64[d]); + knod_mov64(priv, meta, r64[1], bpf_reg64[s]); + knod_mul64(priv, meta, + bpf_reg64[d], + r64[0], + r64[1], + r64[2]); + break; + case BPF_ALU | BPF_MUL | BPF_K: + knod_iset32(&p32[0], imm); + knod_mul_lo32(priv, meta, bpf_reg64[d].lo, + p32[0], bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_MUL | BPF_K: + //r[d] *=3D imm; + knod_iset64(&p64[0], imm); + knod_mov64(priv, meta, r64[0], bpf_reg64[d]); + knod_mov64(priv, meta, r64[1], p64[0]); + knod_mul64(priv, meta, + bpf_reg64[d], + r64[0], + r64[1], + r64[2]); + break; + case BPF_ALU | BPF_DIV | BPF_X: + case BPF_ALU64 | BPF_DIV | BPF_X: + //r[d] /=3D r[s]; + knod_iset64(&p64[0], meta->umin_src); + knod_div(priv, meta, bpf_reg64[d], p64[0], + r64[0], r64[1], r64[2], r64[3]); + break; + case BPF_ALU | BPF_DIV | BPF_K: + case BPF_ALU64 | BPF_DIV | BPF_K: + //r[d] /=3D imm; + knod_iset64(&p64[0], imm); + knod_div(priv, meta, bpf_reg64[d], p64[0], + r64[0], r64[1], r64[2], r64[3]); + break; + case BPF_ALU | BPF_NEG: + knod_iset32(&p32[0], 0); + knod_sub32(priv, meta, bpf_reg64[d].lo, p32[0], + bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_NEG: + //r[d] =3D -r[d]; + WARN_ON_ONCE(1); + break; + case BPF_ALU | BPF_LSH | BPF_X: + knod_lshlrev32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[s].lo, bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_LSH | BPF_X: + //r[d] <<=3D r[s]; + knod_lshlrev64(priv, meta, bpf_reg64[d], + bpf_reg64[s], bpf_reg64[d]); + break; + case BPF_ALU | BPF_LSH | BPF_K: + knod_iset32(&p32[0], imm); + knod_lshlrev32(priv, meta, bpf_reg64[d].lo, p32[0], + bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_LSH | BPF_K: + //r[d] <<=3D imm; + knod_iset64(&p64[0], imm); + knod_lshlrev64(priv, meta, bpf_reg64[d], p64[0], + bpf_reg64[d]); + break; + case BPF_ALU | BPF_RSH | BPF_X: + knod_lshrrev32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[s].lo, + bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_RSH | BPF_X: + //r[d] >>=3D r[s]; + knod_lshrrev64(priv, meta, bpf_reg64[d], + bpf_reg64[s], bpf_reg64[d]); + break; + case BPF_ALU | BPF_RSH | BPF_K: + knod_iset32(&p32[0], imm); + knod_lshrrev32(priv, meta, bpf_reg64[d].lo, p32[0], + bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_RSH | BPF_K: + //r[d] >>=3D imm; + knod_iset64(&p64[0], imm); + knod_lshrrev64(priv, meta, bpf_reg64[d], + p64[0], bpf_reg64[d]); + break; + case BPF_ALU | BPF_ARSH | BPF_X: + knod_ashrrev32(priv, meta, bpf_reg64[d].lo, + bpf_reg64[s].lo, bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_ARSH | BPF_X: + //r[d] >>=3D r[s]; + knod_ashrrev64(priv, meta, bpf_reg64[d], + bpf_reg64[s], bpf_reg64[d]); + break; + case BPF_ALU | BPF_ARSH | BPF_K: + knod_iset32(&p32[0], imm); + knod_ashrrev32(priv, meta, bpf_reg64[d].lo, + p32[0], bpf_reg64[d].lo); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + break; + case BPF_ALU64 | BPF_ARSH | BPF_K: + //r[d] >>=3D imm; + knod_iset64(&p64[0], imm); + knod_ashrrev64(priv, meta, bpf_reg64[d], + p64[0], bpf_reg64[d]); + break; + case BPF_LD | BPF_IMM | BPF_DW: + meta2 =3D list_next_entry(meta, l); + if (WARN_ON_ONCE(!meta2)) + return -EINVAL; + imm2 =3D meta2->insn.imm; + skip =3D true; + imm64 =3D (u64)imm2 << 32 | (u32)imm; + switch (s) { + case 0x00: + //r[d] =3D imm64; + knod_mov64_imm(priv, meta, d * 2, + imm64); + + break; + case 0x01: + /* r[d] =3D param->maps[imm]; */ + imm64 =3D knod_bpf_get_map_gaddr(priv, + meta, + meta2); + map_id =3D knod_bpf_get_map_id(priv, + meta, + meta2); + knod_mov64_imm(priv, meta, d * 2, + imm64); + break; + default: + WARN_ON_ONCE(1); + break; + } + break; + /* Legacy BPF packet access, not needed */ + case BPF_LD | BPF_ABS | BPF_B: + case BPF_LD | BPF_ABS | BPF_H: + case BPF_LD | BPF_ABS | BPF_W: + case BPF_LD | BPF_IND | BPF_B: + case BPF_LD | BPF_IND | BPF_H: + case BPF_LD | BPF_IND | BPF_W: + //err =3D pc | 0x0700; + //exit =3D true; + WARN_ON_ONCE(1); + break; + case BPF_LDX | BPF_MEM | BPF_B: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->sreg.stack_off + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &stack[0], + sizeof(unsigned char), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_load_ubyte, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_load_ubyte, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_load_ubyte, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->sreg.packet_off + + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &pkt_cache[0], + sizeof(unsigned char), + packet_off); + } else { + knod_emit(priv, meta, global_load_ubyte, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + knod_wait_vmcnt(priv, meta); + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + //ptr =3D (__global void *)r[s] + off; + //r[d] =3D *(__global unsigned char *)ptr; + break; + case BPF_LDX | BPF_MEM | BPF_H: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->sreg.stack_off + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &stack[0], + sizeof(unsigned short), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_load_ushort, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_load_ushort, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_load_ushort, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D SCALAR_VALUE) { + knod_emit(priv, meta, global_load_ushort, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->sreg.packet_off + + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &pkt_cache[0], + sizeof(unsigned short), + packet_off); + } else if (priv->isa_version =3D=3D 10 && + (off & 1)) { + knod_bpf_emit_gfx10_unaligned_load( + priv, meta, + sizeof(unsigned short), + bpf_reg64[d], + bpf_reg64[s].lo, off); + } else { + knod_emit(priv, meta, + global_load_ushort, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } + } else { + knod_jit_err(" type =3D %d\n", meta->ptr.type); + WARN_ON_ONCE(1); + } + //ptr =3D (__global void *)r[s] + off; + //r[d] =3D *(__global unsigned short *)ptr; + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, p32[0]); + knod_wait_vmcnt(priv, meta); + break; + case BPF_LDX | BPF_MEM | BPF_W: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->sreg.stack_off + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &stack[0], + sizeof(unsigned int), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + if (off =3D=3D offsetof(struct xdp_md, data)) { + knod_mov32(priv, meta, + bpf_reg64[d].lo, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_VREG_LO, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + knod_mov32(priv, meta, + bpf_reg64[d].hi, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_VREG_HI, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + } else if (off =3D=3D offsetof(struct xdp_md, + data_end)) { + knod_mov32(priv, meta, + bpf_reg64[d].lo, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_END_VREG_LO, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + knod_mov32(priv, meta, + bpf_reg64[d].hi, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_END_VREG_HI, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + } else { + emit_global_load_dwordx2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + bpf_reg64[d].lo, + bpf_reg64[s].lo, + off * 2); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + } + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_load_dword, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_load_dword, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->sreg.packet_off + + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &pkt_cache[0], + sizeof(unsigned int), + packet_off); + } else if (priv->isa_version =3D=3D 10 && + (off & 3)) { + knod_bpf_emit_gfx10_unaligned_load( + priv, meta, + sizeof(unsigned int), + bpf_reg64[d], + bpf_reg64[s].lo, off); + } else { + knod_emit(priv, meta, global_load_dword, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + //ptr =3D (__global void *)r[s] + off; + //r[d] =3D *(__global unsigned int *)ptr; + if (meta->ptr.type !=3D PTR_TO_CTX) { + knod_iset32(&p32[0], 0); + knod_mov32(priv, meta, bpf_reg64[d].hi, + p32[0]); + } + knod_wait_vmcnt(priv, meta); + break; + case BPF_LDX | BPF_MEM | BPF_DW: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->sreg.stack_off + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &stack[0], + sizeof(unsigned long), + 512+stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + if (off =3D=3D offsetof(struct xdp_md, data)) { + knod_mov32(priv, meta, + bpf_reg64[d].lo, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_VREG_LO, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + knod_mov32(priv, meta, + bpf_reg64[d].hi, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_VREG_HI, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + } else if (off =3D=3D offsetof(struct xdp_md, + data_end)) { + knod_mov32(priv, meta, + bpf_reg64[d].lo, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_END_VREG_LO, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + knod_mov32(priv, meta, + bpf_reg64[d].hi, + (struct amdgcn_param32){ + .v =3D KNOD_AMDGPU_DATA_END_VREG_HI, + .type =3D AMDGCN_PARAM_TYPE_VGPR}); + } else { + knod_emit(priv, meta, + global_load_dwordx2, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off * 2); + } + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_load_dwordx2, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_load_dwordx2, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->sreg.packet_off + + off; + knod_bpf_load_size(priv, meta, + &bpf_reg64[d], + &pkt_cache[0], + sizeof(unsigned long), + packet_off); + } else if (priv->isa_version =3D=3D 10 && + (off & 3)) { + knod_bpf_emit_gfx10_unaligned_load( + priv, meta, + sizeof(unsigned long), + bpf_reg64[d], + bpf_reg64[s].lo, off); + } else { + knod_emit(priv, meta, + global_load_dwordx2, + bpf_reg64[d].lo, + bpf_reg64[s].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + //ptr =3D (__global void *)r[s] + off; + //r[d] =3D *(__global unsigned long *)ptr; + knod_wait_vmcnt(priv, meta); + break; + case BPF_STX | BPF_MEM | BPF_B: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_bpf_store_cache_size(priv, meta, + &bpf_reg64[s], + &stack[0], + sizeof(u8), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_byte, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_byte, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_byte, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &bpf_reg64[s], + &pkt_cache[0], + sizeof(u8), + packet_off); + } else { + knod_emit(priv, meta, global_store_byte, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_STX | BPF_MEM | BPF_H: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_bpf_store_cache_size(priv, meta, + &bpf_reg64[s], + &stack[0], + sizeof(u16), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_short, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_short, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_short, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &bpf_reg64[s], + &pkt_cache[0], + sizeof(u16), + packet_off); + } else { + knod_emit(priv, meta, + global_store_short, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_STX | BPF_MEM | BPF_W: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_bpf_store_cache_size(priv, meta, + &bpf_reg64[s], + &stack[0], + sizeof(u32), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_dword, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_dword, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_dword, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &bpf_reg64[s], + &pkt_cache[0], + sizeof(u32), + packet_off); + } else { + knod_emit(priv, meta, + global_store_dword, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_STX | BPF_MEM | BPF_DW: + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_bpf_store_cache_size(priv, meta, + &bpf_reg64[s], + &stack[0], + sizeof(u64), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_dwordx2, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_dwordx2, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_dwordx2, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, meta, + &bpf_reg64[s], + &pkt_cache[0], + sizeof(u64), + packet_off); + } else { + knod_emit(priv, meta, + global_store_dwordx2, + bpf_reg64[s].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_STX | BPF_ATOMIC | BPF_W: + case BPF_STX | BPF_ATOMIC | BPF_DW: + is_dw =3D BPF_SIZE(meta->insn.code) =3D=3D BPF_DW; + atomic_op =3D imm & ~BPF_FETCH; + fetch =3D imm & BPF_FETCH; + + /* + * BPF atomic: *(dst_reg + off) op=3D src_reg + * If BPF_FETCH: src_reg =3D old value + * BPF_CMPXCHG: expect in r0, new in src_reg, + * old value returned in r0. + * + * global_atomic_* with glc=3D1 returns old value in vdst. + * For non-FETCH ops use glc=3D0 (fire-and-forget). + * + * 64-bit atomics (global_atomic_*_x2) hang on GFX9 + * VRAM. GFX10+ supports them. + */ + if (is_dw && priv->isa_version =3D=3D 9) { + pr_err("knod: 64-bit atomic not supported on GFX9\n"); + return -EOPNOTSUPP; + } + + /* + * For CMPXCHG/FETCH: drain pending loads so addr/data + * VGPRs are ready. For non-fetch ADD wave reduction, + * addr was already waited for at map_lookup, and data + * is from ALU - no waitcnt needed. + */ + if (imm =3D=3D BPF_CMPXCHG || fetch) + knod_wait_vmcnt(priv, meta); + + if (imm =3D=3D BPF_CMPXCHG) { + /* cmpswap: data =3D {expect(r0), new(src)}. + * AMD cmpswap data reg pair must be + * consecutive: + * 32-bit: {cmp, new} =3D 2 consecutive VGPRs + * 64-bit: {cmp_lo, cmp_hi, new_lo, new_hi} + * Copy r0 and src into TMP consecutive pair. + */ + struct amdgcn_param32 tmp0_lo, tmp0_hi, + tmp1_lo, tmp1_hi; + + knod_vset32(&tmp0_lo, + KNOD_AMDGPU_TMP_VREG0_LO); + knod_vset32(&tmp0_hi, + KNOD_AMDGPU_TMP_VREG0_HI); + knod_vset32(&tmp1_lo, + KNOD_AMDGPU_TMP_VREG1_LO); + knod_vset32(&tmp1_hi, + KNOD_AMDGPU_TMP_VREG1_HI); + + if (!is_dw) { + /* TMP0_LO =3D r0 (expect), + * TMP0_HI =3D src (new) + */ + knod_mov32(priv, meta, + tmp0_lo, + bpf_reg64[0].lo); + knod_mov32(priv, meta, + tmp0_hi, + bpf_reg64[s].lo); + + emit_global_atomic_cmpswap( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + tmp0_lo, bpf_reg64[d].lo, + tmp0_lo, off, 1); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + knod_wait_vmcnt(priv, meta); + /* Return old value in r0 */ + knod_mov32(priv, meta, + bpf_reg64[0].lo, + tmp0_lo); + } else { + /* 64-bit: + * {r0_lo, r0_hi, src_lo, src_hi} + */ + knod_mov32(priv, meta, + tmp0_lo, + bpf_reg64[0].lo); + knod_mov32(priv, meta, + tmp0_hi, + bpf_reg64[0].hi); + knod_mov32(priv, meta, + tmp1_lo, + bpf_reg64[s].lo); + knod_mov32(priv, meta, + tmp1_hi, + bpf_reg64[s].hi); + + emit_global_atomic_cmpswap_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + tmp0_lo, bpf_reg64[d].lo, + tmp0_lo, off, 1); + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + knod_wait_vmcnt(priv, meta); + knod_mov32(priv, meta, + bpf_reg64[0].lo, + tmp0_lo); + knod_mov32(priv, meta, + bpf_reg64[0].hi, + tmp0_hi); + } + } else if (!fetch && atomic_op =3D=3D BPF_ADD) { + /* + * Wave reduction for BPF_ADD (non-fetch): + * Instead of all lanes doing atomic_add(val), + * count active lanes, multiply by val, and + * have a single lane do atomic_add(count*val). + * + * Assumes src_reg is uniform across all active + * lanes (true for constant increments like + * +=3D1). + * + * s_bcnt1_i32_b64 s_tmp, exec + * v_mul_lo_u32 v_tmp, s_tmp, v_src + * v_mbcnt_lo v_tmp2, exec_lo, 0 + * v_mbcnt_hi v_tmp2, exec_hi, v_tmp2 + * v_cmp_eq_u32 vcc, v_tmp2, 0 + * s_and_saveexec s_save, vcc + * global_atomic_add addr, v_tmp, off + * s_waitcnt vmcnt(0) + * s_mov_b64 exec, s_save + */ + struct amdgcn_param32 v_tmp, v_tmp2, + s_count, s_exec_lo, + s_exec_hi, v_zero; + + knod_vset32(&v_tmp, + KNOD_AMDGPU_TMP_VREG0_LO); + knod_vset32(&v_tmp2, + KNOD_AMDGPU_TMP_VREG0_HI); + knod_sset32(&s_count, + KNOD_AMDGPU_TMP_SREG0_LO); + knod_sset32(&s_exec_lo, + AMDGCN_SREG_EXEC_LO); + knod_sset32(&s_exec_hi, + AMDGCN_SREG_EXEC_LO + 1); + knod_iset32(&v_zero, 0); + + /* s_bcnt1_i32_b64 s_count, exec */ + knod_emit(priv, meta, s_bcnt1_i32_b64, + KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_EXEC_LO); + + /* v_mul_lo_u32 v_tmp, s_count, v_src */ + knod_emit(priv, meta, v_mul_lo_u32, v_tmp, + s_count, bpf_reg64[s].lo); + + /* v_mbcnt_lo_u32_b32 v_tmp2, exec_lo, 0 */ + knod_emit(priv, meta, v_mbcnt_lo_u32_b32, + v_tmp2, s_exec_lo, v_zero); + + /* v_mbcnt_hi_u32_b32 v_tmp2, exec_hi, v_tmp2 */ + knod_emit(priv, meta, v_mbcnt_hi_u32_b32, + v_tmp2, s_exec_hi, v_tmp2); + + /* v_cmp_eq_u32 vcc, 0, v_tmp2 -> + * first active lane + */ + knod_emit(priv, meta, v_cmp_eq_u32, v_zero, + v_tmp2); + + /* s_and_saveexec_b64 s_save, vcc */ + knod_emit(priv, meta, s_and_saveexec_b64, + KNOD_AMDGPU_TMP_SREG0_LO, + AMDGCN_SREG_VCC_LO); + + if (is_dw) { + struct amdgcn_param32 v_tmp_hi; + + knod_vset32(&v_tmp_hi, + KNOD_AMDGPU_TMP_VREG0_HI); + /* + * x2 atomics consume a consecutive + * VGPR pair, and the 32-bit addend + * lands in the host-visible low dword + * when it is placed in the second + * register. + */ + knod_emit(priv, meta, v_mov_b32_e32, + v_tmp_hi, v_tmp); + knod_emit(priv, meta, v_mov_b32_e32, + v_tmp, v_zero); + /* global_atomic_add_x2 addr, + * {0, v_tmp_hi}, off + */ + knod_emit(priv, meta, + global_atomic_add_x2, v_tmp, + bpf_reg64[d].lo, v_tmp, off, + 0); + } else { + /* global_atomic_add addr, v_tmp, off + * (single lane) + */ + knod_emit(priv, meta, global_atomic_add, + v_tmp, + bpf_reg64[d].lo, v_tmp, off, + 0); + } + + /* + * No s_waitcnt needed: glc=3D0 atomic doesn't + * increment vmcnt. The GPU guarantees all + * pending ops complete before wave exit. + */ + + /* s_mov_b64 exec, s_save */ + knod_emit(priv, meta, s_mov_b64, + AMDGCN_SREG_EXEC_LO, + KNOD_AMDGPU_TMP_SREG0_LO); + } else if (!is_dw) { + /* 32-bit: AND, OR, XOR, XCHG, or fetch ops */ + struct amdgcn_param32 vdst, data_p; + + if (fetch) { + vdst =3D bpf_reg64[s].lo; + } else { + knod_vset32(&vdst, + KNOD_AMDGPU_TMP_VREG0_LO); + } + data_p =3D bpf_reg64[s].lo; + + switch (atomic_op) { + case BPF_ADD: + emit_global_atomic_add( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_AND: + emit_global_atomic_and( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_OR: + emit_global_atomic_or( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_XOR: + emit_global_atomic_xor( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + default: /* BPF_XCHG */ + emit_global_atomic_swap( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + } + } else { + /* 64-bit: AND, OR, XOR, XCHG, or fetch ops */ + struct amdgcn_param32 vdst, data_p; + + if (fetch) { + vdst =3D bpf_reg64[s].lo; + } else { + knod_vset32(&vdst, + KNOD_AMDGPU_TMP_VREG0_LO); + } + data_p =3D bpf_reg64[s].lo; + + switch (atomic_op) { + case BPF_ADD: + emit_global_atomic_add_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_AND: + emit_global_atomic_and_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_OR: + emit_global_atomic_or_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + case BPF_XOR: + emit_global_atomic_xor_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + default: /* BPF_XCHG */ + emit_global_atomic_swap_x2( + priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns], + vdst, bpf_reg64[d].lo, + data_p, off, fetch); + break; + } + debug_insn(priv->isa_version, + &meta->amdgpu_insn[meta->amdgpu_insns]); + meta->amdgpu_insns++; + /* Always wait for atomic completion */ + knod_wait_vmcnt(priv, meta); + } + break; + case BPF_ST | BPF_MEM | BPF_B: + knod_iset32(&p32[0], imm); + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_iset64(&p64[0], imm); + knod_bpf_store_cache_size(priv, meta, + &p64[0], + &stack[0], + sizeof(u8), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_byte, p32[0], + bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_byte, p32[0], + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_byte, p32[0], + bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + knod_iset64(&p64[0], imm); + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &p64[0], + &pkt_cache[0], + sizeof(u8), + packet_off); + } else { + knod_emit(priv, meta, global_store_byte, + p64[0].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_ST | BPF_MEM | BPF_H: + knod_iset32(&p32[0], imm); + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_iset64(&p64[0], imm); + knod_bpf_store_cache_size(priv, meta, + &p64[0], + &stack[0], + sizeof(u16), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_short, + p32[0], bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_short, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_short, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + knod_iset64(&p64[0], imm); + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &p64[0], + &pkt_cache[0], + sizeof(u16), + packet_off); + } else { + knod_emit(priv, meta, + global_store_short, p64[0].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_ST | BPF_MEM | BPF_W: + knod_iset32(&p32[0], imm); + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_iset64(&p64[0], imm); + knod_bpf_store_cache_size(priv, meta, + &p64[0], + &stack[0], + sizeof(u32), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_dword, + p32[0], bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_dword, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_dword, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + knod_iset64(&p64[0], imm); + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, + meta, + &p64[0], + &pkt_cache[0], + sizeof(u32), + packet_off); + } else { + knod_emit(priv, meta, + global_store_dword, p64[0].lo, + bpf_reg64[d].lo, off); + } + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_ST | BPF_MEM | BPF_DW: + knod_iset32(&p32[0], imm); + if (meta->ptr.type =3D=3D PTR_TO_STACK) { + stack_off =3D meta->dreg.stack_off + off; + knod_iset64(&p64[0], imm); + knod_bpf_store_cache_size(priv, meta, + &p64[0], + &stack[0], + sizeof(u64), + 512 + stack_off); + } else if (meta->ptr.type =3D=3D PTR_TO_CTX) { + knod_emit(priv, meta, global_store_dwordx2, + p32[0], bpf_reg64[d].lo, off * 2); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_VALUE) { + knod_emit(priv, meta, global_store_dwordx2, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_MAP_KEY) { + knod_emit(priv, meta, global_store_dwordx2, + p32[0], bpf_reg64[d].lo, off); + } else if (meta->ptr.type =3D=3D PTR_TO_PACKET) { + knod_iset64(&p64[0], imm); + if (knod_bpf_pkt_cache) { + packet_off =3D meta->dreg.packet_off + + off; + knod_bpf_store_cache_size(priv, meta, + &p64[0], + &pkt_cache[0], + sizeof(u64), + packet_off); + } else { + knod_emit(priv, meta, + global_store_dwordx2, + p64[0].lo, + bpf_reg64[d].lo, off); + } + knod_iset32(&p32[0], imm); + } else { + WARN_ON_ONCE(1); + } + break; + case BPF_JMP32 | BPF_JA | BPF_K: + if (meta->branch_type =3D=3D KNOD_BR_DIRECT_EXIT) { + knod_bpf_emit_direct_exit_retval(priv, meta, + meta->merge_point); + + /* Unconditional goto exit: + * all active lanes done + */ + knod_emit(priv, meta, s_or_b64, + knod_prog->done_mask_sreg, + knod_prog->done_mask_sreg, + AMDGCN_SREG_EXEC_LO); + knod_emit(priv, meta, s_mov_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_INTEGER_0); + } else if (meta->branch_type =3D=3D KNOD_BR_FORWARD_GOTO) { + /* Structurized: save all active lanes, clear + * EXEC. Lanes resume at merge_point (target). + */ + knod_emit(priv, meta, s_mov_b64, + meta->exec_save_sreg, + AMDGCN_SREG_EXEC_LO); + knod_emit(priv, meta, s_mov_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_INTEGER_0); + } else { + /* Reorder classifies every JA as FORWARD_GOTO + * or DIRECT_EXIT; reaching here is a bug. + */ + WARN_ON_ONCE(1); + } + break; + case BPF_JMP | BPF_JA | BPF_K: + if (meta->branch_type =3D=3D KNOD_BR_DIRECT_EXIT) { + knod_bpf_emit_direct_exit_retval(priv, meta, + meta->merge_point); + + /* Unconditional goto exit: + * all active lanes done + */ + knod_emit(priv, meta, s_or_b64, + knod_prog->done_mask_sreg, + knod_prog->done_mask_sreg, + AMDGCN_SREG_EXEC_LO); + knod_emit(priv, meta, s_mov_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_INTEGER_0); + } else if (meta->branch_type =3D=3D KNOD_BR_FORWARD_GOTO) { + /* Structurized: save all active lanes, clear + * EXEC. Lanes resume at merge_point (target). + */ + knod_emit(priv, meta, s_mov_b64, + meta->exec_save_sreg, + AMDGCN_SREG_EXEC_LO); + knod_emit(priv, meta, s_mov_b64, + AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_INTEGER_0); + } else { + /* Reorder classifies every JA as FORWARD_GOTO + * or DIRECT_EXIT; reaching here is a bug. + */ + WARN_ON_ONCE(1); + } + break; + case BPF_JMP32 | BPF_JEQ | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JEQ | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u64, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JEQ | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_eq_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JEQ | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_eq_u64, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JGT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_gt_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JGT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_gt_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JGT | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_gt_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JGT | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_gt_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JGE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_ge_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JGE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_ge_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JGE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_ge_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JGE | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_ge_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JLT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_lt_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JLT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_lt_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JLT | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_lt_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JLT | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_lt_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JLE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_le_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JLE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_le_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JLE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_le_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JLE | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], 2 * 2); + knod_emit(priv, meta, v_cmp_le_u64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSGT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_gt_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSGT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_gt_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSGT | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_gt_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSGT | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_gt_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSGE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_ge_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSGE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_ge_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSGE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_ge_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSGE | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_ge_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSLT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_lt_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSLT | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_lt_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSLT | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_lt_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSLT | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_lt_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSLE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_le_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSLE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_le_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSLE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_le_i32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSLE | BPF_X: + knod_vset64(¶m64[0], d * 2); + knod_vset64(¶m64[1], s * 2); + knod_emit(priv, meta, v_cmp_le_i64, param64[0], + param64[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSET | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], d * 2); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSET | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], d * 2); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_vset32(¶m[0], (d * 2) + 1); + knod_vset32(¶m[1], (d * 2) + 1); + knod_vset32(¶m[2], KNOD_AMDGPU_TMP_VREG0_HI); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u64, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JSET | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], d * 2); + knod_vset32(¶m[2], s * 2); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JSET | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], d * 2); + knod_vset32(¶m[2], s * 2); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_vset32(¶m[0], (d * 2) + 1); + knod_vset32(¶m[1], (d * 2) + 1); + knod_vset32(¶m[2], (s * 2) + 1); + knod_emit(priv, meta, v_and_b32_e32, param[0], + param[1], param[2]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JNE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JNE | BPF_K: + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_LO); + knod_iset32(¶m[1], imm); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], KNOD_AMDGPU_TMP_VREG0_HI); + knod_iset32(¶m[1], 0); + knod_emit(priv, meta, v_mov_b32_e32, param[0], + param[1]); + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], KNOD_AMDGPU_TMP_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u64, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_JNE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_eq_u32, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP | BPF_JNE | BPF_X: + knod_vset32(¶m[0], d * 2); + knod_vset32(¶m[1], s * 2); + knod_emit(priv, meta, v_cmp_eq_u64, param[0], + param[1]); + knod_bpf_emit_branch_tail(priv, meta, knod_prog, off); + break; + case BPF_JMP32 | BPF_CALL: + case BPF_JMP | BPF_CALL: + switch (imm) { + case 1: + if (map_id =3D=3D -1) { + WARN_ON_ONCE(1); + break; + } + knod_bpf_map_lookup(priv, meta, map_id); + map_id =3D -1; + break; + case 2: { + struct knod_bpf_map_obj *_map_obj; + + if (map_id =3D=3D -1) { + WARN_ON_ONCE(1); + break; + } + _map_obj =3D knod_bpf_map_kaddr(priv, map_id); + if (!_map_obj) { + WARN_ON_ONCE(1); + break; + } + if (_map_obj->map_type =3D=3D BPF_MAP_TYPE_ARRAY) + knod_bpf_map_update_array(priv, meta, + map_id); + else if (_map_obj->map_type =3D=3D + BPF_MAP_TYPE_HASH) + knod_bpf_map_update_hash(priv, meta, + map_id); + else + WARN_ON_ONCE(1); + map_id =3D -1; + break; + } + case 3: { + struct knod_bpf_map_obj *_map_obj; + + if (map_id =3D=3D -1) { + WARN_ON_ONCE(1); + break; + } + _map_obj =3D knod_bpf_map_kaddr(priv, map_id); + if (!_map_obj) { + WARN_ON_ONCE(1); + break; + } + if (_map_obj->map_type =3D=3D BPF_MAP_TYPE_ARRAY) + knod_bpf_map_delete_array(priv, meta, + map_id); + else if (_map_obj->map_type =3D=3D + BPF_MAP_TYPE_HASH) + knod_bpf_map_delete_hash(priv, meta, + map_id); + else + WARN_ON_ONCE(1); + map_id =3D -1; + break; + } + case 5: + knod_bpf_ktime_get_ns(priv, meta); + break; + case 44: + knod_bpf_xdp_adjust_head(priv, meta); + break; + case 65: + knod_bpf_xdp_adjust_tail(priv, meta); + break; + default: + WARN_ON_ONCE(1); + break; + } + break; + case BPF_JMP32 | BPF_EXIT: + case BPF_JMP | BPF_EXIT: + /* Structurized CFG: BPF_EXIT is NOT a terminator. + * Mark all active lanes as done and clear EXEC. + * Actual exit handling (retval store, pkt_cache, + * PASS block, s_endpgm) is in the unified + * fallthrough EXIT at the end of the stream. + * This follows the LLVM StructurizeCFG model where + * all lanes must reach the single exit point. + */ + knod_emit(priv, meta, s_or_b64, + knod_prog->done_mask_sreg, + knod_prog->done_mask_sreg, + AMDGCN_SREG_EXEC_LO); + + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_INTEGER_0); + break; + case BPF_ALU | BPF_END | BPF_TO_BE: { + struct amdgcn_param32 v_dst_lo, v_dst_hi, v_tmp, s_sel; + + knod_vset32(&v_dst_lo, d * 2); + knod_vset32(&v_dst_hi, d * 2 + 1); + knod_vset32(&v_tmp, KNOD_AMDGPU_TMP_VREG0_LO); + knod_sset32(&s_sel, KNOD_AMDGPU_TMP_SREG0_LO); + + switch (imm) { + case 16: + /* bswap16+zext: {0,0,byte0,byte1} */ + knod_iset32(¶m[0], 0x0C0C0001); + knod_emit(priv, meta, s_mov_b32, s_sel, + param[0]); + + knod_emit(priv, meta, v_perm_b32, v_dst_lo, + v_dst_lo, v_dst_lo, s_sel); + + knod_iset32(¶m[0], 0); + knod_emit(priv, meta, v_mov_b32_e32, v_dst_hi, + param[0]); + break; + case 32: + /* bswap32+zext */ + knod_iset32(¶m[0], 0x00010203); + knod_emit(priv, meta, s_mov_b32, s_sel, + param[0]); + + knod_emit(priv, meta, v_perm_b32, v_dst_lo, + v_dst_lo, v_dst_lo, s_sel); + + knod_iset32(¶m[0], 0); + knod_emit(priv, meta, v_mov_b32_e32, v_dst_hi, + param[0]); + break; + case 64: { + struct amdgcn_param32 v_src_hi; + + knod_vset32(&v_src_hi, d * 2 + 1); + + /* bswap32 selector */ + knod_iset32(¶m[0], 0x00010203); + knod_emit(priv, meta, s_mov_b32, s_sel, + param[0]); + + /* tmp =3D bswap32(lo) */ + knod_emit(priv, meta, v_perm_b32, v_tmp, + v_dst_lo, v_dst_lo, s_sel); + + /* new_lo =3D bswap32(hi) */ + knod_emit(priv, meta, v_perm_b32, v_dst_lo, + v_src_hi, v_src_hi, s_sel); + + /* new_hi =3D tmp (bswap32(old_lo)) */ + knod_emit(priv, meta, v_mov_b32_e32, v_dst_hi, + v_tmp); + break; + } + default: + WARN_ON_ONCE(1); + break; + } + break; + } + case BPF_ALU | BPF_END | BPF_TO_LE: { + struct amdgcn_param32 v_dst_lo, v_dst_hi; + + knod_vset32(&v_dst_lo, d * 2); + knod_vset32(&v_dst_hi, d * 2 + 1); + + switch (imm) { + case 16: + knod_iset32(¶m[0], 0xFFFF); + knod_emit(priv, meta, v_and_b32_e32, v_dst_lo, + param[0], v_dst_lo); + + knod_iset32(¶m[0], 0); + knod_emit(priv, meta, v_mov_b32_e32, v_dst_hi, + param[0]); + break; + case 32: + knod_iset32(¶m[0], 0); + knod_emit(priv, meta, v_mov_b32_e32, v_dst_hi, + param[0]); + break; + case 64: + break; + default: + WARN_ON_ONCE(1); + break; + } + break; + } + default: + WARN_ON_ONCE(1); + break; + } + + WARN_ON(meta->amdgpu_insns >=3D KNOD_META_INSNS); + for (i =3D 0; i < meta->amdgpu_insns; i++) + insn_idx +=3D (meta->amdgpu_insn[i].size / 4); + } + + /* Fallthrough EXIT: publish a verdict for every in-bounds lane. + * Lanes that did not reach BPF_EXIT are forced to XDP_DROP below. + */ + meta =3D kzalloc(sizeof(*meta), GFP_KERNEL); + if (!meta) + return -ENOMEM; + list_add_tail(&meta->l, &knod_prog->post_insns); + + /* Any in-bounds lane outside done_mask gets a conservative DROP + * verdict instead of publishing stale VGPR state or leaving the + * recycle-time poison in bd->act. + */ + knod_emit(priv, meta, s_andn2_b64, AMDGCN_SREG_EXEC_LO, + knod_prog->initial_exec_sreg, knod_prog->done_mask_sreg); + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_iset32(&p[1], XDP_DROP); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* Store bd->act for every lane that participated in this dispatch. + * Done lanes retain their low32 action in v0; unfinished lanes publish + * the fallback DROP written above. + */ + knod_emit(priv, meta, s_mov_b64, AMDGCN_SREG_EXEC_LO, + knod_prog->initial_exec_sreg); + + /* BPF/XDP verdicts are low32; do not spend a second GTT dword per + * packet. + */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG0_LO); + knod_vset32(&p[1], KNOD_AMDGPU_SLOT_VREG_LO); + knod_emit(priv, meta, global_store_dword, p[0], p[1], + offsetof(struct spsc_bd, act)); + + if (knod_prog->uses_adjust) + knod_bpf_emit_offlen_writeback(priv, meta); + + /* pkt_cache writeback: flush modified packet data back to VRAM */ + if (knod_bpf_pkt_cache) { + knod_vset32(&p[0], r32[0].v); + knod_vset32(&p[1], + KNOD_AMDGPU_DATA_VREG_LO); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + knod_vset32(&p[0], r32[0].v + 1); + knod_vset32(&p[1], + KNOD_AMDGPU_DATA_VREG_HI); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + knod_global_store_size_cache(priv, meta, + &pkt_cache[0], + r32[0], + 0, /* dst index */ + 0, /* start offset */ + knod_prog->max_packet_off); + } + + /* XDP_PASS detection */ + knod_iset32(&p[0], XDP_PASS); + knod_vset32(&p[1], KNOD_AMDGPU_VREG0_LO); + knod_emit(priv, meta, v_cmp_eq_u32, p[0], p[1]); + + pass_branch_idx =3D meta->amdgpu_insns; + knod_emit(priv, meta, s_cbranch_vccz, 0); + + /* EXEC &=3D VCC (only PASS lanes) */ + knod_emit(priv, meta, s_and_b64, AMDGCN_SREG_EXEC_LO, + AMDGCN_SREG_EXEC_LO, AMDGCN_SREG_VCC_LO); + + /* v_mov param addr to VGPR pair */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_LO); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_HI); + knod_sset32(&p[1], KNOD_AMDGPU_PARAM_SREG_HI); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* Per-queue pass_count: offset TMP_VREG9 by queue_idx * 4 */ + /* v_mov_b32 v2, s15 (queue_idx -> VGPR) */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG1_LO); + knod_sset32(&p[1], + KNOD_AMDGPU_WORKGROUP_ID_Y_SREG); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* v_lshlrev_b32 v2, 2, v2 (queue_idx * 4) */ + knod_vset32(&p[0], KNOD_AMDGPU_VREG1_LO); + knod_iset32(&p[1], 2); + knod_vset32(&p[2], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, meta, v_lshlrev_b32, p[0], p[1], p[2]); + + /* v_add_u32 v40, v2, v40 (param_addr_lo +=3D queue_idx * 4) */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_VREG1_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, meta, v_add_u32, p[0], p[1], p[2]); + + /* v_mov TMP10_LO, 1 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_iset32(&p[1], 1); + knod_emit(priv, meta, v_mov_b32_e32, p[0], p[1]); + + /* global_atomic_add TMP10_LO, TMP9, TMP10_LO, + * offsetof(pass_count) + * GLC=3D1 to receive old_val in vdst (needed for per-lane slot + * index). With GLC=3D0 vdst is NOT written, leaving TMP10_LO + * as the addend (1) -- every PASS lane then computes slot=3D1 + * and races on the same pass_meta_buf entry, leaving slot 0 empty. + */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG10_LO); + knod_emit(priv, meta, global_atomic_add, p[0], p[1], p[2], + offsetof(struct knod_bpf_param, pass_count), 1); + + /* s_waitcnt vmcnt(0) */ + knod_emit(priv, meta, s_waitcnt_vmcnt); + + /* v_sub_u32 v40, v40, v2 (restore param_addr_lo) */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[2], KNOD_AMDGPU_VREG1_LO); + knod_emit(priv, meta, v_sub_u32, p[0], p[1], p[2]); + + /* old_val * 2 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG10_LO); + knod_iset32(&p[1], 1); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG10_LO); + knod_emit(priv, meta, v_lshlrev_b32, p[0], p[1], p[2]); + + /* addr_lo +=3D old_val * 2 */ + knod_vset32(&p[0], KNOD_AMDGPU_TMP_VREG9_LO); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG10_LO); + knod_vset32(&p[2], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, meta, v_add_u32, p[0], p[1], p[2]); + + /* global_store_short pass_indices[old_val], + * BACKLOG_IDX_VREG + * dst=3Ddata, src=3Daddr in wrapper convention + */ + knod_vset32(&p[0], + KNOD_AMDGPU_BACKLOG_IDX_VREG); + knod_vset32(&p[1], KNOD_AMDGPU_TMP_VREG9_LO); + knod_emit(priv, meta, global_store_short, p[0], p[1], + offsetof(struct knod_bpf_param, pass_indices)); + + /* Copy PASS packet data (shader mode) or store src addr (SDMA mode) */ + knod_emit_pass_addr_store(priv, meta); + + /* Patch branch offset */ + pass_dwords =3D 0; + + for (j =3D pass_branch_idx + 1; j < meta->amdgpu_insns; j++) + pass_dwords +=3D meta->amdgpu_insn[j].size / 4; + emit_s_cbranch_vccz(priv->isa_version, + &meta->amdgpu_insn[pass_branch_idx], + pass_dwords); + + knod_emit(priv, meta, s_endpgm); + + for (j =3D 0; j < meta->amdgpu_insns; j++) + insn_idx +=3D meta->amdgpu_insn[j].size / 4; + + if (priv->isa_version =3D=3D 10) { + if (insn_idx % 256) { + meta =3D kzalloc_obj(*meta, GFP_KERNEL); + if (!meta) + return -ENOMEM; + list_add_tail(&meta->l, &knod_prog->post_insns); + } + + while (insn_idx % 256) { + if (meta->amdgpu_insns >=3D KNOD_META_INSNS) { + meta =3D kzalloc_obj(*meta, GFP_KERNEL); + if (!meta) + return -ENOMEM; + list_add_tail(&meta->l, &knod_prog->post_insns); + } + knod_emit(priv, meta, s_code_end); + insn_idx +=3D + meta->amdgpu_insn[meta->amdgpu_insns - 1].size / + 4; + } + } + + return 0; +} + +static int knod_bpf_translate(struct bpf_prog *prog) +{ + struct knod_prog *knod_prog =3D prog->aux->offload->dev_priv; + struct knod_dev *knodev =3D knod_prog->knodev; + int ret; + + knod_bpf_map_setup(prog); + ret =3D knod_bpf_jit(knodev, knod_prog); + if (ret < 0) { + pr_err("knod: failed to JIT: %d\n", ret); + return ret; + } + + knod_setup_bpf_prog(prog); + + return 0; +} + +static void knod_bpf_destroy_prog(struct bpf_prog *prog) +{ + struct knod_prog *knod_prog =3D prog->aux->offload->dev_priv; + struct knod_dev *knodev =3D knod_prog->knodev; + struct knod_bpf_priv *priv =3D knodev->accel->xdp.priv; + + /* + * Normally the prog was already uninstalled (offload with a NULL prog + * flipped back to pass). Guard the abnormal path where the prog is + * freed while still tracked: flip to pass first so the worker stops + * dispatching this code. The compiled code lives in a kernel slot and + * is no longer read once we flip away; knod_prog is CPU-only IR the GPU + * never touches, so it is safe to free synchronously. + */ + if (priv && READ_ONCE(priv->prog) =3D=3D prog) { + WRITE_ONCE(priv->prog, NULL); + knod_bpf_reload_pass(knodev); + } + knod_prog_free(knod_prog); +} + +static const struct bpf_prog_offload_ops knod_bpf_dev_ops =3D { + .insn_hook =3D knod_bpf_verify_insn, + .finalize =3D knod_bpf_finalize, + .prepare =3D knod_bpf_verifier_prep, + .translate =3D knod_bpf_translate, + .destroy =3D knod_bpf_destroy_prog, +}; + +static int knod_bpf_setup_prog_hw_checks(struct knod_dev *knodev, + struct netdev_bpf *bpf) +{ + if (!bpf->prog) + return 0; + + return 0; +} + +static int knod_bpf_map_get_next_key(struct bpf_offloaded_map *offmap, + void *key, void *next_key) +{ + unsigned int *nkey =3D (unsigned int *)next_key; + unsigned int *_key =3D (unsigned int *)key; + + if (offmap->map.map_type =3D=3D BPF_MAP_TYPE_ARRAY || + offmap->map.map_type =3D=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + if (key =3D=3D NULL) + *nkey =3D 0; + else + *nkey =3D (*_key) + 1; + + if (*nkey >=3D offmap->map.max_entries) + return -ENOENT; + } else if (offmap->map.map_type =3D=3D BPF_MAP_TYPE_HASH) { + if (key =3D=3D NULL) + return knod_bpf_map_hash_get_first_key(offmap, + next_key); + else + return knod_bpf_map_hash_get_next_key(offmap, key, + nkey); + } + + return 0; +} + +static int knod_bpf_map_lookup_elem(struct bpf_offloaded_map *offmap, + void *key, void *value) +{ + return __knod_bpf_map_lookup_elem(offmap, key, value); +} + +static int knod_bpf_map_update_elem(struct bpf_offloaded_map *offmap, + void *key, void *value, u64 flags) +{ + return __knod_bpf_map_update_elem(offmap, key, value, flags); +} + +static int knod_bpf_map_delete_elem(struct bpf_offloaded_map *offmap, void= *key) +{ + return __knod_bpf_map_delete_elem(offmap, key); +} + +static const struct bpf_map_dev_ops knod_bpf_map_ops =3D { + .map_get_next_key =3D knod_bpf_map_get_next_key, + .map_lookup_elem =3D knod_bpf_map_lookup_elem, + .map_update_elem =3D knod_bpf_map_update_elem, + .map_delete_elem =3D knod_bpf_map_delete_elem, +}; + +static int knod_bpf_map_alloc(struct knod_dev *knodev, + struct bpf_offloaded_map *offmap) +{ + int err; + + if (offmap->map.map_type !=3D BPF_MAP_TYPE_ARRAY && + offmap->map.map_type !=3D BPF_MAP_TYPE_HASH && + offmap->map.map_type !=3D BPF_MAP_TYPE_PERCPU_ARRAY) { + knod_jit_dbg(" unsupported map type: %d\n", + offmap->map.map_type); + return -EOPNOTSUPP; + } + + err =3D __knod_bpf_map_alloc(knodev, offmap); + if (err) { + knod_jit_dbg(" err =3D %d\n", err); + return err; + } + + offmap->dev_ops =3D &knod_bpf_map_ops; + return 0; +} + +static int knod_bpf_xdp_install(struct knod_dev *knodev, + struct netdev_bpf *bpf) +{ + int err =3D 0; + + ASSERT_RTNL(); + + switch (bpf->command) { + case XDP_SETUP_PROG: + WARN_ON_ONCE(1); + break; + case XDP_SETUP_PROG_HW: + err =3D knod_bpf_setup_prog_hw_checks(knodev, bpf); + if (err) + return err; + + err =3D knod_bpf_xdp_set_prog(knodev, bpf); + break; + case BPF_OFFLOAD_MAP_ALLOC: + err =3D knod_bpf_map_alloc(knodev, bpf->offmap); + break; + case BPF_OFFLOAD_MAP_FREE: + knod_bpf_map_free(knodev, bpf->offmap); + break; + default: + knod_jit_dbg(" bpf->command =3D %d\n", bpf->command); + err =3D -EINVAL; + break; + } + + return err; +} + +static inline int bpf_debugfs_insn(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct seq_file *m, + int insn_idx) +{ + struct amdgcn_insn *insn =3D &meta->amdgpu_insn[insn_idx]; + + if (priv->isa_version =3D=3D 10) + gfx10_debugfs_insn(insn, m); + else if (priv->isa_version =3D=3D 9) + gfx9_debugfs_insn(insn, m); + else + WARN_ON_ONCE(1); + + return insn->size; +} + +/* + * Print one disassembled GPU instruction at @offset, then drop the disasm= 's + * trailing newline and append @tag as a right-hand comment aligned to a f= ixed + * column (tabs expand to 8) so the origin lines up regardless of mnemonic + * width. Returns the instruction size in dwords. + */ +static int bpf_debugfs_insn_tagged(struct knod_bpf_priv *priv, + struct knod_insn_meta *meta, + struct seq_file *m, int j, + int offset, const char *tag) +{ + size_t col, p, line_start =3D m->count; + int sz; + + seq_printf(m, "%d:\t", offset); + sz =3D bpf_debugfs_insn(priv, meta, m, j); + if (seq_has_overflowed(m)) + return sz; + + if (m->count > line_start && m->buf[m->count - 1] =3D=3D '\n') + m->count--; + col =3D 0; + for (p =3D line_start; p < m->count; p++) + col =3D m->buf[p] =3D=3D '\t' ? (col + 8) & ~(size_t)7 : col + 1; + while (col < 96) { + seq_putc(m, ' '); + col++; + } + seq_printf(m, " ; %s\n", tag); + + return sz; +} + +/* + * Print the instructions a second time, re-sorted into BPF source order s= o the + * dump reads like the program. The offsets are the real (reordered) GPU + * offsets, so they appear out of sequence - that shows where the reorder + * placed each block. Synthetic jumps have no BPF source insn and are las= t. + */ +static void bpf_insn_show_bpf_order(struct knod_bpf_priv *priv, + struct seq_file *m) +{ + struct knod_insn_meta *meta; + int idx, max_idx =3D -1, off2, i; + bool synth_hdr =3D false; + char tag[24]; + + seq_puts(m, "=3D=3D=3D[INSTRUCTIONS (bpf order)]=3D=3D=3D\n"); + + list_for_each_entry(meta, &priv->knod_prog->insns, l) + if (meta->bpf_insn_idx > max_idx) + max_idx =3D meta->bpf_insn_idx; + + for (idx =3D 0; idx <=3D max_idx; idx++) { + list_for_each_entry(meta, &priv->knod_prog->insns, l) { + if (meta->bpf_insn_idx !=3D idx || !meta->amdgpu_insns) + continue; + scnprintf(tag, sizeof(tag), "bpf#%d", idx); + off2 =3D meta->amdgpu_insn_idx; + for (i =3D 0; i < meta->amdgpu_insns; i++) + off2 +=3D bpf_debugfs_insn_tagged(priv, meta, m, + i, off2, tag); + } + } + + list_for_each_entry(meta, &priv->knod_prog->insns, l) { + if (meta->bpf_insn_idx >=3D 0 || !meta->amdgpu_insns) + continue; + if (!synth_hdr) { + seq_puts(m, " [synthetic jumps]\n"); + synth_hdr =3D true; + } + scnprintf(tag, sizeof(tag), "synth JA->#%d", + meta->jmp_dst ? meta->jmp_dst->bpf_insn_idx : -1); + off2 =3D meta->amdgpu_insn_idx; + for (i =3D 0; i < meta->amdgpu_insns; i++) + off2 +=3D bpf_debugfs_insn_tagged(priv, meta, m, + i, off2, tag); + } +} + +static int bpf_insn_show(struct seq_file *m, void *v) +{ + struct knod_bpf_priv *priv =3D (struct knod_bpf_priv *)m->private; + struct knod_insn_meta *meta; + struct knod_prog *kp; + int i, insn_idx =3D 0; + bool have_prog; + + if (!priv) + return 0; + + /* + * Show the kernel the GPU actually dispatches: the XDP prog when one is + * attached, otherwise the retained pass-through kernel. + */ + have_prog =3D READ_ONCE(priv->prog); + if (have_prog) { + kp =3D priv->knod_prog; + } else { + kp =3D priv->pass_knod_prog; + seq_puts(m, "no XDP prog attached -- pass-through kernel:\n"); + } + if (!kp) + return 0; + + seq_puts(m, "=3D=3D=3D[PROLOGUE]=3D=3D=3D\n"); + list_for_each_entry(meta, &kp->pre_insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) { + seq_printf(m, "%d:\t", insn_idx); + insn_idx +=3D bpf_debugfs_insn(priv, meta, m, i); + } + } + + /* Emission (RPO) order - the actual GPU layout. Each line is tagged + * with its origin BPF insn since the reorder makes this differ from the + * BPF byte order; synthetic jumps inserted by the reorder have none. + */ + seq_puts(m, "=3D=3D=3D[INSTRUCTIONS]=3D=3D=3D\n"); + list_for_each_entry(meta, &kp->insns, l) { + char tag[24]; + + if (meta->bpf_insn_idx < 0) + scnprintf(tag, sizeof(tag), "synth JA->#%d", + meta->jmp_dst ? + meta->jmp_dst->bpf_insn_idx : -1); + else + scnprintf(tag, sizeof(tag), "bpf#%d", + meta->bpf_insn_idx); + + for (i =3D 0; i < meta->amdgpu_insns; i++) + insn_idx +=3D bpf_debugfs_insn_tagged(priv, meta, m, i, + insn_idx, tag); + } + + seq_puts(m, "=3D=3D=3D[EPILOG]=3D=3D=3D\n"); + list_for_each_entry(meta, &kp->post_insns, l) { + for (i =3D 0; i < meta->amdgpu_insns; i++) { + seq_printf(m, "%d:\t", insn_idx); + insn_idx +=3D bpf_debugfs_insn(priv, meta, m, i); + } + } + + if (have_prog) + bpf_insn_show_bpf_order(priv, m); + + return 0; +} + +static int bpf_insn_open(struct inode *inode, struct file *file) +{ + return single_open(file, bpf_insn_show, inode->i_private); +} + +static const struct file_operations bpf_insn_fops =3D { + .owner =3D THIS_MODULE, + .open =3D bpf_insn_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static const char *knod_branch_type_str(enum knod_branch_type type) +{ + switch (type) { + case KNOD_BR_NONE: return "NONE"; + case KNOD_BR_DIRECT_EXIT: return "DIRECT_EXIT"; + case KNOD_BR_FORWARD_SKIP: return "FORWARD_SKIP"; + case KNOD_BR_FORWARD_GOTO: return "FORWARD_GOTO"; + default: return "UNKNOWN"; + } +} + +static int bpf_cfg_show(struct seq_file *m, void *v) +{ + struct knod_bpf_priv *priv =3D (struct knod_bpf_priv *)m->private; + struct knod_insn_meta *meta; + + if (!priv || !priv->knod_prog) + return 0; + + seq_puts(m, "=3D=3D=3D[STRUCTURIZED CFG]=3D=3D=3D\n"); + seq_printf(m, "exec_save_pairs_used: %u\n", + priv->knod_prog->exec_save_pairs_used); + seq_printf(m, "done_mask: s[%d:%d]\n", + priv->knod_prog->done_mask_sreg, + priv->knod_prog->done_mask_sreg + 1); + seq_printf(m, "initial_exec: s[%d:%d]\n", + priv->knod_prog->initial_exec_sreg, + priv->knod_prog->initial_exec_sreg + 1); + seq_puts(m, "\n"); + + seq_printf(m, "%-6s %-8s %-14s %-10s %-10s %-8s\n", + "bpf#", "opcode", "branch_type", "exec_save", "merge_pt", + "is_merge"); + + list_for_each_entry(meta, &priv->knod_prog->insns, l) { + bool is_jmp =3D is_mbpf_jmp(meta); + + if (!is_jmp && !meta->is_merge_point) + continue; + + seq_printf(m, "%-6d 0x%02x ", + meta->bpf_insn_idx, meta->insn.code); + + if (meta->branch_type !=3D KNOD_BR_NONE) { + seq_printf(m, "%-14s s[%d:%d] ", + knod_branch_type_str(meta->branch_type), + meta->exec_save_sreg, + meta->exec_save_sreg + 1); + if (meta->merge_point) + seq_printf(m, "%-10d ", + meta->merge_point->bpf_insn_idx); + else + seq_printf(m, "%-10s ", "-"); + } else if (is_jmp) { + seq_printf(m, "%-14s %-10s %-10s ", + knod_branch_type_str(KNOD_BR_NONE), + "-", "-"); + } else { + seq_printf(m, "%-14s %-10s %-10s ", + "", "", ""); + } + + if (meta->is_merge_point) { + struct knod_insn_meta *br; + + seq_puts(m, "YES restore:"); + list_for_each_entry(br, &priv->knod_prog->insns, l) { + if ((br->branch_type =3D=3D KNOD_BR_FORWARD_SKIP || + br->branch_type =3D=3D KNOD_BR_FORWARD_GOTO) && + br->merge_point =3D=3D meta) + seq_printf(m, " s[%d:%d](from bpf#%d)", + br->exec_save_sreg, + br->exec_save_sreg + 1, + br->bpf_insn_idx); + } + seq_puts(m, "\n"); + } else { + seq_puts(m, "\n"); + } + } + + /* Basic-block CFG from the reorder analysis (origin BPF order). */ + if (priv->knod_prog->bbs) { + struct knod_bb *bbs =3D priv->knod_prog->bbs; + int nb =3D priv->knod_prog->n_bbs; + int k, s; + + seq_printf(m, "\n[BASIC BLOCKS] %d blocks, %d back-edge(s) -> %s\n", + nb, priv->knod_prog->n_back, + priv->knod_prog->n_back ? "HAS LOOP" : "DAG"); + + for (k =3D 0; k < nb; k++) { + struct knod_bb *bb =3D &bbs[k]; + + seq_printf(m, "BB%-3d bpf#%d..#%d rpo=3D%d idom=3D#%d succ=3D{", + k, bb->leader->bpf_insn_idx, + bb->last->bpf_insn_idx, bb->rpo, + bb->idom ? + bb->idom->leader->bpf_insn_idx : -1); + for (s =3D 0; s < bb->n_succ; s++) + seq_printf(m, "%s#%d", s ? "," : "", + bb->succ[s]->leader->bpf_insn_idx); + seq_printf(m, "}%s\n", + bb->loop_header ? " LOOP_HDR" : ""); + } + } + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(bpf_cfg); + +static int knod_stats_show(struct seq_file *s, void *unused) +{ + struct knod_bpf_priv *priv =3D s->private; + u64 p50 =3D 0, p99 =3D 0, p999 =3D 0, acc; + struct knod_bpf_stats *stats; + u64 ccnt, dcnt; + int i; + + stats =3D &priv->stats; + ccnt =3D stats->completion_count; + dcnt =3D stats->dispatch_count; + seq_printf(s, "enabled: %s\n", + static_branch_unlikely(&knod_stats_key) ? "yes" : "no"); + + seq_puts(s, "\n--- dispatch ---\n"); + seq_printf(s, "count: %llu\n", dcnt); + seq_printf(s, "avg_ns: %llu\n", + dcnt ? stats->dispatch_total_ns / dcnt : 0); + seq_printf(s, "max_ns: %llu\n", stats->dispatch_max_ns); + seq_printf(s, "backlogs_avg: %llu\n", + dcnt ? stats->backlogs_total / dcnt : 0); + + seq_puts(s, "\nbacklogs histogram:\n"); + for (i =3D 0; i < KNOD_BL_BUCKETS; i++) + seq_printf(s, " %-10s %llu\n", + bl_labels[i], stats->backlogs_hist[i]); + + seq_puts(s, "\n--- completion ---\n"); + seq_printf(s, "count: %llu\n", ccnt); + seq_printf(s, "avg_ns: %llu\n", + ccnt ? stats->completion_total_ns / ccnt : 0); + seq_printf(s, "max_ns: %llu\n", + stats->completion_max_ns); + + seq_puts(s, "\nlatency histogram:\n"); + for (i =3D 0; i < KNOD_LAT_BUCKETS; i++) + seq_printf(s, " %-10s %llu\n", + lat_labels[i], stats->completion_hist[i]); + + if (ccnt) { + acc =3D 0; + for (i =3D 0; i < KNOD_LAT_BUCKETS; i++) { + acc +=3D stats->completion_hist[i]; + if (!p50 && acc * 1000 >=3D ccnt * 500) + p50 =3D i; + if (!p99 && acc * 1000 >=3D ccnt * 990) + p99 =3D i; + if (!p999 && acc * 1000 >=3D ccnt * 999) + p999 =3D i; + } + seq_printf(s, "\np50: %s\n", lat_labels[p50]); + seq_printf(s, "p99: %s\n", lat_labels[p99]); + seq_printf(s, "p999: %s\n", lat_labels[p999]); + } + + seq_puts(s, "\n--- decode_act ---\n"); + seq_printf(s, "count: %llu\n", stats->decode_act_count); + seq_printf(s, "avg_ns: %llu\n", + stats->decode_act_count ? + stats->decode_act_total_ns / stats->decode_act_count : 0); + seq_printf(s, "max_ns: %llu\n", stats->decode_act_max_ns); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(knod_stats); + +static ssize_t knod_stats_enable_write(struct file *file, + const char __user *buf, + size_t count, loff_t *ppos) +{ + bool val; + + if (kstrtobool_from_user(buf, count, &val)) + return -EINVAL; + + if (val) + static_branch_enable(&knod_stats_key); + else + static_branch_disable(&knod_stats_key); + + return count; +} + +static ssize_t knod_stats_enable_read(struct file *file, + char __user *buf, + size_t count, loff_t *ppos) +{ + char tmp[4]; + int len; + + len =3D scnprintf(tmp, sizeof(tmp), "%d\n", + static_branch_unlikely(&knod_stats_key) ? 1 : 0); + + return simple_read_from_buffer(buf, count, ppos, tmp, len); +} + +static const struct file_operations knod_stats_enable_fops =3D { + .owner =3D THIS_MODULE, + .read =3D knod_stats_enable_read, + .write =3D knod_stats_enable_write, +}; + +static ssize_t knod_stats_reset_write(struct file *file, + const char __user *buf, + size_t count, loff_t *ppos) +{ + struct knod_bpf_priv *priv =3D file->private_data; + + memset(&priv->stats, 0, sizeof(priv->stats)); + return count; +} + +static const struct file_operations knod_stats_reset_fops =3D { + .owner =3D THIS_MODULE, + .open =3D simple_open, + .write =3D knod_stats_reset_write, +}; + +static int knod_debugfs_init(struct knod_bpf_priv *priv) +{ + struct dentry *dir =3D priv->knod->debug_dir; + struct dentry *bpf_dir; + + if (!dir) + return -ENOENT; + + bpf_dir =3D debugfs_create_dir("bpf", dir); + if (IS_ERR(bpf_dir)) + return PTR_ERR(bpf_dir); + + priv->debug_dir =3D bpf_dir; + + debugfs_create_file("insn", 0644, + bpf_dir, priv, &bpf_insn_fops); + debugfs_create_file("cfg", 0444, bpf_dir, priv, + &bpf_cfg_fops); + debugfs_create_file("stats", 0444, bpf_dir, priv, + &knod_stats_fops); + debugfs_create_file("stats_enable", 0644, bpf_dir, priv, + &knod_stats_enable_fops); + debugfs_create_file("stats_reset", 0200, bpf_dir, priv, + &knod_stats_reset_fops); + debugfs_create_bool("poll_mode", 0644, bpf_dir, &knod_bpf_poll_mode); + debugfs_create_u32("dispatch_delay_us", 0644, bpf_dir, + &knod_bpf_dispatch_delay_us); + + return 0; +} + +static void knod_debugfs_cleanup(struct knod_bpf_priv *priv) +{ + if (!priv->debug_dir) + return; + + debugfs_remove_recursive(priv->debug_dir); + priv->debug_dir =3D NULL; +} + +/* Called when attached or module loading time */ +/* attach: allocate the permanent per-attach priv struct. */ +static int knod_accel_xdp_init(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv; + + priv =3D __knod_accel_xdp_init(accel, knodev); + if (IS_ERR(priv)) + return PTR_ERR(priv); + return 0; +} + +/* detach: free the permanent priv struct. */ +static void knod_accel_xdp_exit(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv =3D accel->xdp.priv; + + __knod_accel_xdp_exit(accel, priv); +} + +/* + * Feature select, phase B: register the BPF offload device so user XDP + * progs/maps can bind to it. Called after ->activate() set up the GPU + * buffers, while xdp_ops already points at the BPF ops. + */ +static int knod_bpf_offload_init(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv =3D accel->xdp.priv; + struct bpf_offload_dev *bpf_dev; + int err; + + bpf_dev =3D bpf_offload_dev_create(&knod_bpf_dev_ops, priv); + err =3D PTR_ERR_OR_ZERO(bpf_dev); + if (err) + return err; + err =3D bpf_offload_dev_netdev_register(bpf_dev, knodev->netdev); + if (err) { + bpf_offload_dev_destroy(bpf_dev); + return err; + } + knod_debugfs_init(priv); + accel->xdp.bpf_dev =3D bpf_dev; + return 0; +} + +/* + * Feature deselect, phase 1: unregister the BPF offload device. This + * force-frees any user XDP progs/maps still bound; the map-free ndo is + * routed back through accel_ops.xdp_ops->xdp_install, so the caller keeps + * xdp_ops pointed at the BPF ops until this returns. + */ +static void knod_bpf_offload_uninit(struct knod_dev *knodev) +{ + struct knod_accel *accel =3D knodev->accel; + struct knod_bpf_priv *priv =3D accel->xdp.priv; + + knod_debugfs_cleanup(priv); + bpf_offload_dev_netdev_unregister(accel->xdp.bpf_dev, knodev->netdev); + bpf_offload_dev_destroy(accel->xdp.bpf_dev); + accel->xdp.bpf_dev =3D NULL; +} + +struct knod_accel_xdp_ops accel_xdp_ops =3D { + /* attach/detach: permanent priv struct */ + .init =3D &knod_accel_xdp_init, + .exit =3D &knod_accel_xdp_exit, + /* feature select: GPU compute buffers (A) + offload dev (B) */ + .activate =3D &knod_bpf_activate, + .deactivate =3D &knod_bpf_deactivate, + .busy =3D &knod_bpf_busy, + .xdp_offload_init =3D &knod_bpf_offload_init, + .xdp_offload_uninit =3D &knod_bpf_offload_uninit, + /* interface up/down (or feature switch): worker + GPU drain */ + .start =3D &knod_bpf_start, + .stop =3D &knod_bpf_stop, + .xdp_install =3D &knod_bpf_xdp_install, +}; + +static int __init knod_bpf_init_module(void) +{ + pr_info("knod-bpf module load\n"); + + /* knod_accel_xdp_register() already calls xdp_ops->init() on every + * registered accel, so a second per-accel init loop here would just + * re-create the "bpf" debugfs dir ("already exists" warning) and leak + * a duplicate offload dev. + */ + knod_dev_lock(); + knod_accel_xdp_register(&accel_xdp_ops); + knod_dev_unlock(); + + return 0; +} +late_initcall(knod_bpf_init_module); + +static void __exit knod_bpf_cleanup_module(void) +{ + struct knod_bpf_priv *priv, *tmp; + struct knod_accel *accel; + + rtnl_lock(); + knod_dev_lock(); + list_for_each_entry_safe(priv, tmp, &priv_list, list) { + accel =3D priv->accel; + if (accel->knodev) + accel_xdp_ops.exit(accel->knodev); + } + knod_accel_xdp_unregister(); + knod_dev_unlock(); + rtnl_unlock(); + pr_info("knod-bpf module unload\n"); +} +module_exit(knod_bpf_cleanup_module); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Taehee Yoo "); +MODULE_DESCRIPTION("AMDGPU BPF offload backend"); +MODULE_VERSION("multi-aql"); diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h b/drivers/gpu/drm/a= md/amdkfd/knod/knod_bpf.h new file mode 100644 index 000000000000..de6df06c4f2f --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h @@ -0,0 +1,597 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KFD_BPF_H_INCLUDED +#define KFD_BPF_H_INCLUDED + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "knod_amdgpu_insn.h" +#include "../../../../../../net/core/devmem.h" +#include "../amdgpu/amdgpu_vm.h" +#include "knod_bpf.h" +#include "kfd_knod.h" + +#define KNOD_BPF_BACKLOGS_MAX 65536 +#define KNOD_BPF_INFLIGHT 3 /* triple-buffered dispatches */ +#define KNOD_BPF_WORKGROUPS_DEFAULT 256 +#define KNOD_BPF_WORKGROUPS_MIN 64 +#define KNOD_BPF_WORKGROUPS_MAX 256 +#define KNOD_BPF_EXPIRE_DEFAULT 10 +#define KNOD_BPF_EXPIRE_MIN 1 +#define KNOD_BPF_EXPIRE_MAX 1000 +#define QUEUE_SIZE_DGPU 8192 +#define QUEUE_SIZE_IGPU 2048 +#define KNOD_MAX_BDS (KNOD_BPF_BACKLOGS_MAX / KNOD_SPSC_MAX) + +#define MAX_KEY_SIZE 64 /* 64Bytes */ +#define MAX_PACKET_CACHE 256 /* 256Bytes */ + +#define knod_prog_first_meta(knod_prog) \ + list_first_entry(&(knod_prog)->insns, struct knod_insn_meta, l) +#define knod_prog_last_meta(knod_prog) \ + list_last_entry(&(knod_prog)->insns, struct knod_insn_meta, l) +#define knod_prog_pre_last_meta(knod_prog) \ + list_last_entry(&(knod_prog)->pre_insns, struct knod_insn_meta, l) +#define knod_meta_next(meta) list_next_entry(meta, l) +#define knod_meta_prev(meta) list_prev_entry(meta, l) + +#define knod_for_each_insn_walk2(knod_prog, pos, next) \ + for (pos =3D list_first_entry(&(knod_prog)->insns, typeof(*pos), l),\ + next =3D list_next_entry(pos, l); \ + &(knod_prog)->insns !=3D &pos->l && \ + &(knod_prog)->insns !=3D &next->l; \ + pos =3D knod_meta_next(pos), \ + next =3D knod_meta_next(pos)) + +#define knod_for_each_insn_walk3(knod_prog, pos, next, next2) \ + for (pos =3D list_first_entry(&(knod_prog)->insns, typeof(*pos), l),\ + next =3D list_next_entry(pos, l), \ + next2 =3D list_next_entry(next, l); \ + &(knod_prog)->insns !=3D &pos->l && \ + &(knod_prog)->insns !=3D &next->l && \ + &(knod_prog)->insns !=3D &next2->l; \ + pos =3D knod_meta_next(pos), \ + next =3D knod_meta_next(pos), \ + next2 =3D knod_meta_next(next)) + +struct xdp_md_obj { + u64 data; + u64 data_end; + u64 data_meta; + /* Below access go through struct xdp_rxq_info */ + u64 ingress_ifindex; /* rxq->dev->ifindex */ + u64 rx_queue_index; /* rxq->queue_index */ + + u64 egress_ifindex; /* txq->dev->ifindex */ + u64 retval; +}; + +struct knod_bpf_subparam_obj { + struct xdp_md_obj ctx; +}; + +#define KNOD_BPF_HASH_NEXT_END 0x7FFFFFFFU +#define KNOD_BPF_HASH_NEXT_DELETED 0x80000000U +#define KNOD_BPF_HASH_NEXT_MASK 0x7FFFFFFFU + +struct knod_bpf_hash_elem_obj { + unsigned int next; + unsigned char kv[]; +}; + +struct knod_bpf_map_hash_meta_obj { + unsigned int n_buckets; + unsigned int hashrnd; + unsigned int cur; + unsigned int elem_size; + void *q; + void *elems; + unsigned int gc_count; + void *gc_list; +}; + +struct knod_bpf_map_array_meta_obj { + u32 per_instance_size; /* value_size * max_entries (one instance) */ + u32 n_instances; /* 1 for ARRAY, num_possible_cpus for PERCPU */ +}; + +union knod_bpf_map_meta_obj { + struct knod_bpf_map_hash_meta_obj hmeta; + struct knod_bpf_map_array_meta_obj ameta; +}; + +struct knod_bpf_map_obj { + enum bpf_map_type map_type; + unsigned int key_size; + unsigned int value_size; + unsigned int max_entries; + unsigned int id; + unsigned long map_extra; /* any per-map-type extra fields */ + unsigned int map_flags; + union knod_bpf_map_meta_obj meta; + int mutex; + unsigned char bucket[]; +}; + +struct knod_bpf_map { + struct list_head list; + struct knod_mem *mem, *queue_mem, *hash_elems_mem, *gc_mem; + /* ptr to mem_k->kaddr */ + struct knod_bpf_map_obj *knod_map_obj; + struct bpf_offloaded_map *offmap; + struct knod_bpf_priv *priv; +}; + +struct knod_bpf_queue_desc { + u64 pool_gaddr; /* SPSC pool GTT address for this queue */ + u64 base_gaddr; /* dma-buf base address for this queue */ + u32 count; /* number of packets from this queue */ + /* was start_idx; kept for global_load_dwordx4 layout */ + u32 _pad; + u32 ring_start; /* acquired cursor at peek time */ + u32 ring_mask; /* capacity - 1 */ +}; + +struct knod_bpf_param { + u32 nr_backlogs; + u32 nr_queues; + u32 spsc_stride; + u32 _pad0; + u64 ktime_ns; /* snapshot of ktime_get_ns() at dispatch */ + u32 pass_count[KNOD_SPSC_MAX]; /* per-queue atomic XDP_PASS counter */ + /* per-queue GTT pass_meta_buf GPU addr */ + u64 pass_meta_buf_gaddr[KNOD_SPSC_MAX]; + struct knod_bpf_queue_desc queues[KNOD_SPSC_MAX]; + /* backlog indices of PASS packets */ + u16 pass_indices[KNOD_BPF_BACKLOGS_MAX]; + struct knod_bpf_subparam_obj sub[KNOD_BPF_BACKLOGS_MAX]; +}; + +struct knod_packet { + union { + netmem_ref netmem; + void *kaddr; + }; + u16 len; + u16 off; +}; + +/* Single Queue Worok */ +struct knod_bpf_work_sq { + struct list_head list; + struct knod_mem *param; + int queue_idx[KNOD_SPSC_MAX]; + struct spsc_bd *bds[KNOD_BPF_BACKLOGS_MAX]; + ktime_t dispatch_time; + s64 sigval; + unsigned long expire; + int backlogs; +}; + +struct knod_bpf_reg_state { + struct bpf_reg_state reg; + int stack_off; + int packet_off; + bool var_off; +}; + +/* Structurized CFG branch types */ +enum knod_branch_type { + KNOD_BR_NONE =3D 0, /* not a branch */ + /* backward jump to exit: inline retval + done_mask update */ + KNOD_BR_DIRECT_EXIT, + KNOD_BR_FORWARD_SKIP, /* forward jump: skip region via EXEC mask */ + KNOD_BR_FORWARD_GOTO, /* forward jump crossing other branch scopes */ +}; + +#define KNOD_META_INSNS 1024 +#define AMDGPU_INSN_SKIP -1 +struct knod_insn_meta { + struct bpf_insn insn; + short bpf_insn_idx; + + struct amdgcn_insn amdgpu_insn[KNOD_META_INSNS]; + u32 amdgpu_insn_idx; + u32 amdgpu_insns; + + union { + /* pointer ops (ld/st/xadd) */ + struct { + struct bpf_reg_state ptr; + struct bpf_insn *paired_st; + s16 ldst_gather_len; + bool ptr_not_const; + struct { + s16 range_start; + s16 range_end; + bool do_init; + } pkt_cache; + bool xadd_over_16bit; + bool xadd_maybe_16bit; + }; + /* jump */ + struct { + struct knod_insn_meta *jmp_dst; + bool jump_neg_op; + u32 num_insns_after_br; /* only for BPF-to-BPF calls */ + /* structurized CFG */ + enum knod_branch_type branch_type; + /* SGPR index for s_and_saveexec_b64 */ + u8 exec_save_sreg; + /* where EXEC is restored */ + struct knod_insn_meta *merge_point; + }; + /* function calls */ + struct { + u32 func_id; + struct bpf_reg_state arg1; + struct knod_bpf_reg_state arg2; + }; + /* We are interested in range info for operands of ALU + * operations. For example, shift amount, multiplicand and + * multiplier etc. + */ + struct { + u64 umin_src; + u64 umax_src; + u64 umin_dst; + u64 umax_dst; + }; + }; + + struct knod_bpf_reg_state sreg; + struct knod_bpf_reg_state dreg; + struct knod_bpf_reg_state kreg; + struct knod_bpf_reg_state vreg; + unsigned int off; + unsigned short flags; + unsigned short subprog_idx; + bool is_merge_point; /* EXEC restore target */ + u8 restore_sreg; /* SGPR to restore EXEC from at merge point */ + int linear_idx; /* position in the (reordered) emission list */ + struct list_head l; +}; + +/* Encode one GPU instruction at @meta's running slot and advance it. + * @meta->amdgpu_insns is both the cursor during emission and the final + * instruction count afterwards. @fn names a knod_amdgpu_insn.h encoder + * without its emit_ prefix (e.g. v_add32 for emit_v_add32); the macro + * pastes it back, so call sites read knod_emit(priv, meta, v_add32, ...). + */ +#define knod_emit(priv, meta, fn, ...) \ + do { \ + struct knod_insn_meta *__m =3D (meta); \ + \ + emit_##fn((priv)->isa_version, \ + &__m->amdgpu_insn[__m->amdgpu_insns], \ + ##__VA_ARGS__); \ + debug_insn((priv)->isa_version, \ + &__m->amdgpu_insn[__m->amdgpu_insns]); \ + __m->amdgpu_insns++; \ + } while (0) + +/* JIT debug/error trace: auto-prefix with "knod_jit :". + * knod_jit_dbg() is a pr_debug(), so it is off by default and toggled + * with dynamic debug; knod_jit_err() always fires. + */ +#define knod_jit_dbg(fmt, ...) \ + pr_debug("knod_jit %s:%d" fmt, __func__, __LINE__, ##__VA_ARGS__) +#define knod_jit_err(fmt, ...) \ + pr_err("knod_jit %s:%d" fmt, __func__, __LINE__, ##__VA_ARGS__) + +#define BPF_SIZE_MASK 0x18 + +struct knod_bb; /* basic-block CFG analysis (knod_bpf.c) */ + +struct knod_prog { + struct knod *knod; + struct knod_dev *knodev; + + u64 *prog; + unsigned int prog_len; + unsigned int __prog_alloc_len; + int max_stack_off; + int max_packet_off; + + struct knod_insn_meta *meta; + enum bpf_prog_type type; + struct list_head pre_insns; + struct list_head post_insns; + struct list_head insns; + unsigned int n_insns; + unsigned int pre_n_insns; + int insn_idx; + + /* Structurized CFG state */ + /* GFX9: 34, GFX10: 32 (s[32:33] safe on RDNA) */ + u8 done_mask_sreg; + u8 exec_save_base; /* GFX9: 36, GFX10: 34 */ + /* in-bounds EXEC snapshot for verdict publish */ + u8 initial_exec_sreg; + /* number of SGPR pairs allocated for EXEC saves */ + u8 exec_save_pairs_used; + bool uses_adjust; + + /* Basic-block CFG analysis, retained for the /bpf/cfg view. */ + struct knod_bb *bbs; + int n_bbs; + int n_back; +}; + +#define KNOD_XDP_MEMCPY 0 +#define KNOD_XDP_PT 1 +#define KNOD_XDP_NETMEM 2 +#define KNOD_XDP_NONE 3 +#define KNOD_XDP_DEFAULT KNOD_XDP_PT + +#define KNOD_LAT_BUCKETS 10 +#define KNOD_BL_BUCKETS 8 + +struct knod_bpf_stats { + u64 dispatch_total_ns; + u64 dispatch_count; + u64 dispatch_max_ns; + + u64 completion_total_ns; + u64 completion_count; + u64 completion_max_ns; + u64 completion_hist[KNOD_LAT_BUCKETS]; + + u64 backlogs_total; + u64 backlogs_hist[KNOD_BL_BUCKETS]; + + u64 decode_act_total_ns; + u64 decode_act_count; + u64 decode_act_max_ns; +}; + +#define KNOD_PASS_SLOT_SIZE PAGE_SIZE + +/* pass_meta_buf slot header, written by the shader (offsetof used by the + * codegen). The host read path is gone now that PASS delivery goes via t= he + * NIC act handler + knod_d2h_copy; the shader still stores {len, src_addr} + * here pending removal of that store. + */ +struct knod_pass_slot_hdr { + u32 len; /* packet length */ + u32 _pad; + u64 src_addr; /* VRAM source address (SDMA mode only) */ +}; + +struct knod_bpf_priv { + struct list_head list; + struct knod *knod; + struct knod_accel *accel; + struct knod_dev *knodev; + struct net_device *dev; + struct knod_prog *knod_prog; + /* retained pass IR for debugfs insn dump */ + struct knod_prog *pass_knod_prog; + struct bpf_prog *prog; + struct amdgpu_vm *vm; + u64 queue_base_gaddr[KNOD_SPSC_MAX]; + struct knod_bpf_work_sq *inflight[KNOD_BPF_INFLIGHT]; + unsigned int inflight_cnt; + ktime_t next_dispatch_time; + struct task_struct *worker_task; + struct list_head free_list_sqw; + struct mutex map_op_lock; + /* maps awaiting deferred free by the worker */ + struct list_head dead_maps; + u32 maps_tick_skip; + struct dentry *debug_dir; + struct knod_bpf_stats stats; + void *prog_buf; + void *pass_prog_buf; + u32 pass_prog_size; + /* descriptor + live shader bytes per kernel slot */ + u32 kernel_image_len[2]; + /* knod->kernels[] slot the GPU dispatches */ + int active_idx; + /* knod->kernels[] slot holding the pass kernel */ + int pass_idx; + /* + * XDP_PASS shader-to-GTT metadata (shader-written; host read path + * removed). GTT metadata: shader-written headers. + */ + struct knod_mem *pass_meta_buf; + u32 pass_pkts_per_queue; /* backlogs / nr_works */ + /* batch size per queue */ + int batch_size; + int nr_works; + int isa_version; + bool installing_kernel; + int start; +}; + +static inline u8 mbpf_class(const struct knod_insn_meta *meta) +{ + return BPF_CLASS(meta->insn.code); +} + +static inline u8 mbpf_src(const struct knod_insn_meta *meta) +{ + return BPF_SRC(meta->insn.code); +} + +static inline u8 mbpf_op(const struct knod_insn_meta *meta) +{ + return BPF_OP(meta->insn.code); +} + +static inline u8 mbpf_mode(const struct knod_insn_meta *meta) +{ + return BPF_MODE(meta->insn.code); +} + +static inline bool is_mbpf_alu(const struct knod_insn_meta *meta) +{ + return mbpf_class(meta) =3D=3D BPF_ALU64 || mbpf_class(meta) =3D=3D BPF_A= LU; +} + +static inline bool is_mbpf_load(const struct knod_insn_meta *meta) +{ + return (meta->insn.code & ~BPF_SIZE_MASK) =3D=3D (BPF_LDX | BPF_MEM); +} + +static inline bool is_mbpf_jmp32(const struct knod_insn_meta *meta) +{ + return mbpf_class(meta) =3D=3D BPF_JMP32; +} + +static inline bool is_mbpf_jmp64(const struct knod_insn_meta *meta) +{ + return mbpf_class(meta) =3D=3D BPF_JMP; +} + +static inline bool is_mbpf_jmp(const struct knod_insn_meta *meta) +{ + return is_mbpf_jmp32(meta) || is_mbpf_jmp64(meta); +} + +static inline bool is_mbpf_store(const struct knod_insn_meta *meta) +{ + return (meta->insn.code & ~BPF_SIZE_MASK) =3D=3D (BPF_STX | BPF_MEM); +} + +static inline bool is_mbpf_load_pkt(const struct knod_insn_meta *meta) +{ + return is_mbpf_load(meta) && meta->ptr.type =3D=3D PTR_TO_PACKET; +} + +static inline bool is_mbpf_store_pkt(const struct knod_insn_meta *meta) +{ + return is_mbpf_store(meta) && meta->ptr.type =3D=3D PTR_TO_PACKET; +} + +static inline bool is_mbpf_classic_load(const struct knod_insn_meta *meta) +{ + u8 code =3D meta->insn.code; + + return BPF_CLASS(code) =3D=3D BPF_LD && + (BPF_MODE(code) =3D=3D BPF_ABS || BPF_MODE(code) =3D=3D BPF_IND); +} + +static inline bool is_mbpf_classic_store(const struct knod_insn_meta *meta) +{ + u8 code =3D meta->insn.code; + + return BPF_CLASS(code) =3D=3D BPF_ST && BPF_MODE(code) =3D=3D BPF_MEM; +} + +static inline bool is_mbpf_classic_store_pkt(const struct knod_insn_meta *= meta) +{ + return is_mbpf_classic_store(meta) && meta->ptr.type =3D=3D PTR_TO_PACKET; +} + +static inline bool is_mbpf_atomic(const struct knod_insn_meta *meta) +{ + return (meta->insn.code & ~BPF_SIZE_MASK) =3D=3D (BPF_STX | BPF_ATOMIC); +} + +static inline bool is_mbpf_mul(const struct knod_insn_meta *meta) +{ + return is_mbpf_alu(meta) && mbpf_op(meta) =3D=3D BPF_MUL; +} + +static inline bool is_mbpf_div(const struct knod_insn_meta *meta) +{ + return is_mbpf_alu(meta) && mbpf_op(meta) =3D=3D BPF_DIV; +} + +static inline bool is_mbpf_mod(const struct knod_insn_meta *meta) +{ + return is_mbpf_alu(meta) && mbpf_op(meta) =3D=3D BPF_MOD; +} + +static inline bool is_mbpf_cond_jump(const struct knod_insn_meta *meta) +{ + u8 op; + + if (is_mbpf_jmp32(meta)) + return true; + + if (!is_mbpf_jmp64(meta)) + return false; + + op =3D mbpf_op(meta); + return op !=3D BPF_JA && op !=3D BPF_EXIT && op !=3D BPF_CALL; +} + +static inline bool is_mbpf_helper_call(const struct knod_insn_meta *meta) +{ + struct bpf_insn insn =3D meta->insn; + + return insn.code =3D=3D (BPF_JMP | BPF_CALL) && + insn.src_reg !=3D BPF_PSEUDO_CALL; +} + +static inline bool is_mbpf_pseudo_call(const struct knod_insn_meta *meta) +{ + struct bpf_insn insn =3D meta->insn; + + return insn.code =3D=3D (BPF_JMP | BPF_CALL) && + insn.src_reg =3D=3D BPF_PSEUDO_CALL; +} + +static inline bool is_mbpf_map_call(const struct knod_insn_meta *meta) +{ + struct bpf_insn insn =3D meta->insn; + + return insn.code =3D=3D (BPF_JMP | BPF_CALL) && insn.imm <=3D 3; +} + +#define STACK_FRAME_ALIGN 64 + +#define FLAG_INSN_IS_JUMP_DST BIT(0) +#define FLAG_INSN_IS_SUBPROG_START BIT(1) +#define FLAG_INSN_PTR_CALLER_STACK_FRAME BIT(2) +/* Instruction is pointless, noop even on its own */ +#define FLAG_INSN_SKIP_NOOP BIT(3) +/* Instruction is optimized out based on preceding instructions */ +#define FLAG_INSN_SKIP_PREC_DEPENDENT BIT(4) +/* Instruction is optimized by the verifier */ +#define FLAG_INSN_SKIP_VERIFIER_OPT BIT(5) +/* Instruction needs to zero extend to high 32-bit */ +#define FLAG_INSN_DO_ZEXT BIT(6) + +#define FLAG_INSN_SKIP_MASK (FLAG_INSN_SKIP_NOOP | \ + FLAG_INSN_SKIP_PREC_DEPENDENT | \ + FLAG_INSN_SKIP_VERIFIER_OPT) +#endif --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B88353B19AE for ; Sun, 19 Jul 2026 18:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484117; cv=none; b=C5O4gA9JiuO5h7SKtlPIxlBPLzby7galEKkA3ghM15vilsh/npXLBesAqrJM6Xs7/xMU4ESI6sGKgIRY9Z6q+PUIHLJ3QGi3/iTc6GHuxuY1GitYTuFolIzPrG4mtP7C6SV37Q9j2ypHBop17ryC8fxCXaOwyps1yWTr5ixkjcQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484117; c=relaxed/simple; bh=iJlLK+zT6pOkAKcyL3dSdt3Ey5y77wGrNw+PwirXbBA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iP3CoJX2BGgdJ9WAXXXqZtf1g9I7t0trnbJenNM+1upn7WKwTX2VkDF+01EUujASS4vqiVXcSTB21mJ512fXZYfddAV3hbeTml4kWjGMQIcUtgzZypLa4M+nERTUuG9eOeN78l4it0zRCRlVL8D8XklhlWU7CxMTOCa4f+WBUIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rlZ/hw2i; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rlZ/hw2i" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2cc7e86e7aeso65280735ad.2 for ; Sun, 19 Jul 2026 11:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484102; x=1785088902; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=Mn9ILc7KfHYPQN6h07+6ERrU/MrbOq4Gmsuwq4UE7rc=; b=rlZ/hw2i4piMPY6y/ki04yW0MoWxVxDOsWzF2l8FUO3hiiy0ia5TCGA7i4a4wTWZjA Oe6Rq1O5yCsu1hQaKMVh2Fmgy5NogpndpfuJgRN52G9hgRYsVkYjoxjzoTuFJUUJM5XJ vjXBQszYQzV3W2XQKPeSBYS94L9B+HIDF4rLSDdWxqioVDyb0ixzc5UlJmA4bwFVlOEb ZSyQmzCG+Ud0qts+HSfp2ECvls+04tIGjTn/fRHZ6O9gF6fBtdx+zi7shDmIj9HQFZYk Kp/C+c/1/k5Wqg0flp0SFLbLK1gVSy3WVaRvJUB7W1R8V/k2Drk49a6+ERd7S+rrEv9q J9xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484102; x=1785088902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=Mn9ILc7KfHYPQN6h07+6ERrU/MrbOq4Gmsuwq4UE7rc=; b=mnIO4zqZUl9yAku/BovRmw8vPOT5tp3Th+Mxj7QEBrZ1uVoQOhiFWteWCDabofF2cp IKLDpRjPc7B8HzF9gBbnpqoNa5KG2mPEvMhqwfO1Ru7KZYck6Z4cjdyfRwPR0yOXfTyg 3M0O3P6TTn5Oy/ubCX3UZx/IJfw+kksP1CmN1jFXcfyQqFabxsYVy1KW6eUYtvvYA4Af luk6GyYwCnpzfJaOo5cv/YFGxTjd78sUIdz/pHq+D1I6hEXBEmWMg2hmfpKnudxxy2ea v9m48DxJwOkbo08a3Vf9up20ZxnXgTnFTwZNvzNgYYcv/uvo7tX0586B2yXW8FeDppnY v8nw== X-Forwarded-Encrypted: i=1; AHgh+RqTNPHj6mG6c28woD7HIO7kqOjAPZBXRNdlq33RNOZsfFaH26v0gg1GkF3bD4fDgegSdt7PKmfB4vwtdtY=@vger.kernel.org X-Gm-Message-State: AOJu0Yzsu9rXe7vW/v4JmMGl3fA/iroZTkq90Wyr2ciBZpcYV0GcOWY1 UN40sEM8h3ER+Hu6h9tqroL9c2XTDk6yjk90HF8sB+0dELx5BGCD5nYa X-Gm-Gg: AfdE7cl5wxj6pmAAi7ajY5UjxcbLcx99gRdNvZbmL402nZ2JUnKh5bmnPsikrlDvCol KpeMqPmAv5UB5xq9cEvon/KI3EuV5un7DQ8cDQTFkIOlk/o12jHnZxdkGC5IxtrjUcaJDe/PM7x lp4yqRoUDe6XUzPgAHm/2CYfdK9rpXKg4r80kcmSELQXyvTeV13W98Mn1+ohyUQmYyYvvvBVASJ 0KmWQjfpVT/AAYHlBNT4NmccTOKZCH77Ya3VNyKGif9RKVgX1Hy1j06JDUUqyN153MIltrdaLjx iHzSytSm1g9O0iYBFZuKK1rM5jT8JVf9YaKbOfEPt0VQDKNxhpAcg0emrcMpyKh4LZeB+BIe0Bp dp6LFulIk0UMMlxsfeMdw2w5bxmCsmw1DjO2JY1ZWImhvn2tKKg/f+wyJomJh9jbjyw== X-Received: by 2002:a17:902:d586:b0:2ca:4cfd:a6df with SMTP id d9443c01a7336-2cf34a0b335mr124560325ad.43.1784484101843; Sun, 19 Jul 2026 11:01:41 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.01.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:01:40 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 10/13] net/mlx5e: add knod XDP offload support Date: Sun, 19 Jul 2026 17:58:54 +0000 Message-ID: <20260719175857.4071636-11-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let mlx5e act as a knod NIC: register with the knod core, feed received packets to the accelerator over the per-queue SPSC ring, and transmit accelerator verdicts (XDP_TX) back through the XDP SQ. Restricted to inline-none TX and single-page MTU. Signed-off-by: Taehee Yoo (cherry picked from commit 218c3a122d6b53638b99441560b7a0da65f6d8c1) --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 10 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 351 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 19 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 38 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 107 +++++- .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 45 +++ 6 files changed, 564 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/eth= ernet/mellanox/mlx5/core/en.h index d507289096c2..1401a24e1740 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -568,6 +569,8 @@ struct mlx5e_icosq { =20 struct mlx5e_frag_page { netmem_ref netmem; + struct page_pool *pp; + u32 page_idx; u16 frags; }; =20 @@ -739,6 +742,11 @@ struct mlx5e_rq { =20 struct mlx5e_xdp_buff mxbuf; =20 + struct knod_dev *knodev; + struct knod_netdev *knetdev; + u32 knod_spsc_prod_head; + bool knod_spsc_prod_valid; + /* AF_XDP zero-copy */ struct xsk_buff_pool *xsk_pool; =20 @@ -985,6 +993,8 @@ struct mlx5e_priv { struct dentry *dfs_root; struct mlx5_devcom_comp_dev *devcom; struct ethtool_fec_hist_range *fec_ranges; + struct knod_dev *knodev; + struct knod_netdev *knetdev; }; =20 static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net= /ethernet/mellanox/mlx5/core/en/xdp.c index d8c7cb8837d7..82adfc9b7c1b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -35,8 +35,18 @@ #include "en/xdp.h" #include "en/params.h" #include +#include #include =20 +INDIRECT_CALLABLE_SCOPE bool +mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptx= d, + int check_result, struct xsk_tx_metadata *meta); + +static inline struct page_pool *mlx5e_knod_bd_pp(struct spsc_bd *bd) +{ + return likely(bd->pp) ? bd->pp : netmem_get_pp(bd->netmem); +} + int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_rq_opt_param *rqo) { @@ -58,6 +68,36 @@ int mlx5e_xdp_max_mtu(struct mlx5e_params *params, return MLX5E_HW2SW_MTU(params, SKB_MAX_HEAD(hr)); } =20 +static inline bool mlx5e_xmit_xdp_offload_buff(struct mlx5e_xdpsq *sq, + struct mlx5e_rq *rq, + struct spsc_bd *bd) +{ + struct mlx5e_xmit_data_frags xdptxdf =3D {}; + struct mlx5e_xmit_data *xdptxd; + + /* attach is restricted to inline-none NICs, so the WQE inlines no + * header and xdptxd->data is never read (left NULL here). + */ + xdptxd =3D &xdptxdf.xd; + xdptxd->len =3D bd->len; + xdptxd->has_frags =3D 0; + xdptxd->dma_addr =3D netmem_to_net_iov(bd->netmem)->desc.dma_addr + + bd->off; + + if (!mlx5e_xmit_xdp_frame(sq, xdptxd, 0, NULL)) + return false; + + mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, + (union mlx5e_xdp_info) { + .mode =3D MLX5E_XDP_XMIT_MODE_OFFLOAD }); + mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, + (union mlx5e_xdp_info) { + .offload.netmem =3D bd->netmem, + .offload.pp =3D mlx5e_knod_bd_pp(bd) }); + + return true; +} + static inline bool mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, struct xdp_buff *xdp) @@ -353,6 +393,154 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, } } =20 +static inline u16 mlx5e_xdpsq_get_avail(struct mlx5e_xdpsq *sq) +{ + if (sq->pc =3D=3D sq->cc) + return sq->wq.fbc.sz_m1 + 1; + + return sq->wq.fbc.sz_m1 & (sq->cc - sq->pc); +} + +static inline u16 mlx5e_xdpsq_get_avail_after_poll(struct mlx5e_xdpsq *sq) +{ + u16 avail =3D mlx5e_xdpsq_get_avail(sq); + + if (likely(avail)) + return avail; + + mlx5e_xmit_xdp_doorbell(sq); + mlx5e_poll_xdpsq_cq(&sq->cq); + + return mlx5e_xdpsq_get_avail(sq); +} + +struct mlx5e_knod_release_batch { + struct spsc_bd *bds[NAPI_POLL_WEIGHT]; + struct spsc_pass_bd pass[NAPI_POLL_WEIGHT]; +}; + +static struct mlx5e_knod_release_batch +mlx5e_knod_release_batch[KNOD_SPSC_MAX]; + +static noinline int +mlx5e_rx_offload_release_pending(struct mlx5e_rq *rq, + struct knod_work_priv *wpriv, + bool flush, int budget) +{ + struct knod_dev *knodev =3D rq->knodev; + struct mlx5e_xdpsq *sq =3D rq->xdpsq; + struct mlx5e_knod_release_batch *batch =3D + &mlx5e_knod_release_batch[rq->ix]; + struct spsc_bd **bds =3D batch->bds; + struct spsc_pass_bd *pass =3D batch->pass; + int cnt, i, done =3D 0; + + while (done < budget) { + int pass_cnt =3D 0; + + if (!mlx5e_xdpsq_get_avail_after_poll(sq)) + break; + cnt =3D min(NAPI_POLL_WEIGHT, budget - done); + + spsc_release(&wpriv->spsc_bds, (void **)bds, cnt, &cnt); + if (!cnt) + break; + + for (i =3D 0; i < cnt; i++) { + switch ((u32)bds[i]->act) { + case KNOD_ACT_INFLIGHT: + fallthrough; + case KNOD_IPSEC_INFLIGHT: + goto stop_release; + case KNOD_IPSEC_PASS: + fallthrough; + case KNOD_IPSEC_DROP: + /* Finish worker has set the final verdict. + * Safe to recycle the netmem page now. + */ + page_pool_recycle_direct_netmem( + mlx5e_knod_bd_pp(bds[i]), + bds[i]->netmem); + break; + case KNOD_TX: + if (!mlx5e_xmit_xdp_offload_buff(rq->xdpsq, rq, + bds[i])) + goto stop_release; + break; + case XDP_DROP: + fallthrough; + case XDP_ABORTED: + rq->stats->xdp_drop++; + page_pool_recycle_direct_netmem( + mlx5e_knod_bd_pp(bds[i]), + bds[i]->netmem); + break; + case XDP_PASS: + /* Hand to the common device->host delivery: + * accumulate here, flush to knod_d2h_copy + * after the bd loop. The source page is + * recycled by knod_d2h_drain once the copy + * has landed, so it is NOT recycled here. + */ + pass[pass_cnt].netmem =3D bds[i]->netmem; + pass[pass_cnt].page_idx =3D bds[i]->page_idx; + pass[pass_cnt].off =3D bds[i]->off; + pass[pass_cnt].len =3D bds[i]->len; + pass_cnt++; + break; + case XDP_REDIRECT: + /* No redirect delivery path yet; recycle. */ + page_pool_recycle_direct_netmem( + mlx5e_knod_bd_pp(bds[i]), + bds[i]->netmem); + break; + default: + /* Unknown value: either the accel shader + * did not stamp a verdict for this slot + * (lane skip bug) or the slot never went + * through an accel at all. Treat as DROP + + * recycle + WARN so the ring keeps advancing. + */ + rq->stats->xdp_drop++; + pr_warn_ratelimited("mlx5 nod: invalid bd->act=3D0x%llx rq%d, treating= as DROP\n", + bds[i]->act, rq->ix); + page_pool_recycle_direct_netmem( + mlx5e_knod_bd_pp(bds[i]), + bds[i]->netmem); + break; + } + } +stop_release: + spsc_release_commit(&wpriv->spsc_bds, i); + done +=3D i; + + /* Issue the device->host copies for this batch's PASS bds. */ + if (pass_cnt) + knod_d2h_copy(knodev, rq->ix, pass, pass_cnt); + + if (i < cnt) + break; + } + + if (flush) + mlx5e_xmit_xdp_doorbell(sq); + + return done; +} + +int mlx5e_rx_offload_act_handler(struct mlx5e_rq *rq, bool flush, int budg= et) +{ + struct knod_work_priv *wpriv =3D &rq->knodev->wpriv[rq->ix]; + + if (!spsc_pending(&wpriv->spsc_bds)) { + if (flush) + mlx5e_xmit_xdp_doorbell(rq->xdpsq); + return 0; + } + + return mlx5e_rx_offload_release_pending(rq, wpriv, flush, budget); +} + static u16 mlx5e_xdpsq_get_next_pi(struct mlx5e_xdpsq *sq, u16 size) { struct mlx5_wq_cyc *wq =3D &sq->wq; @@ -741,6 +929,18 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *= sq, (*xsk_frames)++; break; } + case MLX5E_XDP_XMIT_MODE_OFFLOAD: { + netmem_ref netmem; + struct page_pool *pp; + + xdpi =3D mlx5e_xdpi_fifo_pop(xdpi_fifo); + netmem =3D xdpi.offload.netmem; + pp =3D xdpi.offload.pp; + + page_pool_recycle_direct_netmem(pp, netmem); + + break; + } default: WARN_ON_ONCE(true); } @@ -974,3 +1174,154 @@ void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool = is_mpw) sq->xmit_xdp_frame =3D is_mpw ? mlx5e_xmit_xdp_frame_mpwqe : mlx5e_xmit_xdp_frame; } + +static int mlx5e_rx_offload_xdp_attach(struct knod_dev *knodev) +{ + struct mlx5e_priv *priv =3D netdev_priv(knodev->netdev); + struct mlx5e_params *params =3D &priv->channels.params; + int max_mtu =3D mlx5e_xdp_max_mtu(params, NULL); + + if (knodev->netdev->mtu > max_mtu) { + netdev_warn(knodev->netdev, + "MTU %u too big for single-page RX offload (max %d)\n", + knodev->netdev->mtu, max_mtu); + return -EOPNOTSUPP; + } + + /* The offload TX WQE carries the packet only in a data segment; the + * eth header is not inlined (a zero dummy stands in). NICs that require + * a minimum inline header (e.g. ConnectX-4) would transmit that dummy, + * so only allow attach when the device needs no inline header. + */ + if (params->tx_min_inline_mode !=3D MLX5_INLINE_MODE_NONE) { + netdev_warn(knodev->netdev, + "knod offload requires a NIC with inline header mode 'none'\n"); + return -EOPNOTSUPP; + } + + pr_debug("Attaching XDP offload to netdev %s\n", knodev->netdev->name); + WRITE_ONCE(priv->knodev, knodev); + + return 0; +} + +static int mlx5e_rx_offload_xdp_detach(struct knod_dev *knodev) +{ + struct mlx5e_priv *priv =3D netdev_priv(knodev->netdev); + + pr_debug("Detaching XDP offload from netdev %s\n", + knodev->netdev->name); + WRITE_ONCE(priv->knodev, NULL); + + return 0; +} + +struct knod_nic_ops nic_ops =3D { + .attach =3D mlx5e_rx_offload_xdp_attach, + .detach =3D mlx5e_rx_offload_xdp_detach, +}; + +int mlx5e_knod_init(struct mlx5e_priv *priv) +{ + struct knod_netdev *knetdev; + + knetdev =3D kzalloc_obj(struct knod_netdev, GFP_KERNEL); + if (!knetdev) { + pr_debug("Failed to allocate knetdev\n"); + return -ENOMEM; + } + + INIT_LIST_HEAD(&knetdev->list); + knetdev->dev =3D priv->netdev; + knetdev->priv =3D priv; + knetdev->nic_ops =3D &nic_ops; + knetdev->owner =3D THIS_MODULE; + knetdev->flags |=3D KNOD_FLAGS_XDP; + knod_netdev_register(knetdev); + priv->knetdev =3D knetdev; + + return 0; +} + +void mlx5e_knod_uninit(struct mlx5e_priv *priv) +{ + knod_netdev_unregister(priv->knetdev); + kfree(priv->knetdev); + priv->knetdev =3D NULL; + kfree(priv->knodev); + WRITE_ONCE(priv->knodev, NULL); +} + +void mlx5e_rx_offload_set_napi(struct mlx5e_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + int i; + + if (!knodev) + return; + + for (i =3D 0; i < priv->channels.num; i++) { + struct mlx5e_channel *c =3D priv->channels.c[i]; + struct spsc_ring *r =3D &knodev->wpriv[i].spsc_bds; + + WRITE_ONCE(knodev->wpriv[i].napi, &c->napi); + c->rq.knod_spsc_prod_head =3D READ_ONCE(r->head); + c->rq.knod_spsc_prod_valid =3D true; + } +} + +void mlx5e_rx_offload_clear_napi(struct mlx5e_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + int i; + + if (!knodev) + return; + + for (i =3D 0; i < KNOD_SPSC_MAX; i++) + WRITE_ONCE(knodev->wpriv[i].napi, NULL); +} + +void mlx5e_rx_offload_start(struct mlx5e_priv *priv) +{ + if (!priv->knodev) + return; + + knod_dev_start(priv->knodev); +} + +void mlx5e_rx_offload_stop(struct mlx5e_priv *priv) +{ + struct knod_dev *knodev =3D priv->knodev; + int i; + + if (!priv->knodev) + return; + + knod_dev_stop(knodev); + + synchronize_net(); + for (i =3D 0; i < KNOD_SPSC_MAX; i++) { + struct spsc_bd *bd; + + if (i < priv->channels.num) { + struct mlx5e_rq *rq =3D &priv->channels.c[i]->rq; + + mlx5e_knod_spsc_flush(rq); + rq->knod_spsc_prod_head =3D 0; + rq->knod_spsc_prod_valid =3D false; + } + + WRITE_ONCE(knodev->wpriv[i].napi, NULL); + /* + * RX is quiesced now (worker stopped by knod_dev_stop, NAPI + * drained by synchronize_net). Return any frames the GPU + * worker did not consume back to the page_pool before the RX + * page_pool is torn down on interface down. + */ + spsc_rewind(&knodev->wpriv[i].spsc_bds); + while (!spsc_pop(&knodev->wpriv[i].spsc_bds, (void **)&bd)) + page_pool_put_full_netmem(netmem_get_pp(bd->netmem), + bd->netmem, true); + } +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net= /ethernet/mellanox/mlx5/core/en/xdp.h index 3c54f8962664..4de6babdafc5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h @@ -34,6 +34,7 @@ =20 #include #include +#include =20 #include "en.h" #include "en/txrx.h" @@ -64,6 +65,8 @@ enum mlx5e_xdp_xmit_mode { * page. The UMEM Completion Ring producer pointer has to be increased. */ MLX5E_XDP_XMIT_MODE_XSK, + + MLX5E_XDP_XMIT_MODE_OFFLOAD, }; =20 /* xmit_mode entry is pushed to the fifo per packet, followed by multiple @@ -78,6 +81,9 @@ enum mlx5e_xdp_xmit_mode { * * MLX5E_XDP_XMIT_MODE_XSK: * frame.xsk_meta. + * + * MLX5E_XDP_XMIT_MODE_OFFLOAD: + * offload.netmem. */ #define MLX5E_XDP_FIFO_ENTRIES2DS_MAX_RATIO 4 =20 @@ -93,6 +99,10 @@ union mlx5e_xdp_info { struct page *page; } page; struct xsk_tx_metadata_compl xsk_meta; + struct { + netmem_ref netmem; + struct page_pool *pp; + } offload; }; =20 struct mlx5e_xsk_param; @@ -100,14 +110,21 @@ int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_rq_opt_param *rqo); bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct bpf_prog *prog, struct mlx5e_xdp_buff *mlctx); +int mlx5e_rx_offload_act_handler(struct mlx5e_rq *rq, bool flush, int budg= et); +void mlx5e_knod_spsc_flush(struct mlx5e_rq *rq); void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq); bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq); void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq); void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw); +int mlx5e_knod_init(struct mlx5e_priv *priv); +void mlx5e_knod_uninit(struct mlx5e_priv *priv); void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq); int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frame= s, u32 flags); - +void mlx5e_rx_offload_start(struct mlx5e_priv *priv); +void mlx5e_rx_offload_stop(struct mlx5e_priv *priv); +void mlx5e_rx_offload_set_napi(struct mlx5e_priv *priv); +void mlx5e_rx_offload_clear_napi(struct mlx5e_priv *priv); extern const struct xdp_metadata_ops mlx5e_xdp_metadata_ops; extern const struct xsk_tx_metadata_ops mlx5e_xsk_tx_metadata_ops; =20 diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/ne= t/ethernet/mellanox/mlx5/core/en_main.c index aa8610cedaa8..cfc6ba6dc492 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -918,6 +918,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params, pool_size =3D 1 << params->log_rq_mtu_frames; =20 rq->mkey_be =3D cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey); + rq->knodev =3D rq->priv->knodev; =20 switch (rq->wq_type) { case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ: @@ -1022,7 +1023,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params, pp_params.queue_idx =3D rq->ix; =20 /* Shampo header data split allow for unreadable netmem */ - if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state)) + if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state) || rq->knodev) pp_params.flags |=3D PP_FLAG_ALLOW_UNREADABLE_NETMEM; =20 /* page_pool can be used even when there is no rq->xdp_prog, @@ -2873,7 +2874,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv= , int ix, c->netdev =3D priv->netdev; c->mkey_be =3D cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey); c->num_tc =3D mlx5e_get_dcb_num_tc(params); - c->xdp =3D !!params->xdp_prog; + c->xdp =3D !!params->xdp_prog || !!priv->knodev; c->stats =3D &priv->channel_stats[ix]->ch; c->aff_mask =3D irq_get_effective_affinity_mask(irq); c->lag_port =3D mlx5e_enumerate_lag_port(mdev, ix); @@ -3365,6 +3366,8 @@ void mlx5e_activate_priv_channels(struct mlx5e_priv *= priv) =20 if (priv->rx_res) mlx5e_rx_res_channels_activate(priv->rx_res, &priv->channels); + + mlx5e_rx_offload_set_napi(priv); } =20 static void mlx5e_cancel_tx_timeout_work(struct mlx5e_priv *priv) @@ -3376,6 +3379,8 @@ static void mlx5e_cancel_tx_timeout_work(struct mlx5e= _priv *priv) =20 void mlx5e_deactivate_priv_channels(struct mlx5e_priv *priv) { + mlx5e_rx_offload_clear_napi(priv); + if (priv->rx_res) mlx5e_rx_res_channels_deactivate(priv->rx_res); =20 @@ -3592,6 +3597,7 @@ int mlx5e_open(struct net_device *netdev) mlx5e_modify_admin_state(priv->mdev, MLX5_PORT_UP); mutex_unlock(&priv->state_lock); =20 + mlx5e_rx_offload_start(priv); return err; } =20 @@ -3623,6 +3629,7 @@ int mlx5e_close(struct net_device *netdev) if (!netif_device_present(netdev)) return -ENODEV; =20 + mlx5e_rx_offload_stop(priv); mutex_lock(&priv->state_lock); mlx5e_modify_admin_state(priv->mdev, MLX5_PORT_DOWN); err =3D mlx5e_close_locked(netdev); @@ -4475,7 +4482,7 @@ void mlx5e_set_xdp_feature(struct mlx5e_priv *priv) params->packet_merge.type =3D=3D MLX5E_PACKET_MERGE_NONE) val =3D NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | NETDEV_XDP_ACT_XSK_ZEROCOPY | - NETDEV_XDP_ACT_RX_SG; + NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_HW_OFFLOAD; =20 if (netdev->netdev_ops->ndo_xdp_xmit && params->xdp_prog) val |=3D NETDEV_XDP_ACT_NDO_XMIT | @@ -4738,6 +4745,14 @@ int mlx5e_change_mtu(struct net_device *netdev, int = new_mtu, goto out; } =20 + if (priv->knodev && new_mtu > mlx5e_xdp_max_mtu(&new_params, NULL)) { + netdev_warn(netdev, + "MTU %d too big for single-page RX offload (max %d)\n", + new_mtu, mlx5e_xdp_max_mtu(&new_params, NULL)); + err =3D -EINVAL; + goto out; + } + if (priv->xsk.refcnt && !mlx5e_xsk_validate_mtu(netdev, &priv->channels, &new_params, priv->mdev)) { @@ -5261,15 +5276,30 @@ static int mlx5e_xdp_set(struct net_device *netdev,= struct bpf_prog *prog) =20 static int mlx5e_xdp(struct net_device *dev, struct netdev_bpf *xdp) { + struct mlx5e_priv *priv =3D netdev_priv(dev); + struct knod_dev *knodev =3D priv->knodev; + int rc; + switch (xdp->command) { case XDP_SETUP_PROG: return mlx5e_xdp_set(dev, xdp->prog); case XDP_SETUP_XSK_POOL: return mlx5e_xsk_setup_pool(dev, xdp->xsk.pool, xdp->xsk.queue_id); + case XDP_SETUP_PROG_HW: + case BPF_OFFLOAD_MAP_ALLOC: + case BPF_OFFLOAD_MAP_FREE: + if (!knodev) + return -EOPNOTSUPP; + + rc =3D knod_dev_xdp_install(knodev, xdp); + break; + default: return -EINVAL; } + + return rc; } =20 #ifdef CONFIG_MLX5_ESWITCH @@ -6931,6 +6961,7 @@ static int _mlx5e_probe(struct auxiliary_device *adev) mlx5e_dcbnl_init_app(priv); mlx5_core_uplink_netdev_set(mdev, netdev); mlx5e_params_print_info(mdev, &priv->channels.params); + mlx5e_knod_init(priv); return 0; =20 err_resume: @@ -6982,6 +7013,7 @@ static void _mlx5e_remove(struct auxiliary_device *ad= ev) struct mlx5e_priv *priv =3D netdev_priv(netdev); struct mlx5_core_dev *mdev =3D edev->mdev; =20 + mlx5e_knod_uninit(priv); mlx5_eswitch_safe_aux_devs_remove(mdev); mlx5_core_uplink_netdev_set(mdev, NULL); =20 diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/= ethernet/mellanox/mlx5/core/en_rx.c index 6fbc0441c4b8..ff4627a0895c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -276,15 +276,20 @@ static int mlx5e_page_alloc_fragmented(struct page_po= ol *pp, struct mlx5e_frag_page *frag_page) { netmem_ref netmem =3D page_pool_dev_alloc_netmems(pp); + u32 page_idx =3D 0; =20 if (unlikely(!netmem)) return -ENOMEM; =20 page_pool_fragment_netmem(netmem, MLX5E_PAGECNT_BIAS_MAX); + if (netmem_is_net_iov(netmem)) + page_idx =3D net_iov_binding_idx(netmem_to_net_iov(netmem)); =20 *frag_page =3D (struct mlx5e_frag_page) { - .netmem =3D netmem, - .frags =3D 0, + .netmem =3D netmem, + .pp =3D pp, + .page_idx =3D page_idx, + .frags =3D 0, }; =20 return 0; @@ -1572,19 +1577,92 @@ static void mlx5e_fill_mxbuf(struct mlx5e_rq *rq, s= truct mlx5_cqe64 *cqe, mxbuf->rq =3D rq; } =20 +static inline int mlx5e_knod_spsc_produce_defer(struct mlx5e_rq *rq, + struct knod_work_priv *wpriv, + struct spsc_bd **bd) +{ + struct spsc_ring *r =3D &wpriv->spsc_bds; + unsigned int head; + unsigned int tail; + + if (unlikely(!rq->knod_spsc_prod_valid)) { + rq->knod_spsc_prod_head =3D READ_ONCE(r->head); + rq->knod_spsc_prod_valid =3D true; + } + + head =3D rq->knod_spsc_prod_head; + /* acquire tail to observe the slots the GPU worker has released */ + tail =3D smp_load_acquire(&r->tail); + if (unlikely(head - tail > r->mask)) + return -ENOSPC; + + *bd =3D r->slots[head & r->mask]; + rq->knod_spsc_prod_head =3D head + 1; + + return 0; +} + +void mlx5e_knod_spsc_flush(struct mlx5e_rq *rq) +{ + struct knod_work_priv *wpriv; + struct spsc_ring *r; + unsigned int head; + + if (unlikely(!rq->knodev || !rq->knod_spsc_prod_valid)) + return; + + wpriv =3D &rq->knodev->wpriv[rq->ix]; + r =3D &wpriv->spsc_bds; + head =3D rq->knod_spsc_prod_head; + if (head =3D=3D READ_ONCE(r->head)) + return; + + /* drain WC descriptor stores before publishing the new head */ + wmb(); + /* release: publish the produced descriptors to the GPU worker */ + smp_store_release(&r->head, head); +} + static struct sk_buff * mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info = *wi, struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { struct mlx5e_frag_page *frag_page =3D wi->frag_page; u16 rx_headroom =3D rq->buff.headroom; + struct knod_work_priv *wpriv; struct bpf_prog *prog; struct sk_buff *skb; + struct spsc_bd *bd; u32 metasize =3D 0; void *va, *data; dma_addr_t addr; u32 frag_size; =20 + if (likely(rq->knodev)) { + wpriv =3D &rq->knodev->wpriv[rq->ix]; + if (unlikely(mlx5e_knod_spsc_produce_defer(rq, wpriv, &bd))) { + mlx5e_knod_spsc_flush(rq); + mlx5e_rx_offload_act_handler(rq, false, INT_MAX); + if (mlx5e_knod_spsc_produce_defer(rq, wpriv, &bd)) { + rq->stats->buff_alloc_err++; + return NULL; + } + } + + bd->netmem =3D frag_page->netmem; + bd->pp =3D frag_page->pp; + bd->len =3D cqe_bcnt; + bd->off =3D wi->offset + rx_headroom; + bd->page_idx =3D frag_page->page_idx; + frag_page->frags++; + rq->stats->packets++; + rq->stats->bytes +=3D cqe_bcnt; + pr_debug("mlx5_nod: spsc produce q=3D%d len=3D%u\n", + rq->ix, cqe_bcnt); + + return NULL; + } + va =3D netmem_address(frag_page->netmem) + wi->offset; data =3D va + rx_headroom; frag_size =3D MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt); @@ -2101,8 +2179,10 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,= struct mlx5e_mpw_info *wi, { struct mlx5e_frag_page *frag_page =3D &wi->alloc_units.frag_pages[page_id= x]; u16 rx_headroom =3D rq->buff.headroom; + struct knod_work_priv *wpriv; struct bpf_prog *prog; struct sk_buff *skb; + struct spsc_bd *bd; u32 metasize =3D 0; void *va, *data; dma_addr_t addr; @@ -2114,6 +2194,29 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,= struct mlx5e_mpw_info *wi, return NULL; } =20 + if (likely(rq->knodev)) { + wpriv =3D &rq->knodev->wpriv[rq->ix]; + if (unlikely(mlx5e_knod_spsc_produce_defer(rq, wpriv, &bd))) { + mlx5e_knod_spsc_flush(rq); + mlx5e_rx_offload_act_handler(rq, false, INT_MAX); + if (mlx5e_knod_spsc_produce_defer(rq, wpriv, &bd)) { + rq->stats->buff_alloc_err++; + return NULL; + } + } + + bd->netmem =3D frag_page->netmem; + bd->pp =3D frag_page->pp; + bd->len =3D cqe_bcnt; + bd->off =3D head_offset + rx_headroom; + bd->page_idx =3D frag_page->page_idx; + frag_page->frags++; + rq->stats->packets++; + rq->stats->bytes +=3D cqe_bcnt; + + return NULL; + } + va =3D netmem_address(frag_page->netmem) + head_offset; data =3D va + rx_headroom; frag_size =3D MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/ne= t/ethernet/mellanox/mlx5/core/en_txrx.c index 185105606469..10c2ffd93dab 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -177,6 +177,51 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budg= et) if (likely(budget - work_done)) work_done +=3D mlx5e_poll_rx_cq(&rq->cq, budget - work_done); =20 + if (likely(rq->knodev)) + mlx5e_knod_spsc_flush(rq); + + /* KNOD release can process thousands of completed verdicts and enqueue + * XDP_TX MPWQEs. Refill RX WQEs first so the NIC is not left waiting + * for descriptors while the release side drains. The normal post below + * stays in place to publish pages recycled by this release pass. + */ + if (likely(rq->knodev)) + busy |=3D INDIRECT_CALL_2(rq->post_wqes, + mlx5e_post_rx_mpwqes, + mlx5e_post_rx_wqes, + rq); + + /* Drain SPSC bd ring + IPsec desc_ring unconditionally. + * napi_schedule from the GPU finish_worker may wake us with + * zero new CQEs, so act_handler (called per-CQE inside + * poll_rx_cq) won't run. Without this top-level call, + * PASS/DROP-stamped bds are never recycled after traffic stops + * and the SPSC ring fills up. + */ + if (likely(rq->knodev)) { + struct knod_work_priv *wpriv =3D &rq->knodev->wpriv[rq->ix]; + struct napi_struct *napi; + + work_done +=3D mlx5e_rx_offload_act_handler(rq, true, + budget - work_done); + + /* KNOD direct XDP_TX keeps the RX netmem owned by the TX SQ + * until the NIC reports TX completion. The normal NAPI order + * polls the XDP SQ before RX CQ processing, then the KNOD + * release pass can enqueue and doorbell a large burst of + * MPWQEs. Poll once more here so completions that arrived + * during RX/release processing are visible before the final + * RX repost below. + */ + if (rq->xdpsq) + busy |=3D mlx5e_poll_xdpsq_cq(&rq->xdpsq->cq); + + napi =3D READ_ONCE(wpriv->napi); + if (napi) + knod_dev_xdp_drain_pass(rq->knodev, napi, rq->ix, + budget); + } + busy |=3D work_done =3D=3D budget; =20 mlx5e_poll_ico_cq(&c->icosq.cq); --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E5223B27D3 for ; Sun, 19 Jul 2026 18:01:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484135; cv=none; b=iLqCwnpuEDmHcx1XRfWvz7VFtyi6+nGIXEHJRMUbrJpY8ikDRvBrY9BffYVL4rpo8mWn3b6s1i7mkhpWJ7hgCCESHSBIubjkYSuTtJcRQQPF0qsBAY9OyporfMow9KSV2JMC7BZlqW+s8bIvYIOLScRr/Y/0qBdKS+zUsYQ1Xw0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484135; c=relaxed/simple; bh=9x+KxMip3DrZ2abMwha3hDKHh9v0LyqG+xlGpmM7+7s=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BXIasY0FXgzO7eR7gblFLrmG8mjrfITM5jyOgR2ckYBkawBdQu43OyFNDraVkn7Ey4DzOKtaY6hpaeCa7ENx/wIVRK6SP4rXn5E0doE9lCwSfeJH9k14ZAXygBMoniiBkIYztICmsi068oAJrvU+uReFK0tPOQR1UJ9pBnLYrdk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rY+RQRwm; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rY+RQRwm" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2cad8076b01so113856835ad.2 for ; Sun, 19 Jul 2026 11:01:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484113; x=1785088913; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=/kzD1p6yL30PU6FFChKoKisLHf+QG0tGQ4P1mDlrkEk=; b=rY+RQRwm+Deo9laNwaLOgUpSG4troWaCbZFkDTZZRRgI1MJXP3Xik9CT27C3JFkIJA IeGneALsthSD8DXfFfdV6IxEZ9VPzunuAp/Y/hsu+yDjTTjqIJqQ1Xq+yhm6HuQ5GcOb A0ZihB0eIKqp8lvmhceCOA4Chmv82br6+uCYnLgTWPiGhdbKCKBjjWLzyhtpY/Zf6reB BdEcJjYwDKYCAYa25NOwznSf9r4dzyT6U3MsrsDqoTKiU+lv2Qd7nQXmPqdJ0U3FMuxJ PMIUeZVlcBkQjzw9gKGKlQwzM4Secj+kSKZtOUAkNcPOmTal7RMFMSIZ49/iLcg9bt9f EN0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484113; x=1785088913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=/kzD1p6yL30PU6FFChKoKisLHf+QG0tGQ4P1mDlrkEk=; b=GEK3AdNc76n+uSwQz4rJ3xZ2Qt68O1DkOUuLflWddbQXROGMbe5ANrKnKk6SMD2KsO LokdN183Djtg2IXl5VKJdKOF7bkIWGQZkdIaxiv511kicmluYOcqTfX9ddYvT0s9Pa7K ktcPNjo3V4mKMVoKmn5dk1Qk2DGDpOHkUnI4pANwl5NyXpEmVAasBYHzp7P/R7P2hW0Z azCj9HZ1s4PUGoVDJmNw625zKpYBJjyuGv25X1CxjcTFvGk8voKQ+TbBDtof3JRm7X+Y Xs4kKgUAvEXwoTFbF7xZup6UoSbo24ktO476oajgm1/LohNy4N2DLtzwG9tvG9nTsMcu vTwg== X-Forwarded-Encrypted: i=1; AHgh+RpFso+gpFwzzNx6T2zksqqqpL9Im7d/YLd/9FlxBAr2ELb4v3Oi54DVrzs9cnlplMYwHUoCTxafM+XjIGw=@vger.kernel.org X-Gm-Message-State: AOJu0YyYSCS/DDlJ6CaCIKZtW5PzAr+EYBb5ZxdDF8P8zi3ZEuHZhe9K BGl25pa0uVEWyMRbvXGT+0B29KM1/i5hMbmR79WigCQbtGmSfP+5A6Mz X-Gm-Gg: AfdE7clTnCDtVfwX2DMmOFaK8qDhy93zyf8eqQsRpHKMhSyoLd+/McLZd3LOOSqKtjG T4xoYQbNsDt78U34jq79GEbJvb1m8hUXQRReo7P3o9e2C6vY3BSlsGgpjp6Yegs64Xenqc+/svt JZR8ak1Wtn19DI85XJ0fb5ARA+uWJHDkae7WDifDjEVcrleUZoU3bSNU0+mgXcccAAze8jyM4fU FHBUf0qpOz7kEtqOkK7E6U73bYSF7k6KykSgAbMRqor6ungcQM3ueYWkOzwSp353F8RsxJdd8VP ZGj77+KKH9m3QnXU8Ise3Ss0p6BsoLYTmPdraVfXY619ELy3WdmocpsdEvDPfaGvDbCMgocOu1y vHe3Fe+z6eKbzXq5f3J+jM6u272aUU4m8n0OU65BqORlqS8J4WdY9B9ZoKqrOwnZk8BNYnM1e7C Dl X-Received: by 2002:a17:903:46c3:b0:2cf:4c0f:5129 with SMTP id d9443c01a7336-2cf4c0f5449mr64949115ad.17.1784484113118; Sun, 19 Jul 2026 11:01:53 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:01:52 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 11/13] bnxt_en: add knod XDP offload support Date: Sun, 19 Jul 2026 17:58:55 +0000 Message-ID: <20260719175857.4071636-12-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let bnxt_en act as a knod NIC: register with the knod core, feed received packets to the accelerator over the per-queue SPSC ring, and transmit accelerator verdicts back through the XDP TX ring. Restricted to page-mode MTU. Signed-off-by: Taehee Yoo (cherry picked from commit c9ef892dd5859cc71b0f8a546d87032cf70bb442) --- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 155 +++++++++- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 12 +- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 274 +++++++++++++++++- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h | 12 +- 4 files changed, 442 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethern= et/broadcom/bnxt/bnxt.c index 7513618793da..6ddf4bafc3d4 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -993,11 +993,36 @@ static inline u8 *__bnxt_alloc_rx_frag(struct bnxt *b= p, dma_addr_t *mapping, return page_address(page) + offset; } =20 +static int bnxt_alloc_rx_off_netmem(struct bnxt *bp, + struct bnxt_rx_ring_info *rxr, + u16 prod, gfp_t gfp) +{ + struct rx_bd *rxbd =3D + &rxr->rx_desc_ring[RX_RING(bp, prod)][RX_IDX(prod)]; + struct bnxt_sw_rx_bd *rx_buf =3D &rxr->rx_buf_ring[RING_RX(bp, prod)]; + unsigned int offset =3D 0; + dma_addr_t mapping; + netmem_ref netmem; + + netmem =3D __bnxt_alloc_rx_netmem(bp, &mapping, rxr, &offset, gfp); + if (!netmem) + return -ENOMEM; + + mapping +=3D bp->rx_dma_offset; + rx_buf->data =3D (void *)netmem; + rx_buf->data_ptr =3D netmem_address(netmem) + bp->rx_offset; + rx_buf->mapping =3D mapping; + + rxbd->rx_bd_haddr =3D cpu_to_le64(mapping); + return 0; +} + int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 prod, gfp_t gfp) { struct rx_bd *rxbd =3D &rxr->rx_desc_ring[RX_RING(bp, prod)][RX_IDX(prod)= ]; struct bnxt_sw_rx_bd *rx_buf =3D &rxr->rx_buf_ring[RING_RX(bp, prod)]; + unsigned int offset =3D 0; dma_addr_t mapping; =20 if (BNXT_RX_PAGE_MODE(bp)) { @@ -1012,6 +1037,16 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_= rx_ring_info *rxr, rx_buf->data =3D page; rx_buf->data_ptr =3D page_address(page) + offset + bp->rx_offset; rx_buf->offset =3D offset; + } else if (BNXT_RX_OFFLOAD_MODE(bp)) { + netmem_ref netmem =3D __bnxt_alloc_rx_netmem(bp, &mapping, rxr, + &offset, gfp); + if (!netmem) + return -ENOMEM; + + mapping +=3D bp->rx_dma_offset; + rx_buf->data =3D (void *)netmem; + rx_buf->data_ptr =3D netmem_address(netmem) + bp->rx_offset; + rx_buf->offset =3D offset; } else { u8 *data =3D __bnxt_alloc_rx_frag(bp, &mapping, rxr, gfp); =20 @@ -1250,6 +1285,25 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt = *bp, return skb; } =20 +static struct sk_buff *bnxt_rx_offload_netmem(struct bnxt *bp, + struct bnxt_rx_ring_info *rxr, + u16 cons, void *data, + u8 *data_ptr, + dma_addr_t dma_addr, + unsigned int offset_and_len) +{ + u16 prod =3D rxr->rx_prod; + int err; + + err =3D bnxt_alloc_rx_off_netmem(bp, rxr, prod, GFP_ATOMIC); + if (unlikely(err)) { + bnxt_reuse_rx_data(rxr, cons, data); + return NULL; + } + + return NULL; +} + static struct sk_buff *bnxt_rx_skb(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons, void *data, u8 *data_ptr, @@ -2249,6 +2303,36 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_= cp_ring_info *cpr, len =3D flags >> RX_CMP_LEN_SHIFT; dma_addr =3D rx_buf->mapping; =20 + if (BNXT_RX_OFFLOAD_MODE(bp)) { + if (bnxt_alloc_rx_off_netmem(bp, rxr, rxr->rx_prod, + GFP_ATOMIC)) { + bnxt_reuse_rx_data(rxr, cons, data); + bnapi->cp_ring.sw_stats->rx.rx_buf_errors++; + } else { + struct knod_work_priv *wpriv; + struct spsc_bd *bd; + + if (bnapi->index >=3D KNOD_SPSC_MAX) { + rc =3D 1; + goto next_rx; + } + + wpriv =3D &bp->knodev->wpriv[bnapi->index]; + if (spsc_produce(&wpriv->spsc_bds, (void **)&bd)) { + bnxt_reuse_rx_data(rxr, cons, data); + } else { + bd->netmem =3D (netmem_ref)data; + bd->len =3D len; + bd->off =3D bp->rx_offset; + bd->page_idx =3D net_iov_binding_idx( + netmem_to_net_iov((netmem_ref)data)); + spsc_produce_commit(&wpriv->spsc_bds); + } + } + rc =3D 1; + goto next_rx; + } + if (bnxt_xdp_attached(bp, rxr)) { bnxt_xdp.rxcmp =3D rxcmp; bnxt_xdp.rxcmp1 =3D rxcmp1; @@ -3272,6 +3356,11 @@ static int bnxt_poll(struct napi_struct *napi, int b= udget) napi_complete(napi); return 0; } + + if (BNXT_RX_OFFLOAD_MODE(bp)) + work_done +=3D bnxt_rx_offload_act_handler(bnapi, + budget - work_done); + while (1) { work_done +=3D bnxt_poll_work(bp, cpr, budget - work_done); =20 @@ -3296,6 +3385,7 @@ static int bnxt_poll(struct napi_struct *napi, int bu= dget) &dim_sample); net_dim(&cpr->dim, &dim_sample); } + return work_done; } =20 @@ -3363,6 +3453,11 @@ static int bnxt_poll_p5(struct napi_struct *napi, in= t budget) cpr->has_more_work =3D 0; work_done =3D __bnxt_poll_cqs(bp, bnapi, budget); } + + if (BNXT_RX_OFFLOAD_MODE(bp)) + work_done +=3D bnxt_rx_offload_act_handler(bnapi, + budget - work_done); + while (1) { u16 type; =20 @@ -3427,6 +3522,7 @@ static int bnxt_poll_p5(struct napi_struct *napi, int= budget) &dim_sample); net_dim(&cpr->dim, &dim_sample); } + return work_done; } =20 @@ -3458,6 +3554,13 @@ static void bnxt_free_one_tx_ring_skbs(struct bnxt *= bp, tx_buf->xdpf =3D NULL; i++; continue; + } else if (tx_buf->action =3D=3D BNXT_NETMEM_TX) { + page_pool_recycle_direct_netmem( + netmem_get_pp(tx_buf->netmem), + tx_buf->netmem); + tx_buf->action =3D 0; + tx_buf->netmem =3D 0; + continue; } =20 skb =3D tx_buf->skb; @@ -3551,6 +3654,10 @@ static void bnxt_free_one_rx_ring(struct bnxt *bp, s= truct bnxt_rx_ring_info *rxr rx_buf->data =3D NULL; if (BNXT_RX_PAGE_MODE(bp)) page_pool_recycle_direct(rxr->page_pool, data); + else if (BNXT_RX_OFFLOAD_MODE(bp)) + page_pool_put_full_netmem( + netmem_get_pp((netmem_ref)data), + (netmem_ref)data, false); else page_pool_free_va(rxr->head_pool, data, true); } @@ -3905,7 +4012,7 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp, struct page_pool *pool; =20 pp.pool_size =3D bnxt_rx_agg_ring_fill_level(bp, rxr) / agg_size_fac; - if (BNXT_RX_PAGE_MODE(bp)) + if (BNXT_RX_PAGE_MODE(bp) || BNXT_RX_OFFLOAD_MODE(bp)) pp.pool_size +=3D bp->rx_ring_size / rx_size_fac; =20 pp.order =3D get_order(rxr->rx_page_size); @@ -3923,8 +4030,10 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp, return PTR_ERR(pool); rxr->page_pool =3D pool; =20 - rxr->need_head_pool =3D page_pool_is_unreadable(pool); - rxr->need_head_pool |=3D !!pp.order; + if (!BNXT_RX_OFFLOAD_MODE(bp)) { + rxr->need_head_pool =3D page_pool_is_unreadable(pool); + rxr->need_head_pool |=3D !!pp.order; + } if (bnxt_separate_head_pool(rxr)) { pp.order =3D 0; pp.max_len =3D PAGE_SIZE; @@ -4673,6 +4782,9 @@ static int bnxt_init_rx_rings(struct bnxt *bp) if (BNXT_RX_PAGE_MODE(bp)) { bp->rx_offset =3D NET_IP_ALIGN + XDP_PACKET_HEADROOM; bp->rx_dma_offset =3D XDP_PACKET_HEADROOM; + } else if (BNXT_RX_OFFLOAD_MODE(bp)) { + bp->rx_offset =3D NET_IP_ALIGN + XDP_PACKET_HEADROOM; + bp->rx_dma_offset =3D XDP_PACKET_HEADROOM; } else { bp->rx_offset =3D BNXT_RX_OFFSET; bp->rx_dma_offset =3D BNXT_RX_DMA_OFFSET; @@ -4961,12 +5073,14 @@ void bnxt_set_ring_params(struct bnxt *bp) /* Changing allocation mode of RX rings. * TODO: Update when extending xdp_rxq_info to support allocation modes. */ -static void __bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode) +static void __bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode) { struct net_device *dev =3D bp->dev; =20 - if (page_mode) { - bp->flags &=3D ~(BNXT_FLAG_AGG_RINGS | BNXT_FLAG_NO_AGG_RINGS); + if (page_mode & BNXT_FLAG_RX_PAGE_MODE) { + bp->flags &=3D ~(BNXT_FLAG_AGG_RINGS | + BNXT_FLAG_NO_AGG_RINGS | + BNXT_FLAG_RX_OFFLOAD_MODE); bp->flags |=3D BNXT_FLAG_RX_PAGE_MODE; =20 if (bp->xdp_prog->aux->xdp_has_frags) @@ -4982,15 +5096,24 @@ static void __bnxt_set_rx_skb_mode(struct bnxt *bp,= bool page_mode) bp->rx_skb_func =3D bnxt_rx_page_skb; } bp->rx_dir =3D DMA_BIDIRECTIONAL; + } else if (page_mode & BNXT_FLAG_RX_OFFLOAD_MODE) { + bp->flags &=3D ~(BNXT_FLAG_AGG_RINGS | + BNXT_FLAG_RX_PAGE_MODE); + bp->flags |=3D (BNXT_FLAG_RX_OFFLOAD_MODE | + BNXT_FLAG_NO_AGG_RINGS); + dev->max_mtu =3D min_t(u16, bp->max_mtu, BNXT_MAX_PAGE_MODE_MTU); + bp->rx_skb_func =3D bnxt_rx_offload_netmem; + bp->rx_dir =3D DMA_BIDIRECTIONAL; } else { dev->max_mtu =3D bp->max_mtu; bp->flags &=3D ~BNXT_FLAG_RX_PAGE_MODE; + bp->flags &=3D ~BNXT_FLAG_RX_OFFLOAD_MODE; bp->rx_dir =3D DMA_FROM_DEVICE; bp->rx_skb_func =3D bnxt_rx_skb; } } =20 -void bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode) +void bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode) { __bnxt_set_rx_skb_mode(bp, page_mode); =20 @@ -5663,6 +5786,12 @@ static int bnxt_alloc_mem(struct bnxt *bp, bool irq_= re_init) BNXT_RING_TO_TC(bp, txr->txq_index); bnapi2->tx_ring[txr->tx_napi_idx] =3D txr; bnapi2->tx_int =3D bnxt_tx_int; + } else if (BNXT_RX_OFFLOAD_MODE(bp)) { + bnapi2 =3D bp->bnapi[j]; + bnapi2->flags |=3D BNXT_NAPI_FLAG_XDP; + bnapi2->tx_ring[0] =3D txr; + bnapi2->tx_int =3D bnxt_tx_int_xdp; + j++; } else { bnapi2 =3D bp->bnapi[j]; bnapi2->flags |=3D BNXT_NAPI_FLAG_XDP; @@ -11958,6 +12087,8 @@ static void bnxt_del_napi(struct bnxt *bp) if (!bp->bnapi) return; =20 + bnxt_rx_offload_clear_napi(bp); + for (i =3D 0; i < bp->rx_nr_rings; i++) netif_queue_set_napi(bp->dev, i, NETDEV_QUEUE_TYPE_RX, NULL); for (i =3D 0; i < bp->tx_nr_rings - bp->tx_nr_rings_xdp; i++) @@ -11997,6 +12128,8 @@ static void bnxt_init_napi(struct bnxt *bp) bnapi =3D bp->bnapi[cp_nr_rings]; netif_napi_add_locked(bp->dev, &bnapi->napi, bnxt_poll_nitroa0); } + + bnxt_rx_offload_set_napi(bp); } =20 static void bnxt_disable_napi(struct bnxt *bp) @@ -13356,6 +13489,7 @@ static int bnxt_open(struct net_device *dev) BNXT_RESTART_ULP_SP_EVENT); } } + bnxt_rx_offload_start(bp); =20 return rc; } @@ -13443,6 +13577,7 @@ static int bnxt_close(struct net_device *dev) { struct bnxt *bp =3D netdev_priv(dev); =20 + bnxt_rx_offload_stop(bp); bnxt_close_nic(bp, true, true); bnxt_hwrm_shutdown_link(bp); bnxt_hwrm_if_change(bp, false); @@ -13643,6 +13778,8 @@ bnxt_get_stats64(struct net_device *dev, struct rtn= l_link_stats64 *stats) stats->tx_errors =3D BNXT_GET_TX_PORT_STATS64(tx, tx_err); } clear_bit(BNXT_STATE_READ_STATS, &bp->state); + if (BNXT_RX_OFFLOAD_MODE(bp)) + knod_dev_get_stats64(bp->knodev, stats); } =20 static void bnxt_get_one_ring_drv_stats(struct bnxt *bp, @@ -16518,6 +16655,7 @@ static void bnxt_remove_one(struct pci_dev *pdev) =20 bnxt_aux_devices_del(bp); =20 + bnxt_knod_uninit(bp); unregister_netdev(dev); bnxt_ptp_clear(bp); =20 @@ -17080,7 +17218,7 @@ static int bnxt_init_one(struct pci_dev *pdev, cons= t struct pci_device_id *ent) } =20 dev->xdp_features =3D NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | - NETDEV_XDP_ACT_RX_SG; + NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_HW_OFFLOAD; =20 #ifdef CONFIG_BNXT_SRIOV init_waitqueue_head(&bp->sriov_cfg_wait); @@ -17198,6 +17336,7 @@ static int bnxt_init_one(struct pci_dev *pdev, cons= t struct pci_device_id *ent) bnxt_print_device_info(bp); =20 pci_save_state(pdev); + bnxt_knod_init(bp); =20 return 0; init_err_cleanup: diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethern= et/broadcom/bnxt/bnxt.h index 6335dfc14c98..d99590db6efa 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -29,6 +29,8 @@ #include #include #include +#include +#include #include #include #include @@ -884,10 +886,14 @@ struct nqe_cn { #define BNXT_REDIRECT_EVENT 8 #define BNXT_TX_CMP_EVENT 0x10 =20 +#define BNXT_NETMEM_ACT 0xf0 +#define BNXT_NETMEM_TX (XDP_TX + BNXT_NETMEM_ACT) + struct bnxt_sw_tx_bd { union { struct sk_buff *skb; struct xdp_frame *xdpf; + netmem_ref netmem; }; DEFINE_DMA_UNMAP_ADDR(mapping); DEFINE_DMA_UNMAP_LEN(len); @@ -2300,6 +2306,7 @@ struct bnxt { #define BNXT_FLAG_TX_COAL_CMPL 0x8000000 #define BNXT_FLAG_PORT_STATS_EXT 0x10000000 #define BNXT_FLAG_HDS 0x20000000 + #define BNXT_FLAG_RX_OFFLOAD_MODE 0x40000000 #define BNXT_FLAG_AGG_RINGS (BNXT_FLAG_JUMBO | BNXT_FLAG_GRO | \ BNXT_FLAG_LRO | BNXT_FLAG_HDS) =20 @@ -2324,6 +2331,7 @@ struct bnxt { (bp)->link_info.phy_state =3D=3D BNXT_PHY_STATE_ENABLED) #define BNXT_CHIP_TYPE_NITRO_A0(bp) ((bp)->flags & BNXT_FLAG_CHIP_NITRO_A0) #define BNXT_RX_PAGE_MODE(bp) ((bp)->flags & BNXT_FLAG_RX_PAGE_MODE) +#define BNXT_RX_OFFLOAD_MODE(bp) ((bp)->flags & BNXT_FLAG_RX_OFFLOAD_MODE) #define BNXT_SUPPORTS_TPA(bp) (!BNXT_CHIP_TYPE_NITRO_A0(bp) && \ (!((bp)->flags & BNXT_FLAG_CHIP_P5_PLUS) ||\ (bp)->max_tpa_v2) && !is_kdump_kernel()) @@ -2748,6 +2756,8 @@ struct bnxt { #define BNXT_DUMP_LIVE_WITH_CTX_L1_CACHE 3 =20 struct bpf_prog *xdp_prog; + struct knod_netdev *knetdev; + struct knod_dev *knodev; =20 struct bnxt_ptp_cfg *ptp_cfg; u8 ptp_all_rx_tstamp; @@ -2964,7 +2974,7 @@ u32 bnxt_fw_health_readl(struct bnxt *bp, int reg_idx= ); bool bnxt_bs_trace_avail(struct bnxt *bp, u16 type); void bnxt_set_tpa_flags(struct bnxt *bp); void bnxt_set_ring_params(struct bnxt *); -void bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode); +void bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode); void bnxt_insert_usr_fltr(struct bnxt *bp, struct bnxt_filter_base *fltr); void bnxt_del_one_usr_fltr(struct bnxt *bp, struct bnxt_filter_base *fltr); int bnxt_hwrm_func_drv_rgtr(struct bnxt *bp, unsigned long *bmap, diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/et= hernet/broadcom/bnxt/bnxt_xdp.c index 9e5009be8e98..a264a2cf8302 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c @@ -156,6 +156,11 @@ void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi= *bnapi, int budget) tx_buf =3D &txr->tx_buf_ring[RING_TX(bp, tx_cons)]; page_pool_recycle_direct(rxr->page_pool, tx_buf->page); } + } else if (tx_buf->action =3D=3D BNXT_NETMEM_TX) { + page_pool_recycle_direct_netmem(rxr->page_pool, + tx_buf->netmem); + tx_buf->action =3D 0; + tx_buf->netmem =3D 0; } else { bnxt_sched_reset_txr(bp, txr, tx_cons); return; @@ -423,7 +428,7 @@ static int bnxt_xdp_set(struct bnxt *bp, struct bpf_pro= g *prog) bpf_prog_put(old); =20 if (prog) { - bnxt_set_rx_skb_mode(bp, true); + bnxt_set_rx_skb_mode(bp, BNXT_FLAG_RX_PAGE_MODE); xdp_features_set_redirect_target_locked(dev, true); } else { xdp_features_clear_redirect_target_locked(dev); @@ -441,19 +446,123 @@ static int bnxt_xdp_set(struct bnxt *bp, struct bpf_= prog *prog) return 0; } =20 + +static int bnxt_xdp_offload_set(struct bnxt *bp, int enable) +{ + struct net_device *dev =3D bp->dev; + int tx_xdp =3D 0, tx_cp, rc, tc; + + netdev_assert_locked(dev); + + if (!(bp->flags & BNXT_FLAG_SHARED_RINGS)) { + netdev_warn(dev, "ethtool rx/tx channels must be combined to support XDP= .\n"); + return -EOPNOTSUPP; + } + if (enable && dev->mtu > BNXT_MAX_PAGE_MODE_MTU) { + netdev_warn(dev, "MTU %d larger than %d for single-page RX offload.\n", + dev->mtu, BNXT_MAX_PAGE_MODE_MTU); + return -EOPNOTSUPP; + } + if (enable) + tx_xdp =3D bp->rx_nr_rings; + + tc =3D bp->num_tc; + if (!tc) + tc =3D 1; + rc =3D bnxt_check_rings(bp, bp->tx_nr_rings_per_tc, bp->rx_nr_rings, + true, tc, tx_xdp); + if (rc) { + netdev_warn(dev, "Unable to reserve enough TX rings to support XDP.\n"); + return rc; + } + if (netif_running(dev)) + bnxt_close_nic(bp, true, false); + + if (enable) { + bnxt_set_rx_skb_mode(bp, BNXT_FLAG_RX_OFFLOAD_MODE); + xdp_features_set_redirect_target_locked(dev, true); + } else { + xdp_features_clear_redirect_target_locked(dev); + bnxt_set_rx_skb_mode(bp, false); + } + bp->tx_nr_rings_xdp =3D tx_xdp; + bp->tx_nr_rings =3D bp->tx_nr_rings_per_tc * tc + tx_xdp; + tx_cp =3D bnxt_num_tx_to_cp(bp, bp->tx_nr_rings); + bp->cp_nr_rings =3D max_t(int, tx_cp, bp->rx_nr_rings); + bnxt_set_tpa_flags(bp); + bnxt_set_ring_params(bp); + + if (netif_running(dev)) + return bnxt_open_nic(bp, true, false); + + return 0; +} + +void bnxt_rx_offload_set_napi(struct bnxt *bp) +{ + struct knod_dev *knodev =3D bp->knodev; + int i; + + if (!BNXT_RX_OFFLOAD_MODE(bp)) + return; + + for (i =3D 0; i < KNOD_SPSC_MAX && i < bp->cp_nr_rings; i++) + WRITE_ONCE(knodev->wpriv[i].napi, &bp->bnapi[i]->napi); +} + +void bnxt_rx_offload_clear_napi(struct bnxt *bp) +{ + struct knod_dev *knodev =3D bp->knodev; + int i; + + if (!BNXT_RX_OFFLOAD_MODE(bp)) + return; + + for (i =3D 0; i < KNOD_SPSC_MAX; i++) + WRITE_ONCE(knodev->wpriv[i].napi, NULL); +} + +void bnxt_rx_offload_start(struct bnxt *bp) +{ + if (!BNXT_RX_OFFLOAD_MODE(bp)) + return; + + knod_dev_start(bp->knodev); +} + +void bnxt_rx_offload_stop(struct bnxt *bp) +{ + if (!BNXT_RX_OFFLOAD_MODE(bp)) + return; + + knod_dev_stop(bp->knodev); +} + int bnxt_xdp(struct net_device *dev, struct netdev_bpf *xdp) { struct bnxt *bp =3D netdev_priv(dev); + struct knod_dev *knodev; int rc; =20 + knodev =3D bp->knodev; + switch (xdp->command) { case XDP_SETUP_PROG: rc =3D bnxt_xdp_set(bp, xdp->prog); break; + case XDP_SETUP_PROG_HW: + case BPF_OFFLOAD_MAP_ALLOC: + case BPF_OFFLOAD_MAP_FREE: + if (!knodev) + return -EOPNOTSUPP; + + rc =3D knod_dev_xdp_install(knodev, xdp); + break; default: rc =3D -EINVAL; break; } + return rc; } =20 @@ -528,3 +637,166 @@ int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *h= ash, *rss_type =3D hash_type; return 0; } + +int bnxt_rx_offload_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, + u16 cons, void *data, unsigned int len, int index, + u8 *event) +{ + struct knod_dev *knodev =3D READ_ONCE(bp->knodev); + + return knodev->accel_ops->xdp_ops->rx_netmem(bp->knodev, (netmem_ref)data= , len, + bp->rx_offset, index); +} + +static int bnxt_rx_offload_xdp_attach(struct knod_dev *knodev) +{ + struct bnxt *bp =3D netdev_priv(knodev->netdev); + int rc; + + WRITE_ONCE(bp->knodev, knodev); + rc =3D bnxt_xdp_offload_set(bp, true); + if (rc) + WRITE_ONCE(bp->knodev, NULL); + + return rc; +} + +static int bnxt_rx_offload_xdp_detach(struct knod_dev *knodev) +{ + struct bnxt *bp =3D netdev_priv(knodev->netdev); + + WRITE_ONCE(bp->knodev, NULL); + bnxt_xdp_offload_set(bp, false); + + return 0; +} + +static void __bnxt_xmit_netmem(struct bnxt *bp, struct bnxt_tx_ring_info *= txr, + dma_addr_t mapping, u32 len, + netmem_ref netmem) +{ + struct bnxt_sw_tx_bd *tx_buf; + + tx_buf =3D bnxt_xmit_bd(bp, txr, mapping, len, NULL); + tx_buf->action =3D BNXT_NETMEM_TX; + tx_buf->netmem =3D netmem; + netmem_dma_unmap_addr_set(netmem, tx_buf, mapping, mapping); + dma_unmap_len_set(tx_buf, len, 0); +} + +struct knod_nic_ops nic_ops =3D { + .attach =3D bnxt_rx_offload_xdp_attach, + .detach =3D bnxt_rx_offload_xdp_detach, +}; + +int bnxt_rx_offload_act_handler(struct bnxt_napi *bnapi, int budget) +{ + struct bnxt_tx_ring_info *txr =3D bnapi->tx_ring[0]; + struct bnxt_rx_ring_info *rxr =3D bnapi->rx_ring; + struct spsc_bd *bds[NAPI_POLL_WEIGHT]; + u32 tx_avail, cnt, i, nxmit =3D 0; + struct knod_dev *knodev; + struct knod_work_priv *wpriv; + struct napi_struct *napi; + struct bnxt *bp =3D bnapi->bp; + dma_addr_t mapping; + + knodev =3D bp->knodev; + if (!knodev) + return 0; + + if (bnapi->index >=3D bp->dev->real_num_tx_queues) { + this_cpu_inc(knodev->stats->tx_dropped); + return 0; + } + + wpriv =3D &knodev->wpriv[bnapi->index]; + napi =3D READ_ONCE(wpriv->napi); + if (!napi) + return 0; + + tx_avail =3D bnxt_tx_avail(bp, txr); + cnt =3D min_t(u32, NAPI_POLL_WEIGHT, tx_avail); + cnt =3D min_t(u32, cnt, budget); + if (!cnt) + return 0; + + spsc_release(&wpriv->spsc_bds, (void **)bds, cnt, &cnt); + if (!cnt) + return 0; + + for (i =3D 0; i < cnt; i++) { + switch (bds[i]->act) { + case KNOD_ACT_INFLIGHT: + case KNOD_IPSEC_INFLIGHT: + goto stop_release; + case XDP_TX: + mapping =3D netmem_to_net_iov(bds[i]->netmem)->desc.dma_addr + + bds[i]->off; + __bnxt_xmit_netmem(bp, txr, mapping, bds[i]->len, + bds[i]->netmem); + nxmit++; + break; + case XDP_ABORTED: + fallthrough; + case XDP_DROP: + fallthrough; + case XDP_PASS: + fallthrough; + case XDP_REDIRECT: + fallthrough; + default: + page_pool_recycle_direct_netmem(rxr->page_pool, + bds[i]->netmem); + this_cpu_inc(knodev->stats->tx_dropped); + break; + } + } +stop_release: + if (nxmit) { + wmb(); + bnxt_db_write(bp, &txr->tx_db, txr->tx_prod); + } + spsc_release_commit(&wpriv->spsc_bds, i); + + /* + * Device->host delivery: drain the framework pending ring for this NIC + * RX queue and push the built skbs up the stack. Covers all features + * (bpf/none deliver directly, ipsec via knod_dev->post_copy); no-op + * when nothing is attached. + */ + knod_dev_xdp_drain_pass(knodev, napi, bnapi->index, budget); + + return i; +} + +int bnxt_knod_init(struct bnxt *bp) +{ + struct knod_netdev *knetdev; + + knetdev =3D kzalloc_obj(struct knod_netdev, GFP_KERNEL); + if (!knetdev) { + pr_debug("Failed to allocate knetdev\n"); + return -ENOMEM; + } + + INIT_LIST_HEAD(&knetdev->list); + knetdev->dev =3D bp->dev; + knetdev->priv =3D bp; + knetdev->nic_ops =3D &nic_ops; + knetdev->owner =3D THIS_MODULE; + knetdev->flags |=3D KNOD_FLAGS_XDP; + knod_netdev_register(knetdev); + bp->knetdev =3D knetdev; + + return 0; +} + +void bnxt_knod_uninit(struct bnxt *bp) +{ + knod_netdev_unregister(bp->knetdev); + kfree(bp->knetdev); + bp->knetdev =3D NULL; + kfree(bp->knodev); + WRITE_ONCE(bp->knodev, NULL); +} diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h b/drivers/net/et= hernet/broadcom/bnxt/bnxt_xdp.h index fb4f9143929f..7f2a850e4787 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h @@ -24,9 +24,14 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp, dma_addr_t mapping, u32 len, struct xdp_buff *xdp); void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi *bnapi, int budget); +void bnxt_tx_int_offload(struct bnxt *bp, struct bnxt_napi *bnapi, int bud= get); bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons, struct xdp_buff *xdp, struct page *page, u8 **data_ptr, unsigned int *len, u8 *event); +void bnxt_rx_offload_stop(struct bnxt *bp); +void bnxt_rx_offload_start(struct bnxt *bp); +void bnxt_rx_offload_set_napi(struct bnxt *bp); +void bnxt_rx_offload_clear_napi(struct bnxt *bp); int bnxt_xdp(struct net_device *dev, struct netdev_bpf *xdp); int bnxt_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames, u32 flags); @@ -43,5 +48,10 @@ struct sk_buff *bnxt_xdp_build_skb(struct bnxt *bp, stru= ct sk_buff *skb, struct xdp_buff *xdp); int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, enum xdp_rss_hash_type *rss_type); - +int bnxt_rx_offload_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, + u16 cons, void *data, unsigned int len, int index, + u8 *event); +int bnxt_rx_offload_act_handler(struct bnxt_napi *bnapi, int budget); +int bnxt_knod_init(struct bnxt *bp); +void bnxt_knod_uninit(struct bnxt *bp); #endif --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CC963B8934 for ; Sun, 19 Jul 2026 18:02:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484145; cv=none; b=Q7Io264s95DX/G+7Ha7M/NliQJTSU/ZOQp6KEL02xXysMTdpdf7e1t+u4BVAoTKjJVXn6D5hEvYEafTt2LDXEYh1P9KcY4NjqODaSKgrvT0TpeyCRxwp8revqpV22n3JUd/v2vZcKKyKem9Oaoo6GwgImBFAk4vbu2ONRD65N3I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484145; c=relaxed/simple; bh=Dfe8PX9EwXjJhm9Lnw+rhy+oUGJuYVGlKOauhy5RtYo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TLV4Q0mhFAUcLXHHnAiHn9aJwT5DeHwcxV3zoed0sTPGeF6d5y/FNLoIagSDpbUr26gr0jj7UDjlY0uzHpjmOIOba2zhU+08JFpwZanf00rO3jsgPBxOclwqOBM7KiobmZ3R13wf7xu26vftUYqICRamJ2NTzkr7X3MUsUPfrC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Iy9GTXAC; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Iy9GTXAC" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2cc7ef7ec27so103413265ad.1 for ; Sun, 19 Jul 2026 11:02:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484125; x=1785088925; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=FubUQWQXxGaLpagBZT9hiz5/sPKLgHImpVUa5N7oZFI=; b=Iy9GTXACKLjjCFoIEkSh3MC5gzJvk0HM28ylxyhW3lBJZw28iolngPyVsjUqkrL0IP lofpJ65qGrTdxCUfTIjKv6ci7nSCeTgkp3pj7JPjainrHwytrxPUog750jObhLSzIDqa c3KiyFugwkW1a2fZrTmVPs50jH93aX/jvLHA8S4sYMC5pDKEi2HYNUNXeyOrRd+KQruX Kz2GFWpx8nHuhxtX2psP0Zz6vn5rmpTNJS9Gi29IkfZuOnOOQaSrXkKZNpsDzKOZT7/R eP7BJQmxmnmoxFI7mVrB/7JmeyYyrg41kE+/PFQENDIKG+ksJVpNYbCzIiCGlUEzoVMN D/qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484125; x=1785088925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=FubUQWQXxGaLpagBZT9hiz5/sPKLgHImpVUa5N7oZFI=; b=r+p1Yj/qd0vj/8KnHj/qYgormoLSs6h/RXC0oRIJBr0K1Tj5sNIPvw+zSNghNWhclM 3HJag4GmfD7lswyDs8Obg3Y+NrXKZ660u2uR3193JYpYPGii9D0KJlGX/EOgStOMEwgY PXyZwSxFpl8jixFt+lriD5MXKEUBGpFFQJ+ZZndjNEz9jsK/kmTRW6Wb1FLH97ARmVpt qat5wJsCpVMQJh1izE/hFYLYU+rUjQTABcbPfJGDefSZUkofQ1Jy6Kj2ogmXO5BhCd87 VYNJZCDKYgx+yUlzqRWgjpsqMyq0NsmH1FtEZGrufREK0VdomrmvD3q/37RcN6RrezS2 BpIQ== X-Forwarded-Encrypted: i=1; AHgh+RqjL9yXlrU5ma1Lx5N1uyHvlq04shvE3XGlFB1Db+R0EG1Oj//MP9w0CdwDMaWyY5zAnVzviKR5Okwngw0=@vger.kernel.org X-Gm-Message-State: AOJu0YzjmgC60w9dmfBuTS21K3b5ydqajmOQetsB0hPG11heVTqUQrtY o6QVwHCq9QLP1JvkYBKDGq9vQUemta17ajVTb0OFhhWX4/0s6qxLUNpi X-Gm-Gg: AfdE7clSNQ+qnk+M6qLl+Rki40QyG2J9zJpH3XBTmotKDAWQqKVqU4NlhYDtPLxNdoF pNhkfJbxezpUOgoGVIR7+J+wSIvDoL04yKmRagaVPWWQEScnSXkpHm2rfhhUz9RP0PeMwBialSE 9xKOVlcq0wqucC76d2++BXVUdyiYxKZSQLlb29s8hP5ckS01e641idlHVgvm1PgEpMqmk29JFk4 SfUw5KCmHsiGxIy7YhvgQCIaMGyy8h3E8kR3g+3XDo7DxkkuwQSEZVvi5CoQBLPkDCHGh3qUFkP pDZNu/fKhwSoZRrQDIdHb1pMSwC9XpI9MRWcTsxwQ8ZmHC1vrKbIqiKDfXz9/dpmklNaf+nBF8C LgrKLZ4bPzkVkTZ81lvWEmlRf4txUNpZ8zaXrnkDOhMLnEZXVgRVqAkBwOy/4+b5WyA== X-Received: by 2002:a17:903:3510:b0:2cc:6817:d9bc with SMTP id d9443c01a7336-2cf349ea184mr118699745ad.38.1784484124796; Sun, 19 Jul 2026 11:02:04 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.01.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:02:03 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 12/13] selftests: drivers/net: add knod tests Date: Sun, 19 Jul 2026 17:58:56 +0000 Message-ID: <20260719175857.4071636-13-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add attach/detach and XDP offload selftests for knod, exercising the genetlink control plane and the BPF data path over a knod-capable NIC. Signed-off-by: Taehee Yoo (cherry picked from commit 320315bfd9a324ab029c2fe9eee52997592e5d0c) --- .../selftests/drivers/net/knod/Makefile | 17 ++ .../testing/selftests/drivers/net/knod/config | 7 + .../selftests/drivers/net/knod/knod_attach.sh | 135 +++++++++++++ .../drivers/net/knod/knod_xdp_ktime.sh | 173 +++++++++++++++++ .../drivers/net/knod/knod_xdp_loop.sh | 129 +++++++++++++ .../testing/selftests/drivers/net/knod/lib.sh | 181 ++++++++++++++++++ .../drivers/net/knod/xdp_ktime.bpf.c | 32 ++++ .../selftests/drivers/net/knod/xdp_loop.bpf.c | 42 ++++ 8 files changed, 716 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/knod/Makefile create mode 100644 tools/testing/selftests/drivers/net/knod/config create mode 100755 tools/testing/selftests/drivers/net/knod/knod_attach.sh create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_ktime= .sh create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_loop.= sh create mode 100755 tools/testing/selftests/drivers/net/knod/lib.sh create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c diff --git a/tools/testing/selftests/drivers/net/knod/Makefile b/tools/test= ing/selftests/drivers/net/knod/Makefile new file mode 100644 index 000000000000..08138529b55e --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/Makefile @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0 + +TEST_PROGS :=3D \ + knod_xdp_ktime.sh \ + knod_xdp_loop.sh \ + knod_attach.sh \ +# end of TEST_PROGS + +TEST_FILES :=3D \ + lib.sh \ +# end of TEST_FILES + +TEST_GEN_FILES +=3D $(patsubst %.c,%.o,$(wildcard *.bpf.c)) + +include ../../../lib.mk + +include ../../../net/bpf.mk diff --git a/tools/testing/selftests/drivers/net/knod/config b/tools/testin= g/selftests/drivers/net/knod/config new file mode 100644 index 000000000000..a60ece9da112 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/config @@ -0,0 +1,7 @@ +CONFIG_BPF=3Dy +CONFIG_BPF_SYSCALL=3Dy +CONFIG_XDP_SOCKETS=3Dy +CONFIG_KNOD=3Dm +CONFIG_HSA_AMD=3Dy +CONFIG_HSA_AMD_KNOD_BPF=3Dm +CONFIG_DEBUG_FS=3Dy diff --git a/tools/testing/selftests/drivers/net/knod/knod_attach.sh b/tool= s/testing/selftests/drivers/net/knod/knod_attach.sh new file mode 100755 index 000000000000..7ee6aca33ad8 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/knod_attach.sh @@ -0,0 +1,135 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# knod_attach.sh - exercise the knod attach/detach control plane. +# +# Checks the NIC<->accel binding lifecycle (attach makes the pair appear i= n the +# xdev list, detach removes it) and that malformed or impossible attach +# requests are rejected without taking the framework down. +# +# Requires: +# - KNOD (knod + amdgpu) modules loaded +# - AMD GPU with KNOD support +# - a NIC registered with knod +# - iproute2, root +# +# Environment: +# NIC=3D (required) NIC to test on +# ACCEL_ID=3D (optional) GPU accel ID, auto-detected if omitted +# +# Exit: 0=3Dpass, 1=3Dfail, 4=3Dskip + +set -o pipefail + +SELFDIR=3D$(dirname "$(readlink -f "$0")") +source "$SELFDIR/lib.sh" + +: "${NIC:=3D}" +: "${ACCEL_ID:=3D}" + +PASS=3D0 +FAIL=3D0 + +cleanup() { + if [ -n "$NIC" ]; then + knod_detach "$NIC" 2>/dev/null + ip link set dev "$NIC" down 2>/dev/null + fi +} +trap cleanup EXIT + +check_result() { + local desc=3D$1 + local ret=3D$2 + + if [ "$ret" -eq 0 ]; then + knod_pass "$desc" + PASS=3D$((PASS + 1)) + else + knod_fail "$desc" + FAIL=3D$((FAIL + 1)) + fi +} + +# reject =3D=3D the ynl attach request fails (exit nonzero) +expect_reject() { + local desc=3D$1 + local json=3D$2 + + if knod_ynl --do attach --json "$json" 2>/dev/null; then + knod_detach "$NIC" 2>/dev/null # undo an unexpected success + check_result "$desc" 1 + else + check_result "$desc" 0 + fi +} + +# -- prereq ------------------------------------------------------ +knod_check_prereq + +if [ -z "$NIC" ]; then + knod_skip "NIC env var not set" +fi + +if ! ip link show "$NIC" >/dev/null 2>&1; then + knod_skip "NIC $NIC does not exist" +fi + +accel_id=3D$(knod_find_accel) +if [ -z "$accel_id" ]; then + knod_skip "no KNOD accelerator found" +fi +[ -n "$ACCEL_ID" ] && accel_id=3D"$ACCEL_ID" + +echo "=3D=3D=3D KNOD attach/detach control-plane test =3D=3D=3D" +echo " NIC: $NIC" +echo " ACCEL_ID: $accel_id" +echo "" + +# attach requires the interface down; start from a known detached state +ip link set dev "$NIC" down 2>/dev/null +knod_detach "$NIC" 2>/dev/null + +# -- positive lifecycle ---------------------------------------- +knod_attach "$NIC" "$accel_id" +check_result "attach $NIC -> accel $accel_id" $? + +knod_xdev_has "$NIC" +check_result "xdev lists $NIC after attach" $? + +knod_detach "$NIC" +check_result "detach $NIC" $? + +if knod_xdev_has "$NIC"; then + check_result "xdev drops $NIC after detach" 1 +else + check_result "xdev drops $NIC after detach" 0 +fi + +# -- negative requests must be rejected ------------------------ +nic_ifindex=3D$(knod_ifindex "$NIC") +expect_reject "reject attach with no accel id" "{\"nic-ifindex\":$nic_= ifindex}" +expect_reject "reject attach to nonexistent accel" "{\"nic-ifindex\":$nic_= ifindex,\"accel-id\":999999}" +expect_reject "reject attach of nonexistent NIC" "{\"nic-ifindex\":99999= 9,\"accel-id\":$accel_id}" + +ip link set dev "$NIC" up 2>/dev/null +expect_reject "reject attach while NIC is up" "{\"nic-ifindex\":$nic_= ifindex,\"accel-id\":$accel_id}" +ip link set dev "$NIC" down 2>/dev/null + +# -- framework survived the bad requests ----------------------- +knod_kernel_alive +check_result "framework responsive after bad requests" $? + +# -- re-attach still works (state not corrupted) --------------- +knod_attach "$NIC" "$accel_id" +check_result "re-attach after errors" $? +knod_detach "$NIC" 2>/dev/null + +# -- summary -------------------------------------------------- +echo "" +echo "=3D=3D=3D Results: $PASS passed, $FAIL failed =3D=3D=3D" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi +exit 0 diff --git a/tools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh b/t= ools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh new file mode 100755 index 000000000000..8df003052cb7 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh @@ -0,0 +1,173 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# knod_xdp_ktime.sh - test bpf_ktime_get_ns() on GPU XDP offload +# +# Loads an XDP program that calls bpf_ktime_get_ns() and stores +# the result in an offloaded BPF_MAP_TYPE_ARRAY, then verifies +# the GPU-side timestamp is sane. +# +# Requires: +# - KNOD (knod + amdgpu) modules loaded +# - AMD GPU with KNOD support +# - NIC with xdpoffload support (mlx5, bnxt) +# - bpftool, iproute2 +# - root privileges +# - xdp_ktime.bpf.o (built by make) +# +# Environment: +# NIC=3D (required) NIC to test on +# ACCEL_ID=3D (optional) GPU accel ID, auto-detected if omitted +# REMOTE_IP=3D (optional) ping target to generate traffic +# +# Exit: 0=3Dpass, 1=3Dfail, 4=3Dskip + +set -o pipefail + +SELFDIR=3D$(dirname "$(readlink -f "$0")") +source "$SELFDIR/lib.sh" + +: "${NIC:=3D}" +: "${ACCEL_ID:=3D}" +: "${REMOTE_IP:=3D}" + +PASS=3D0 +FAIL=3D0 +BPF_OBJ=3D"$SELFDIR/xdp_ktime.bpf.o" + +cleanup() { + if [ -n "$NIC" ]; then + knod_cleanup "$NIC" + fi +} +trap cleanup EXIT + +check_result() { + local desc=3D$1 + local ret=3D$2 + + if [ "$ret" -eq 0 ]; then + knod_pass "$desc" + PASS=3D$((PASS + 1)) + else + knod_fail "$desc" + FAIL=3D$((FAIL + 1)) + fi +} + +# -- prereq ------------------------------------------------------ +knod_check_prereq + +if [ -z "$NIC" ]; then + knod_skip "NIC env var not set" +fi + +if ! ip link show "$NIC" >/dev/null 2>&1; then + knod_skip "NIC $NIC does not exist" +fi + +accel_id=3D$(knod_find_accel) +if [ -z "$accel_id" ]; then + knod_skip "no KNOD accelerator found" +fi +[ -n "$ACCEL_ID" ] && accel_id=3D"$ACCEL_ID" + +echo "=3D=3D=3D KNOD XDP ktime_get_ns test =3D=3D=3D" +echo " NIC: $NIC" +echo " ACCEL_ID: $accel_id" +echo "" + +# -- check BPF object ------------------------------------------ +if [ ! -f "$BPF_OBJ" ]; then + echo "FAIL: $BPF_OBJ not found (run make first)" + exit 1 +fi + +# -- attach NIC to GPU, select bpf feature --------------------- +ip link set dev "$NIC" down 2>/dev/null +knod_attach "$NIC" "$accel_id" +if [ $? -ne 0 ]; then + echo "FAIL: attach failed" + exit 1 +fi + +knod_feature_select "$accel_id" bpf +if [ $? -ne 0 ]; then + knod_skip "cannot select bpf feature" +fi + +# -- load XDP offload program ---------------------------------- +knod_xdp_load "$NIC" "$BPF_OBJ" +if [ $? -ne 0 ]; then + echo "FAIL: xdpoffload load failed" + exit 1 +fi + +# -- find prog/map IDs ----------------------------------------- +prog_id=3D$(bpftool prog show 2>/dev/null | \ + awk '/xdp_ktime_test/ {sub(/:/, "", $1); print $1; exit}') +if [ -z "$prog_id" ]; then + echo "FAIL: cannot find loaded BPF program" + exit 1 +fi +knod_log "prog_id=3D$prog_id" + +map_id=3D$(knod_get_map_id "$prog_id") +if [ -z "$map_id" ]; then + echo "FAIL: cannot find BPF map" + exit 1 +fi +knod_log "map_id=3D$map_id" + +# -- bring up interface and generate traffic ------------------- +ip link set dev "$NIC" up + +if [ -n "$REMOTE_IP" ]; then + knod_log "ping $REMOTE_IP to generate traffic" + ping -c 5 -W 1 "$REMOTE_IP" >/dev/null 2>&1 || true +else + knod_log "waiting for ambient traffic (10s)" + sleep 10 +fi + +# -- bring down interface before reading map ------------------ +ip link set dev "$NIC" down + +# -- read map and verify -------------------------------------- +ktime_val=3D$(knod_map_lookup_u64 "$map_id" 0) +pkt_count=3D$(knod_map_lookup_u64 "$map_id" 1) + +knod_log "ktime_ns=3D$ktime_val pkt_count=3D$pkt_count" + +# Test 1: packets were processed +rc=3D0 +[ "$pkt_count" -gt 0 ] || rc=3D1 +check_result "packets processed (count=3D$pkt_count)" $rc + +# Test 2: ktime is non-zero +rc=3D0 +[ "$ktime_val" -gt 0 ] || rc=3D1 +check_result "ktime non-zero ($ktime_val)" $rc + +# Test 3: ktime is within 30s of current time +if [ "$ktime_val" -gt 0 ]; then + now_ns=3D$(awk '{printf "%.0f", $1 * 1000000000}' /proc/uptime) + if [ -n "$now_ns" ]; then + diff=3D$(( now_ns - ktime_val )) + abs_diff=3D${diff#-} + rc=3D0 + [ "$abs_diff" -lt 30000000000 ] || rc=3D1 + check_result "ktime within 30s of wall clock (diff=3D${diff}ns)" $rc + else + knod_log "skipping wall clock check (/proc/uptime unavailable)" + fi +fi + +# -- summary -------------------------------------------------- +echo "" +echo "=3D=3D=3D Results: $PASS passed, $FAIL failed =3D=3D=3D" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi +exit 0 diff --git a/tools/testing/selftests/drivers/net/knod/knod_xdp_loop.sh b/to= ols/testing/selftests/drivers/net/knod/knod_xdp_loop.sh new file mode 100755 index 000000000000..d7cba1236968 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/knod_xdp_loop.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# knod_xdp_loop.sh - the KNOD JIT must reject a bounded-loop XDP program. +# +# Loop emission is not implemented yet, so a program with a real back-edge= has +# to be rejected with -EOPNOTSUPP at JIT time rather than miscompiled. Th= is +# checks that the load fails, that the back-edge is reported to dmesg, and= that +# nothing crashed or faulted in the reject path. +# +# Requires: +# - KNOD (knod + amdgpu) modules loaded +# - AMD GPU with KNOD support +# - NIC with xdpoffload support (mlx5, bnxt) +# - bpftool, iproute2, root +# - xdp_loop.bpf.o (built by make) +# +# Environment: +# NIC=3D (required) NIC to test on +# ACCEL_ID=3D (optional) GPU accel ID, auto-detected if omitted +# +# Exit: 0=3Dpass, 1=3Dfail, 4=3Dskip + +set -o pipefail + +SELFDIR=3D$(dirname "$(readlink -f "$0")") +source "$SELFDIR/lib.sh" + +: "${NIC:=3D}" +: "${ACCEL_ID:=3D}" + +PASS=3D0 +FAIL=3D0 +BPF_OBJ=3D"$SELFDIR/xdp_loop.bpf.o" + +cleanup() { + if [ -n "$NIC" ]; then + knod_cleanup "$NIC" + fi +} +trap cleanup EXIT + +check_result() { + local desc=3D$1 + local ret=3D$2 + + if [ "$ret" -eq 0 ]; then + knod_pass "$desc" + PASS=3D$((PASS + 1)) + else + knod_fail "$desc" + FAIL=3D$((FAIL + 1)) + fi +} + +# -- prereq ------------------------------------------------------ +knod_check_prereq + +if [ -z "$NIC" ]; then + knod_skip "NIC env var not set" +fi + +if ! ip link show "$NIC" >/dev/null 2>&1; then + knod_skip "NIC $NIC does not exist" +fi + +accel_id=3D$(knod_find_accel) +if [ -z "$accel_id" ]; then + knod_skip "no KNOD accelerator found" +fi +[ -n "$ACCEL_ID" ] && accel_id=3D"$ACCEL_ID" + +if [ ! -f "$BPF_OBJ" ]; then + echo "FAIL: $BPF_OBJ not found (run make first)" + exit 1 +fi + +echo "=3D=3D=3D KNOD XDP loop-rejection test =3D=3D=3D" +echo " NIC: $NIC" +echo " ACCEL_ID: $accel_id" +echo "" + +# -- setup: attach NIC to GPU, then select the bpf feature ----- +# feature_select needs the accel already attached (it swaps the live +# worker), so attach first. +ip link set dev "$NIC" down 2>/dev/null +knod_attach "$NIC" "$accel_id" +if [ $? -ne 0 ]; then + echo "FAIL: attach failed" + exit 1 +fi + +knod_feature_select "$accel_id" bpf +if [ $? -ne 0 ]; then + knod_skip "cannot select bpf feature" +fi + +# remember where dmesg is now so we only scan messages from this load +dmesg_mark=3D$(dmesg | wc -l) + +# -- load must fail -------------------------------------------- +knod_log "loading bounded-loop program (expecting rejection)" +if knod_xdp_load "$NIC" "$BPF_OBJ" 2>/dev/null; then + # unexpectedly accepted - unload and fail + knod_xdp_unload "$NIC" + check_result "loop program rejected at load" 1 +else + check_result "loop program rejected at load" 0 +fi + +new_dmesg=3D$(dmesg | tail -n +"$((dmesg_mark + 1))") + +# -- back-edge reported ---------------------------------------- +rc=3D1 +echo "$new_dmesg" | grep -q "knod_loop:.*back-edge" && rc=3D0 +check_result "loop back-edge reported in dmesg" $rc + +# -- framework still alive ------------------------------------- +knod_kernel_alive +check_result "system responsive after rejection" $? + +# -- summary -------------------------------------------------- +echo "" +echo "=3D=3D=3D Results: $PASS passed, $FAIL failed =3D=3D=3D" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi +exit 0 diff --git a/tools/testing/selftests/drivers/net/knod/lib.sh b/tools/testin= g/selftests/drivers/net/knod/lib.sh new file mode 100755 index 000000000000..d0d9b03f109d --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/lib.sh @@ -0,0 +1,181 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# lib.sh - KNOD XDP offload test utilities +# +# The KNOD control plane is the "knod" generic-netlink family; it is driven +# here through the in-tree ynl CLI (tools/net/ynl/pyynl/cli.py) so the tes= ts +# need no dedicated user-space tool. + +KSRC=3D$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../../.." && pwd) +readonly KNOD_YNL=3D"$KSRC/tools/net/ynl/pyynl/cli.py" +readonly KNOD_SPEC=3D"$KSRC/Documentation/netlink/specs/knod.yaml" + +KNOD_NIC=3D"" +KNOD_ACCEL_ID=3D"" +KNOD_CLEANUP_DONE=3D0 + +knod_log() { echo " [INFO] $*"; } +knod_pass() { echo " [PASS] $*"; } +knod_fail() { echo " [FAIL] $*"; } +knod_skip() { echo " [SKIP] $*"; exit 4; } + +# Invoke the knod generic-netlink family via the ynl CLI. +knod_ynl() { + python3 "$KNOD_YNL" --spec "$KNOD_SPEC" "$@" +} + +knod_ifindex() { + cat "/sys/class/net/$1/ifindex" 2>/dev/null +} + +knod_check_prereq() { + if [ "$(id -u)" -ne 0 ]; then + knod_skip "must be root" + fi + + if ! command -v python3 >/dev/null 2>&1; then + knod_skip "python3 not found (needed for the ynl CLI)" + fi + + if ! command -v jq >/dev/null 2>&1; then + knod_skip "jq not found" + fi + + if ! knod_ynl --dump accel-get >/dev/null 2>&1; then + knod_skip "knod genl family not available (module not loaded?)" + fi + + if ! command -v bpftool >/dev/null 2>&1; then + knod_skip "bpftool not found" + fi + + if ! command -v ip >/dev/null 2>&1; then + knod_skip "iproute2 (ip) not found" + fi +} + +# Auto-detect the id of the first amdgpu accelerator. +knod_find_accel() { + if [ -n "$KNOD_ACCEL_ID" ]; then + echo "$KNOD_ACCEL_ID" + return 0 + fi + + knod_ynl --dump accel-get --output-json 2>/dev/null | \ + jq -r 'map(select(.name | startswith("amdgpu"))) | .[0].id // empty' +} + +# Locate the knod debugfs directory (the DRI minor number varies). +knod_debug_dir() { + local d + + for d in /sys/kernel/debug/dri/*/knod; do + [ -d "$d" ] && { echo "$d"; return 0; } + done + return 1 +} + +# Activate a KNOD offload feature ("none", "bpf", "ipsec") on . +knod_feature_select() { + local accel_id=3D$1 + local feat=3D$2 + + knod_log "feature_select accel $accel_id -> $feat" + knod_ynl --do accel-set \ + --json "{\"id\":$accel_id,\"feature-ena\":\"$feat\"}" >/dev/null +} + +# Confirm the framework is still responsive (used after an expected failur= e to +# catch an oops/hang in the reject path). The accel inventory is persistent +# (independent of attach), so a successful dump means the family is alive. +knod_kernel_alive() { + knod_ynl --dump accel-get >/dev/null 2>&1 +} + +# Is currently bound to an accel (present in the dev list)? +knod_xdev_has() { + local nic=3D$1 + local ifindex + + ifindex=3D$(knod_ifindex "$nic") || return 1 + knod_ynl --dump dev-get --output-json 2>/dev/null | \ + jq -e --argjson i "$ifindex" \ + 'any(.[]; .["nic-ifindex"] =3D=3D $i)' >/dev/null +} + +knod_attach() { + local nic=3D$1 + local accel_id=3D$2 + local ifindex + + ifindex=3D$(knod_ifindex "$nic") || return 1 + knod_log "attach $nic (ifindex $ifindex) to accel $accel_id" + knod_ynl --do attach \ + --json "{\"nic-ifindex\":$ifindex,\"accel-id\":$accel_id}" >/dev/null +} + +knod_detach() { + local nic=3D$1 + local ifindex + + ifindex=3D$(knod_ifindex "$nic") || return 1 + knod_log "detach $nic" + knod_ynl --do detach \ + --json "{\"nic-ifindex\":$ifindex}" >/dev/null 2>&1 +} + +knod_xdp_load() { + local nic=3D$1 + local obj=3D$2 + + knod_log "xdpoffload load $obj on $nic" + ip link set dev "$nic" xdpoffload obj "$obj" sec xdp +} + +knod_xdp_unload() { + local nic=3D$1 + + knod_log "xdpoffload off on $nic" + ip link set dev "$nic" xdpoffload off 2>/dev/null +} + +knod_cleanup() { + local nic=3D$1 + + [ "$KNOD_CLEANUP_DONE" -eq 1 ] && return + KNOD_CLEANUP_DONE=3D1 + + knod_log "cleanup $nic" + knod_xdp_unload "$nic" + ip link set dev "$nic" down 2>/dev/null + knod_detach "$nic" +} + +knod_get_map_id() { + local prog_id=3D$1 + + bpftool prog show id "$prog_id" 2>/dev/null | \ + grep -o 'map_ids [0-9]*' | awk '{print $2}' +} + +knod_map_lookup_u64() { + local map_id=3D$1 + local key=3D$2 + local hex + + hex=3D$(bpftool map lookup id "$map_id" \ + key $key 0 0 0 2>/dev/null | \ + grep -o 'value:.*' | sed 's/value: //') + if [ -z "$hex" ]; then + echo 0 + return + fi + + printf '%d' "$(echo "$hex" | awk '{ + v =3D 0; + for (i =3D 8; i >=3D 1; i--) + v =3D v * 256 + strtonum("0x" $i); + printf "0x%x", v; + }')" +} diff --git a/tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c b/too= ls/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c new file mode 100644 index 000000000000..5025e41cf3e8 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 2); + __type(key, __u32); + __type(value, __u64); +} ktime_map SEC(".maps"); + +SEC("xdp") +int xdp_ktime_test(struct xdp_md *ctx) +{ + __u32 key_ts =3D 0; + __u32 key_cnt =3D 1; + __u64 ts =3D bpf_ktime_get_ns(); + __u64 *cnt; + __u64 new_cnt; + + bpf_map_update_elem(&ktime_map, &key_ts, &ts, BPF_ANY); + + cnt =3D bpf_map_lookup_elem(&ktime_map, &key_cnt); + if (cnt) { + new_cnt =3D *cnt + 1; + bpf_map_update_elem(&ktime_map, &key_cnt, &new_cnt, BPF_ANY); + } + + return XDP_PASS; +} + +char LICENSE[] SEC("license") =3D "GPL"; diff --git a/tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c b/tool= s/testing/selftests/drivers/net/knod/xdp_loop.bpf.c new file mode 100644 index 000000000000..87c8251cc185 --- /dev/null +++ b/tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +/* + * Minimal bounded-loop XDP program to exercise the KNOD JIT's loop detect= ion. + * + * The trip count is read from the packet (runtime) and unrolling is disab= led, + * so the compiler keeps a real loop with a back-edge instead of folding it + * into straight-line code. The body is an xorshift step - a non-affine + * recurrence the compiler cannot reduce to a closed form (a simple sum li= ke + * "sum +=3D i" gets turned into n*(n-1)/2 and the loop disappears). It t= ouches + * no memory inside the loop, so the verifier is happy, and it has the sim= plest + * shape: one back-edge, a single exit (the loop condition), no break and = no + * early return. + */ +SEC("xdp") +int xdp_loop_test(struct xdp_md *ctx) +{ + void *data =3D (void *)(long)ctx->data; + void *data_end =3D (void *)(long)ctx->data_end; + __u8 *pkt =3D data; + __u32 sum; + int i, n; + + if ((void *)(pkt + 1) > data_end) + return XDP_DROP; + + n =3D pkt[0] & 0x3f; /* runtime bound, 0..63 */ + sum =3D pkt[0] | 1; /* nonzero xorshift seed */ + +#pragma clang loop unroll(disable) + for (i =3D 0; i < n; i++) { + sum ^=3D sum << 13; + sum ^=3D sum >> 17; + sum ^=3D sum << 5; + } + + return (sum & 1) ? XDP_PASS : XDP_DROP; +} + +char LICENSE[] SEC("license") =3D "GPL"; --=20 2.43.0 From nobody Sat Jul 25 03:46:05 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 270303B995D for ; Sun, 19 Jul 2026 18:02:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484203; cv=none; b=YFXchBPy95odufXonQsdV1coSTorzXon+1PaNcLcIPVnClpa9ogYIND0hBMk+W51X8A69TfVr0P1MT3+dwmRbj8VBkSaP6yYt5s4/YI2MP8THSZDFQpS1rN/VVDipadntkYvWAW2MtExZufERSwdGEJVPFgSL+CXOE3DUSizfoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784484203; c=relaxed/simple; bh=kw8RXLQJ5L/pi5bv+kifeE7bwsxh0qwnPTg/JisPKls=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LGayIstyWiQQSEd80wIpU8V66KEQ4rJUuKGdChpZm87AEyER5CctB/qujHYryQugv6A3AB5f9EZ/ch1ZNQRdGIYi3V6zLiWBL+n30YyacNOVCB9NGRgyLKGhVINQneuwvAhVVrtS/arZOyaYqauBkzfBUA1O+CHvnw8q3WHsQc0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QtG4dukU; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QtG4dukU" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cf27856f9cso36568985ad.2 for ; Sun, 19 Jul 2026 11:02:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784484141; x=1785088941; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=zg6prSrF11vlTrFRmegzqdrSMPgY3uw3Tb6xFm9uGhI=; b=QtG4dukUIT/ZO5h1AX8N4Y+Eaiahn6jkNfyn36vu5gQAmt8eng4eVFrYSK8O+SbEPu uGCE83T4hLunv4Ufi1NJxoN4l+esfdYDF+bC8yefHhxone/5C3CUT4/604CUfaPsdSwx CwTDj2wrA0hKLbxYg3Jw9j/Go/t+hcc6eFRYUtRVoJwZVTuUFvyxnD2YOjZkKwFQFARe 4FKh4KXQOW642G08AHE/YJC/EGo0tHgzwxycoLU9ymjNwU+W6Lai+cfxDxM/OyRY51L9 6ibEp6aYrV50yS+Wv8FVLoowv61MQ/A7gVp/Hwg505v1vRFhJG4oAAreItCWJrpLmDv0 OzFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784484141; x=1785088941; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=zg6prSrF11vlTrFRmegzqdrSMPgY3uw3Tb6xFm9uGhI=; b=VAKyl6YJRJ/Nj7EvhiXl83yK+UZL8slcPZnCYPACcn718Cpbh7W58W4vhG2AWjK2x9 ePnYIekV6z8KEAsBK6xTDBlz2S+ubKPcQZMcgeVbM95t6xBz/fNr9h/l4jS3v06X2rXO u9pREY0nT4JANAXbUiCH1hYGxceplToPUySh36fs25DrqqjDerWyLbxHwJ9kvEl1yepA bf3A1l9DoZZ0IzsGfyRXbPYfkykBG1S9U3ccziAZby6QmBrx0QMMh1bY9ZeUzutDSUUZ 7TPDM31gsbA8isOxd4im+WA9g3WC7AJJ+pakDm+T+bjLcrjl/TqK5a2DVJooMsF3MFxQ kXeA== X-Forwarded-Encrypted: i=1; AHgh+Rq3PevPpnZwP2COaAbmNKlrjNbPk7qWRgcNLCqflLroijpoq2C+Z78F6l57HxJ1m94A41apaMxH6pT9ycQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy5SJ/PU8RbmNpzv1+4twCqODipwO4J/lwAJI+st4dHV4xkVXAO lYKp7pqIbhHTTUAG6GUUC860Hl1sZAtsLweatBHYcfUdyZEZRTEb2u/n X-Gm-Gg: AfdE7cnPq1fCkXAkruEeG+ZionqZ/bTV6ceyCASbsKiXrKtLG7cqVDvjTth5bPUUZ6W EKFjCYZHIfB47NxgoDN7fDMX/tt3sRyPBxv2+KTxaClhvamSmEFlH8lRt3IhylQh6WOX85TWbDE Q0Xl3K3booHp/5MXrkHjgY0+XXmZdaWtMg8uvHyCjVT7eqQLfjViLhVYrQFYJClZYsIshSEWD3w GS0Tran5wpXYHDWOGTdXLbvlYuZ5gNiEzpVoWXs3aIukC3HTs/hJW78dS23n5ywF5udS6eDfBii dgO6cuQXvZxaFOhVUjIu/9+bBv2xbxz5M5cw1elYjD18ELEOG6NcDI2rr2C64X8hFTTLFmKWuMI H7Ov4ixrLB6yBN2q5kQZfqLCjzKlYa+Go5ltVwZySVOzvtQpcH8f9TXzMuD82wDT8Zw== X-Received: by 2002:a17:903:2986:b0:2ce:b7b7:30e3 with SMTP id d9443c01a7336-2cf348575cbmr130100785ad.17.1784484138203; Sun, 19 Jul 2026 11:02:18 -0700 (PDT) Received: from ap.. ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf34769bd6sm43511065ad.74.2026.07.19.11.02.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jul 2026 11:02:16 -0700 (PDT) From: Taehee Yoo To: Alex Deucher , Alexei Starovoitov , amd-gfx@lists.freedesktop.org, Andrew Lunn , Andrii Nakryiko , Bill Wendling , bpf@vger.kernel.org, =?UTF-8?q?Christian=20K=C3=B6nig?= , Daniel Borkmann , David Airlie , "David S. Miller" , Donald Hunter , dri-devel@lists.freedesktop.org, Eduard Zingerman , Emil Tsalapatis , Eric Dumazet , Felix Kuehling , Hoyeon Lee , Ilias Apalodimas , Jakub Kicinski , Jesper Dangaard Brouer , Jiri Olsa , John Fastabend , Justin Stitt , Kees Cook , Kumar Kartikeya Dwivedi , Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Mark Bloch , Martin KaFai Lau , Michael Chan , Nathan Chancellor , netdev@vger.kernel.org, Nick Desaulniers , Paolo Abeni , Pavan Chebbi , Saeed Mahameed , Shuah Khan , Simona Vetter , Simon Horman , Song Liu , Stanislav Fomichev , Sumit Semwal , Taehee Yoo , Tariq Toukan , Yonghong Song Subject: [RFC PATCH net-next 13/13] drm/amdkfd: add IPsec full-packet offload Date: Sun, 19 Jul 2026 17:58:57 +0000 Message-ID: <20260719175857.4071636-14-ap420073@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260719175857.4071636-1-ap420073@gmail.com> References: <20260719175857.4071636-1-ap420073@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a second knod feature, alongside BPF, to show the accel ops are not tied to a single use case: GPU-offloaded IPsec ESP. xfrm SAs installed with XFRM_DEV_OFFLOAD_PACKET are handed to the GPU, which parses ESP, looks up the SA, checks anti-replay, does AES-GCM decrypt and writes the verdict back. Built as a separate module (knod_ipsec). This is a functional proof of concept: it demonstrates that a full-packet crypto feature can run on top of the knod accelerator, not a hardened, production-ready implementation. Signed-off-by: Taehee Yoo (cherry picked from commit 56ea4f90b43677e6944f1bc3ca4cf02e736ec340) --- drivers/gpu/drm/amd/amdkfd/Kconfig | 9 + drivers/gpu/drm/amd/amdkfd/Makefile | 3 + .../gpu/drm/amd/amdkfd/knod/aesgcm_shader.h | 984 ++++ .../drm/amd/amdkfd/knod/ipsec_fused_gfx10.h | 1796 +++++++ .../drm/amd/amdkfd/knod/ipsec_fused_gfx9.h | 1349 ++++++ drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c | 4273 +++++++++++++++++ drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h | 596 +++ .../testing/selftests/drivers/net/knod/config | 1 + 8 files changed, 9011 insertions(+) create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkf= d/Kconfig index 708aa5fc051b..67efae53e395 100644 --- a/drivers/gpu/drm/amd/amdkfd/Kconfig +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig @@ -60,3 +60,12 @@ config HSA_AMD_KNOD_BPF the verdict path (PASS/DROP/TX) off the host CPU. =20 If unsure, say N. + +config HSA_AMD_KNOD_IPSEC + tristate "KNOD IPsec (xfrm) full-packet offload" + depends on HSA_AMD_KNOD && XFRM_OFFLOAD && INET_ESP + help + GPU-accelerated IPsec ESP full-packet offload via KNOD. + Accepts xfrm SAs configured with XFRM_DEV_OFFLOAD_PACKET and + performs ESP parse, SA lookup, anti-replay, AES-GCM decrypt and + verdict writeback on the GPU. diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdk= fd/Makefile index 4df3850e1466..c5d801ddabb7 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -77,3 +77,6 @@ AMDKFD_FILES +=3D $(AMDKFD_PATH)/kfd_knod.o endif =20 obj-$(CONFIG_HSA_AMD_KNOD_BPF) +=3D $(AMDKFD_PATH)/knod/knod_bpf.o + + +obj-$(CONFIG_HSA_AMD_KNOD_IPSEC) +=3D $(AMDKFD_PATH)/knod/knod_ipsec.o diff --git a/drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h b/drivers/gpu/= drm/amd/amdkfd/knod/aesgcm_shader.h new file mode 100644 index 000000000000..ae5276edb9fd --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h @@ -0,0 +1,984 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +/* + * AES-GCM GPU shader emit helpers for GFX9 (Vega) / GFX10 (RDNA) + * + * Per-step emitters (AES rounds, encrypt block, GHASH gfmul, T-table + * lookup) shared by the IPsec fused shaders. + */ + +#ifndef AESGCM_SHADER_H_ +#define AESGCM_SHADER_H_ + +#include +#include +#include "knod_amdgpu_insn.h" + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Emit pattern macros (file-local) + * + * _E(fn, ...) - emit instruction, advance n by instruction size + * _BR(fn, ...) - emit branch, save position, advance n + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +#define _E(fn, ...) (n +=3D fn(__VA_ARGS__) / 4) +#define _BR(fn, ...) ({ int _p =3D n; n +=3D fn(__VA_ARGS__) / 4; _p; }) + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * AES T-table Round Helper + * + * Emits one AES round using T-table lookups from LDS. + * + * T-table layout in LDS (4KB total): + * T0: offset 0..1023 (256 x 4 bytes) + * T1: offset 1024..2047 + * T2: offset 2048..3071 + * T3: offset 3072..4095 + * + * AES round function: + * new[0] =3D T0[s0.b0] ^ T1[s1.b1] ^ T2[s2.b2] ^ T3[s3.b3] ^ rk[0] + * new[1] =3D T0[s1.b0] ^ T1[s2.b1] ^ T2[s3.b2] ^ T3[s0.b3] ^ rk[1] + * new[2] =3D T0[s2.b0] ^ T1[s3.b1] ^ T2[s0.b2] ^ T3[s1.b3] ^ rk[2] + * new[3] =3D T0[s3.b0] ^ T1[s0.b1] ^ T2[s1.b2] ^ T3[s2.b3] ^ rk[3] + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* + * Emit code to extract a byte from a VGPR and compute LDS T-table address. + * + * @buf: instruction buffer + * @n: current position + * @vdst_addr: output VGPR for LDS address + * @vdst_val: output VGPR for loaded T-table value (ds_read destination) + * @v_state: input VGPR (state word) + * @byte_pos: byte position (0, 1, 2, 3) + * @table_base: LDS base offset for this table (0, 1024, 2048, 3072) + * @s_mask: SGPR holding 0xFF + * @v_tmp: temp VGPR for byte extraction + */ +static int emit_ttable_lookup_gfx9(u32 *buf, int n, + int v_addr, int v_val, + int v_state, int byte_pos, + int table_base, int s_mask, int v_tmp) +{ + /* + * Extract byte: result =3D (state >> (byte_pos*8)) & 0xFF + * Then: LDS addr =3D result * 4 + table_base + */ + if (byte_pos =3D=3D 0) { + /* byte 0: just mask, no shift needed */ + /* v_and_b32: src0=3DSGPR(mask), vsrc1=3DVGPR(state) */ + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(v_tmp), P_S(s_mask), + P_V(v_state)); + } else if (byte_pos =3D=3D 3) { + /* byte 3: shift right 24, no mask needed */ + /* v_lshrrev: vdst =3D vsrc1 >> src0 =3D v_state >> 24 */ + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(v_tmp), P_I(24), + P_V(v_state)); + } else { + /* byte 1 or 2: shift then mask */ + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(v_tmp), + P_I(byte_pos * 8), P_V(v_state)); + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(v_tmp), P_S(s_mask), + P_V(v_tmp)); + } + + /* LDS addr =3D byte * 4 + table_base */ + /* v_lshlrev: vdst =3D vsrc1 << src0 =3D v_tmp << 2 */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(v_addr), P_I(2), + P_V(v_tmp)); + if (table_base > 0) + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(v_addr), + P_L(table_base), P_V(v_addr)); + + /* Issue LDS read (don't wait yet -- caller batches reads) */ + _E(emit_gfx9_ds_read_b32, I9(buf, n), v_val, v_addr, 0); + + return n; +} + +/* + * Emit one full AES round (rounds 1 to Nr-1). + * + * @buf, @n: instruction buffer and position + * @s0-s3: input state VGPRs + * @d0-d3: output state VGPRs (new state) + * @rk: SGPR base for round key (4 consecutive SGPRs) + * @s_mask: SGPR holding 0xFF + * @v_tmp: temp VGPR for byte extraction + * @v_addr: temp VGPR for LDS address + * @vt0-vt3: temp VGPRs for 4 T-table values per column + */ +static int emit_aes_round_gfx9(u32 *buf, int n, + int s0, int s1, int s2, int s3, + int d0, int d1, int d2, int d3, + int rk, int s_mask, int v_tmp, int v_addr, + int vt0, int vt1, int vt2, int vt3) +{ + /* + * Issue all 16 LDS reads (4 per column), then wait once. + * This maximizes LDS throughput by overlapping reads. + * + * Column 0: T0[s0.b0] ^ T1[s1.b1] ^ T2[s2.b2] ^ T3[s3.b3] + * Column 1: T0[s1.b0] ^ T1[s2.b1] ^ T2[s3.b2] ^ T3[s0.b3] + * Column 2: T0[s2.b0] ^ T1[s3.b1] ^ T2[s0.b2] ^ T3[s1.b3] + * Column 3: T0[s3.b0] ^ T1[s0.b1] ^ T2[s1.b2] ^ T3[s2.b3] + * + * We reuse d0-d3 and vt0-vt3 as temporaries for the 16 results. + * Process column by column to minimize register pressure. + */ + + /* ---- Column 0 ---- */ + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt0, s0, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt1, s1, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt2, s2, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt3, s3, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx9_s_waitcnt, I9(buf, n), 0xF, 0); /* lgkmcnt=3D0 */ + + /* XOR: d0 =3D vt0 ^ vt1 ^ vt2 ^ vt3 ^ rk */ + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d0), P_V(vt0), P_V(vt1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d0), P_V(d0), P_V(vt2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d0), P_V(d0), P_V(vt3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d0), P_S(rk), P_V(d0)); + + /* ---- Column 1 ---- */ + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt0, s1, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt1, s2, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt2, s3, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt3, s0, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx9_s_waitcnt, I9(buf, n), 0xF, 0); + + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d1), P_V(vt0), P_V(vt1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d1), P_V(d1), P_V(vt2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d1), P_V(d1), P_V(vt3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d1), P_S(rk + 1), P_V(d1)); + + /* ---- Column 2 ---- */ + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt0, s2, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt1, s3, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt2, s0, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt3, s1, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx9_s_waitcnt, I9(buf, n), 0xF, 0); + + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d2), P_V(vt0), P_V(vt1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d2), P_V(d2), P_V(vt2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d2), P_V(d2), P_V(vt3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d2), P_S(rk + 2), P_V(d2)); + + /* ---- Column 3 ---- */ + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt0, s3, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt1, s0, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt2, s1, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt3, s2, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx9_s_waitcnt, I9(buf, n), 0xF, 0); + + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d3), P_V(vt0), P_V(vt1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d3), P_V(d3), P_V(vt2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d3), P_V(d3), P_V(vt3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(d3), P_S(rk + 3), P_V(d3)); + + return n; +} + +/* + * Emit the last AES round (SubBytes + ShiftRows + AddRoundKey, no MixColu= mns). + * + * S-box is extracted from T0: S(x) =3D (T0[x] >> 8) & 0xFF + * (In standard AES T-table encoding: T0[x] =3D {2*S(x), S(x), S(x), 3*S(x= )}) + * + * Last round output: + * d[c] =3D S(s[c].b0) | (S(s[(c+1)%4].b1)<<8) | + * (S(s[(c+2)%4].b2)<<16) | (S(s[(c+3)%4].b3)<<24) ^ rk[c] + */ +static int emit_aes_last_round_gfx9(u32 *buf, int n, + int s0, int s1, int s2, int s3, + int d0, int d1, int d2, int d3, + int rk, int s_mask, int v_tmp, int v_addr, + int vt0, int vt1, int vt2, int vt3) +{ + /* + * For each column, look up T0 for all 4 bytes, + * extract S(x) =3D (T0[x] >> 8) & 0xFF, then assemble. + */ + int cols[4][4] =3D { + {s0, s1, s2, s3}, /* column 0 */ + {s1, s2, s3, s0}, /* column 1 */ + {s2, s3, s0, s1}, /* column 2 */ + {s3, s0, s1, s2}, /* column 3 */ + }; + int dsts[4] =3D {d0, d1, d2, d3}; + int col; + + for (col =3D 0; col < 4; col++) { + /* Look up T0 for all 4 bytes of this column */ + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt0, cols[col][0], + 0, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt1, cols[col][1], + 1, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt2, cols[col][2], + 2, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx9(buf, n, v_addr, vt3, cols[col][3], + 3, 0, s_mask, v_tmp); + _E(emit_gfx9_s_waitcnt, I9(buf, n), 0xF, 0); + + /* + * Extract S(x) from T0[x]: + * S(x) =3D (T0[x] >> 8) & 0xFF + */ + /* byte 0: S(x) in bits [15:8] of T0, place in bits [7:0] */ + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(vt0), P_I(8), + P_V(vt0)); + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(vt0), P_S(s_mask), + P_V(vt0)); + + /* byte 1: S(x) in bits [15:8], place in bits [15:8] */ + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(vt1), P_L(0xFF00), + P_V(vt1)); + + /* byte 2: S(x) in bits [15:8], place in bits [23:16] */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(vt2), P_I(8), + P_V(vt2)); + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(vt2), P_L(0xFF0000), + P_V(vt2)); + + /* byte 3: S(x) in bits [15:8], place in bits [31:24] */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(vt3), P_I(16), + P_V(vt3)); + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(vt3), + P_L(0xFF000000), P_V(vt3)); + + /* Assemble: d =3D b0 | b1 | b2 | b3 ^ rk */ + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(dsts[col]), P_V(vt0), + P_V(vt1)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(dsts[col]), + P_V(dsts[col]), P_V(vt2)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(dsts[col]), + P_V(dsts[col]), P_V(vt3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(dsts[col]), + P_S(rk + col), P_V(dsts[col])); + } + + return n; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX9 AES-GCM emit helpers + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* + * Register allocation: + * + * SGPRs: + * s[0:3] system: private_segment_buffer + * s[4:5] system: dispatch_ptr + * s[6:7] system: queue_ptr + * s[8:9] system: kernarg_segment_ptr + * s[10:11] system: dispatch_id + * s[12:13] system: flat_scratch_init + * s14 system: private_segment_size + * s15 system: workgroup_id_x + * s16 system: workgroup_id_y (batch lane) + * s17 system: workgroup_id_z + * + * s[18:19] subparam base address (computed) + * s[20:21] input buffer address (from subparam) + * s[22:23] output buffer address (from subparam) + * s[24:25] round keys address (from subparam) + * s26 nbytes (from subparam) + * s27 temp / block count + * s[28:31] current round key (loaded per-round) + * s32 0xFF constant + * s[34:35] T-table VRAM address pair (temp) + * s[36:37] IV word 0-1 (from subparam.iv) + * s38 IV word 2 (from subparam.iv, 4 bytes) + * s[48:49] saved EXEC + * + * VGPRs: + * v0 workitem_id_x (tid) + * v[1:4] AES state A + * v[5:8] AES state B / T-table results + * v9 byte extraction temp + * v10 LDS address temp + * v11 block_id (global) + * v[12:13] 64-bit global memory address + * v[14:17] data words (plaintext/ciphertext) + * v18 temp + */ + +#define SR_KEYS 24 /* s[24:25] */ +#define SR_NBLOCKS 27 +#define SR_RK 28 /* s[28:31] round key */ +#define SR_MASK 32 /* s32 =3D 0xFF */ +#define SR_LOOP_CTR 33 /* s33 =3D loop counter (Phase 4) */ +#define SR_T_ADDR 34 /* s[34:35] temp for T-table load */ +#define SR_IV0 36 /* s36 =3D IV word 0 */ +#define SR_IV1 37 /* s37 =3D IV word 1 */ +#define SR_IV2 38 /* s38 =3D IV word 2 */ +#define SR_NR_ROUNDS 39 /* s39 =3D nr_rounds from subparam */ +#define SR_EXEC_SAVE 48 /* s[48:49] */ + +#define VR_TID 0 /* workitem_id_x */ +#define VR_S0 1 /* state A: v[1:4] */ +#define VR_S1 2 +#define VR_S2 3 +#define VR_S3 4 +#define VR_D0 5 /* state B / T-table: v[5:8] */ +#define VR_D1 6 +#define VR_D2 7 +#define VR_D3 8 +#define VR_TMP 9 /* byte extraction temp */ +#define VR_ADDR 10 /* LDS address temp */ +#define VR_BLK 11 /* block_id */ +#define VR_GA_LO 12 /* 64-bit global addr lo */ +#define VR_GA_HI 13 /* 64-bit global addr hi */ +#define VR_DATA0 14 /* plaintext/ciphertext v[14:17] */ +#define VR_DATA1 15 +#define VR_DATA2 16 +#define VR_DATA3 17 +#define VR_TMP2 18 +/* Phase 5.5: saved AES(K, J0) result for ICV finalization */ +#define VR_J0_0 19 +#define VR_J0_1 20 +#define VR_J0_2 21 +#define VR_J0_3 22 + +/* Phase 7 GHASH: additional SGPR for exec save during sub-masking */ +#define SR_GHASH_EXEC 40 /* s[40:41] */ +#define SR_BSWAP 44 /* s44 =3D 0x00010203 (bswap32 selector for v_perm_b32= ) */ +#define SR_RK2 56 /* s[56:59] alternate round key for double-buffered AES= */ + +/* + * sizeof(knod_aesgcm_subparam) =3D 80 + * Offsets within subparam: + * 0: u64 in + * 8: u64 out + * 16: u64 keys + * 24: u64 h_table + * 32: u64 aad + * 40: u64 tag + * 48: u32 nbytes + * 52: u32 aad_len + * 56: u8 iv[12] (3 words at offsets 56, 60, 64 -- last only 4 bytes) + * 68: u32 op + * 72: u32 nr_rounds (10 or 14) + * 76: u32 _pad + * + * T-table addresses in knod_aesgcm_param (after sub[128]): + * 128 * 80 =3D 10240: u64 t0 + * 10248: u64 t1 + * 10256: u64 t2 + * 10264: u64 t3 + */ +#define SUBPARAM_SIZE 80 + +#define OFF_T0 (AESGCM_MAX_DIM_Y * SUBPARAM_SIZE) + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX9 AES Block Encrypt Helper + * + * Encrypts the 128-bit block in v[VR_S0:VR_S3] using T-tables in LDS. + * Expects SR_KEYS =3D round keys GPU addr (will be advanced). + * Expects SR_NR_ROUNDS =3D number of AES rounds. + * Result in v[VR_S0:VR_S3]. + * Clobbers: v[VR_D0:VR_D3], v[VR_DATA0:VR_DATA3], v[VR_TMP], v[VR_ADDR], + * s[SR_RK:SR_RK+3], s[SR_NBLOCKS], s[SR_LOOP_CTR], s[SR_KEYS+1] + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +static int emit_aes_encrypt_block_gfx9(u32 *buf, int n) +{ + int br_loop, loop_top; + + /* Round 0: XOR with first round key */ + _E(emit_gfx9_s_load_dwordx4, I9(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S0), P_S(SR_RK), + P_V(VR_S0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S1), P_S(SR_RK + 1), + P_V(VR_S1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S2), P_S(SR_RK + 2), + P_V(VR_S2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S3), P_S(SR_RK + 3), + P_V(VR_S3)); + + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx9_s_addc_u32, I9(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + + /* pair_count =3D nr_rounds / 2 - 1 */ + _E(emit_gfx9_s_lshr_b32, I9(buf, n), P_S(SR_NBLOCKS), P_S(SR_NR_ROUNDS), + P_I(1)); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_NBLOCKS), P_L(0xFFFFFFFFu), + P_S(SR_NBLOCKS)); + + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_LOOP_CTR), P_I(0)); + + /* Prefetch round 1 key - overlaps with loop-entry overhead */ + _E(emit_gfx9_s_load_dwordx4, I9(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + + loop_top =3D n; + + /* Odd round: S -> D (SR_RK was prefetched) */ + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx9_s_addc_u32, I9(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx9_s_load_dwordx4, I9(buf, n), P_S(SR_RK2), P_S(SR_KEYS), 0); + n =3D emit_aes_round_gfx9(buf, n, VR_S0, VR_S1, VR_S2, VR_S3, + VR_D0, VR_D1, VR_D2, VR_D3, + SR_RK, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + /* Even round: D -> S (SR_RK2 was prefetched during odd round) */ + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx9_s_addc_u32, I9(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx9_s_load_dwordx4, I9(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + n =3D emit_aes_round_gfx9(buf, n, VR_D0, VR_D1, VR_D2, VR_D3, + VR_S0, VR_S1, VR_S2, VR_S3, + SR_RK2, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_LOOP_CTR), P_I(1), + P_S(SR_LOOP_CTR)); + _E(emit_gfx9_s_cmp_lt_u32, I9(buf, n), P_S(SR_LOOP_CTR), + P_S(SR_NBLOCKS)); + br_loop =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + /* Final odd round: S -> D (SR_RK prefetched from last even) */ + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx9_s_addc_u32, I9(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx9_s_load_dwordx4, I9(buf, n), P_S(SR_RK2), P_S(SR_KEYS), 0); + n =3D emit_aes_round_gfx9(buf, n, VR_S0, VR_S1, VR_S2, VR_S3, + VR_D0, VR_D1, VR_D2, VR_D3, + SR_RK, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + /* Last round: D -> S (SR_RK2 prefetched during final odd) */ + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + n =3D emit_aes_last_round_gfx9(buf, n, VR_D0, VR_D1, VR_D2, VR_D3, + VR_S0, VR_S1, VR_S2, VR_S3, + SR_RK2, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + return n; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX9 GF(2^128) Multiply Helper + * + * Computes Z =3D X * Y in GF(2^128) with GCM polynomial. + * + * Input: X in v[VR_DATA0:VR_DATA3], Y in v[VR_D0:VR_D3] + * Output: Z in v[VR_S0:VR_S3] + * Clobbers: v[VR_TMP], v[VR_TMP2], v[VR_ADDR] (holds 0xE1000000), + * v[VR_DATA0:VR_DATA3] (shifted), v[VR_D0:VR_D3] (shifted), + * s[SR_LOOP_CTR] + * + * GCM bit ordering: bit 0 =3D MSB of first byte. + * Algorithm: Shoup's method (right-shift V, test MSB of X). + * Z =3D 0, V =3D Y + * for i =3D 0..127: + * if MSB(X) set: Z ^=3D V + * lsb =3D V[3] & 1 + * V >>=3D 1 (128-bit right shift) + * if lsb: V[0] ^=3D 0xE1000000 + * X <<=3D 1 (128-bit left shift) + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +static int emit_gfmul_128_gfx9(u32 *buf, int n) +{ + int loop_top, br_loop; + + /* Z =3D 0 */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S0), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S1), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S2), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S3), P_I(0)); + + /* Preload reduction constant into VR_ADDR (v10) */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_ADDR), P_L(0xE1000000)); + + /* Loop counter */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_LOOP_CTR), P_I(0)); + + loop_top =3D n; + + /* Step 1: Test MSB of X[0] via signed compare (bit 31 set =3D negative) = */ + _E(emit_gfx9_v_cmp_gt_i32, I9(buf, n), P_I(0), P_V(VR_DATA0)); + + /* + * Step 2: Conditional Z ^=3D V. + * v_cndmask selects V[i] or 0 based on VCC, then XOR into Z. + * v_cndmask_b32: vdst =3D VCC ? vsrc1 : src0 + * We want: VR_TMP2 =3D VCC ? VR_D0 : 0 + * So src0=3D0, vsrc1=3DVR_D0 + */ + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S0), P_V(VR_S0), + P_V(VR_TMP2)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S1), P_V(VR_S1), + P_V(VR_TMP2)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S2), P_V(VR_S2), + P_V(VR_TMP2)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D3)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_TMP2)); + + /* + * Step 3: Save LSB of V[3] for reduction. + */ + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(VR_TMP), P_I(1), P_V(VR_D3)); + + /* + * Step 4: 128-bit right shift V (v[D0:D3]). + * v_alignbit_b32(dst, hi, lo, 1) =3D {hi,lo} >> 1 (lower 32 bits). + */ + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_D3), P_V(VR_D2), + P_V(VR_D3), P_I(1)); + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_D2), P_V(VR_D1), + P_V(VR_D2), P_I(1)); + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_D1), P_V(VR_D0), + P_V(VR_D1), P_I(1)); + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(VR_D0), P_I(1), P_V(VR_D0)); + + /* + * Step 5: Conditional reduction V[0] ^=3D 0xE1000000. + * If saved LSB was 1, XOR with reduction polynomial. + * 0xE1000000 is preloaded in VR_ADDR (v10). + */ + _E(emit_gfx9_v_cmp_ne_u32, I9(buf, n), P_I(0), P_V(VR_TMP)); + /* VR_TMP2 =3D VCC ? VR_ADDR : 0 */ + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_ADDR)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_D0), P_V(VR_D0), + P_V(VR_TMP2)); + + /* + * Step 6: 128-bit left shift X (v[DATA0:DATA3]). + * X[0] =3D (X[0] << 1) | (X[1] >> 31) + * X[1] =3D (X[1] << 1) | (X[2] >> 31) + * X[2] =3D (X[2] << 1) | (X[3] >> 31) + * X[3] =3D X[3] << 1 + */ + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_DATA0), P_V(VR_DATA0), + P_V(VR_DATA1), P_I(31)); + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_DATA1), P_V(VR_DATA1), + P_V(VR_DATA2), P_I(31)); + _E(emit_gfx9_v_alignbit_b32, I9(buf, n), P_V(VR_DATA2), P_V(VR_DATA2), + P_V(VR_DATA3), P_I(31)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_DATA3), P_I(1), + P_V(VR_DATA3)); + + /* Loop control: 128 iterations */ + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_LOOP_CTR), P_I(1), + P_S(SR_LOOP_CTR)); + _E(emit_gfx9_s_cmp_lt_u32, I9(buf, n), P_S(SR_LOOP_CTR), P_L(128)); + br_loop =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + return n; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX10 (RDNA) AES-GCM emit helpers + * + * Same as GFX9 but with GFX10 encodings; GLOBAL saddr mode (scalar base + * + VGPR offset) avoids VOP3B v_add_co_u32 for 64-bit addresses. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* ---- GFX10 T-table lookup ---- */ + +static int emit_ttable_lookup_gfx10(u32 *buf, int n, + int v_addr, int v_val, + int v_state, int byte_pos, + int table_base, int s_mask, + int v_tmp) +{ + if (byte_pos =3D=3D 0) { + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(v_tmp), + P_S(s_mask), P_V(v_state)); + } else if (byte_pos =3D=3D 3) { + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(v_tmp), P_I(24), + P_V(v_state)); + } else { + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(v_tmp), + P_I(byte_pos * 8), P_V(v_state)); + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(v_tmp), + P_S(s_mask), P_V(v_tmp)); + } + + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(v_addr), P_I(2), + P_V(v_tmp)); + if (table_base > 0) + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(v_addr), + P_L(table_base), P_V(v_addr)); + + _E(emit_gfx10_ds_read_b32, I10(buf, n), v_val, v_addr); + + return n; +} + +/* ---- GFX10 AES round ---- */ + +static int emit_aes_round_gfx10(u32 *buf, int n, + int s0, int s1, int s2, int s3, + int d0, int d1, int d2, int d3, + int rk, int s_mask, int v_tmp, int v_addr, + int vt0, int vt1, int vt2, int vt3) +{ + /* Column 0: T0[s0.b0] ^ T1[s1.b1] ^ T2[s2.b2] ^ T3[s3.b3] */ + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt0, s0, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt1, s1, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt2, s2, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt3, s3, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx10_s_waitcnt, I10(buf, n), 0x3F, 0); + + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d0), P_V(vt0), P_V(vt1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d0), P_V(d0), P_V(vt2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d0), P_V(d0), P_V(vt3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d0), P_S(rk), P_V(d0)); + + /* Column 1: T0[s1.b0] ^ T1[s2.b1] ^ T2[s3.b2] ^ T3[s0.b3] */ + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt0, s1, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt1, s2, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt2, s3, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt3, s0, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx10_s_waitcnt, I10(buf, n), 0x3F, 0); + + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d1), P_V(vt0), P_V(vt1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d1), P_V(d1), P_V(vt2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d1), P_V(d1), P_V(vt3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d1), P_S(rk + 1), + P_V(d1)); + + /* Column 2 */ + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt0, s2, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt1, s3, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt2, s0, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt3, s1, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx10_s_waitcnt, I10(buf, n), 0x3F, 0); + + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d2), P_V(vt0), P_V(vt1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d2), P_V(d2), P_V(vt2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d2), P_V(d2), P_V(vt3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d2), P_S(rk + 2), + P_V(d2)); + + /* Column 3 */ + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt0, s3, 0, 0, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt1, s0, 1, 1024, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt2, s1, 2, 2048, s_mask, + v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt3, s2, 3, 3072, s_mask, + v_tmp); + _E(emit_gfx10_s_waitcnt, I10(buf, n), 0x3F, 0); + + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d3), P_V(vt0), P_V(vt1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d3), P_V(d3), P_V(vt2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d3), P_V(d3), P_V(vt3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(d3), P_S(rk + 3), + P_V(d3)); + + return n; +} + +/* ---- GFX10 AES last round ---- */ + +static int emit_aes_last_round_gfx10(u32 *buf, int n, + int s0, int s1, int s2, int s3, + int d0, int d1, int d2, int d3, + int rk, int s_mask, int v_tmp, int v_addr, + int vt0, int vt1, int vt2, int vt3) +{ + int cols[4][4] =3D { + {s0, s1, s2, s3}, + {s1, s2, s3, s0}, + {s2, s3, s0, s1}, + {s3, s0, s1, s2}, + }; + int dsts[4] =3D {d0, d1, d2, d3}; + int col; + + for (col =3D 0; col < 4; col++) { + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt0, cols[col][0], + 0, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt1, cols[col][1], + 1, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt2, cols[col][2], + 2, 0, s_mask, v_tmp); + n =3D emit_ttable_lookup_gfx10(buf, n, v_addr, vt3, cols[col][3], + 3, 0, s_mask, v_tmp); + _E(emit_gfx10_s_waitcnt, I10(buf, n), 0x3F, 0); + + /* S(x) =3D (T0[x] >> 8) & 0xFF */ + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(vt0), P_I(8), + P_V(vt0)); + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(vt0), P_S(s_mask), + P_V(vt0)); + + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(vt1), P_L(0xFF00), + P_V(vt1)); + + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(vt2), P_I(8), + P_V(vt2)); + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(vt2), + P_L(0xFF0000), P_V(vt2)); + + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(vt3), P_I(16), + P_V(vt3)); + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(vt3), + P_L(0xFF000000), P_V(vt3)); + + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(dsts[col]), + P_V(vt0), P_V(vt1)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(dsts[col]), + P_V(dsts[col]), P_V(vt2)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(dsts[col]), + P_V(dsts[col]), P_V(vt3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(dsts[col]), + P_S(rk + col), P_V(dsts[col])); + } + + return n; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX10 AES Encrypt Block Helper + * + * Same as emit_aes_encrypt_block_gfx9 but with GFX10 instructions. + * Input: plaintext in v[VR_S0:VR_S3] + * Output: ciphertext in v[VR_S0:VR_S3] + * Requires: SR_KEYS, SR_NR_ROUNDS set. T-tables in LDS. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +static int emit_aes_encrypt_block_gfx10(u32 *buf, int n) +{ + int br_loop, loop_top; + + /* Round 0: XOR with first round key */ + _E(emit_gfx10_s_load_dwordx4, I10(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S0), P_S(SR_RK), + P_V(VR_S0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S1), P_S(SR_RK + 1), + P_V(VR_S1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S2), P_S(SR_RK + 2), + P_V(VR_S2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S3), P_S(SR_RK + 3), + P_V(VR_S3)); + + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx10_s_addc_u32, I10(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + + /* pair_count =3D nr_rounds / 2 - 1 */ + _E(emit_gfx10_s_lshr_b32, I10(buf, n), P_S(SR_NBLOCKS), + P_S(SR_NR_ROUNDS), P_I(1)); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_NBLOCKS), P_L(0xFFFFFFFFu), + P_S(SR_NBLOCKS)); + + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_LOOP_CTR), P_I(0)); + + /* Prefetch round 1 key - overlaps with loop-entry overhead */ + _E(emit_gfx10_s_load_dwordx4, I10(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + + loop_top =3D n; + + /* Odd round: S -> D (SR_RK was prefetched) */ + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx10_s_addc_u32, I10(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx10_s_load_dwordx4, I10(buf, n), P_S(SR_RK2), P_S(SR_KEYS), + 0); + n =3D emit_aes_round_gfx10(buf, n, VR_S0, VR_S1, VR_S2, VR_S3, + VR_D0, VR_D1, VR_D2, VR_D3, + SR_RK, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + /* Even round: D -> S (SR_RK2 was prefetched during odd round) */ + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx10_s_addc_u32, I10(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx10_s_load_dwordx4, I10(buf, n), P_S(SR_RK), P_S(SR_KEYS), 0); + n =3D emit_aes_round_gfx10(buf, n, VR_D0, VR_D1, VR_D2, VR_D3, + VR_S0, VR_S1, VR_S2, VR_S3, + SR_RK2, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_LOOP_CTR), P_I(1), + P_S(SR_LOOP_CTR)); + _E(emit_gfx10_s_cmp_lt_u32, I10(buf, n), P_S(SR_LOOP_CTR), + P_S(SR_NBLOCKS)); + br_loop =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + /* Final odd round: S -> D (SR_RK prefetched from last even) */ + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_KEYS), P_I(16), + P_S(SR_KEYS)); + _E(emit_gfx10_s_addc_u32, I10(buf, n), P_S(SR_KEYS + 1), P_I(0), + P_S(SR_KEYS + 1)); + _E(emit_gfx10_s_load_dwordx4, I10(buf, n), P_S(SR_RK2), P_S(SR_KEYS), + 0); + n =3D emit_aes_round_gfx10(buf, n, VR_S0, VR_S1, VR_S2, VR_S3, + VR_D0, VR_D1, VR_D2, VR_D3, + SR_RK, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + /* Last round: D -> S (SR_RK2 prefetched during final odd) */ + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + n =3D emit_aes_last_round_gfx10(buf, n, VR_D0, VR_D1, VR_D2, VR_D3, + VR_S0, VR_S1, VR_S2, VR_S3, + SR_RK2, SR_MASK, VR_TMP, VR_ADDR, + VR_DATA0, VR_DATA1, VR_DATA2, VR_DATA3); + + return n; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * GFX10 GF(2^128) Multiply Helper + * + * Same algorithm as GFX9 version but with GFX10 instructions. + * Input: X in v[VR_DATA0:VR_DATA3], Y in v[VR_D0:VR_D3] + * Output: Z in v[VR_S0:VR_S3] + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +static int emit_gfmul_128_gfx10(u32 *buf, int n) +{ + int loop_top, br_loop; + + /* Z =3D 0 */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S0), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S1), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S2), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S3), P_I(0)); + + /* Preload reduction constant into VR_ADDR (v10) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_ADDR), + P_L(0xE1000000)); + + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_LOOP_CTR), P_I(0)); + + loop_top =3D n; + + /* Step 1: Test MSB of X[0] via signed compare (bit 31 set =3D negative) = */ + _E(emit_gfx10_v_cmp_gt_i32, I10(buf, n), P_I(0), P_V(VR_DATA0)); + + /* Step 2: Conditional Z ^=3D V */ + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S0), P_V(VR_S0), + P_V(VR_TMP2)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S1), P_V(VR_S1), + P_V(VR_TMP2)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S2), P_V(VR_S2), + P_V(VR_TMP2)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_D3)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_TMP2)); + + /* Step 3: Save LSB of V[3] */ + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(VR_TMP), P_I(1), + P_V(VR_D3)); + + /* Step 4: 128-bit right shift V */ + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_D3), P_V(VR_D2), + P_V(VR_D3), P_I(1)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_D2), P_V(VR_D1), + P_V(VR_D2), P_I(1)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_D1), P_V(VR_D0), + P_V(VR_D1), P_I(1)); + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(VR_D0), P_I(1), + P_V(VR_D0)); + + /* Step 5: Conditional reduction V[0] ^=3D 0xE1000000 */ + _E(emit_gfx10_v_cmp_ne_u32, I10(buf, n), P_I(0), P_V(VR_TMP)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), P_V(VR_TMP2), P_I(0), + P_V(VR_ADDR)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_D0), P_V(VR_D0), + P_V(VR_TMP2)); + + /* Step 6: 128-bit left shift X */ + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA0), P_V(VR_DATA0), + P_V(VR_DATA1), P_I(31)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA1), P_V(VR_DATA1), + P_V(VR_DATA2), P_I(31)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA2), P_V(VR_DATA2), + P_V(VR_DATA3), P_I(31)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_DATA3), P_I(1), + P_V(VR_DATA3)); + + /* Loop control: 128 iterations */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_LOOP_CTR), P_I(1), + P_S(SR_LOOP_CTR)); + _E(emit_gfx10_s_cmp_lt_u32, I10(buf, n), P_S(SR_LOOP_CTR), P_L(128)); + br_loop =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + return n; +} + +#undef _E +#undef _BR + +#endif /* AESGCM_SHADER_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h b/drivers/= gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h new file mode 100644 index 000000000000..555bf2962f73 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h @@ -0,0 +1,1796 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +/* + * KNOD IPsec fused RX shader - GFX9 (Vega10/20). + * + * Full AES-GCM decrypt pipeline for inbound ESP packets: + * 1) ESP header parse -> SPI + seq extract + * 2) SA table linear scan -> resolve SPI to slot index + * 3) Cooperative T-table load (VRAM -> LDS, 256 threads) + * 4) AES-CTR decrypt ciphertext -> out_addr + * 5) Parallel GHASH over (AAD || ciphertext || len) + * 6) ICV verify (GHASH ^ AES(K,J0) vs received tag) + * 7) ESP trailer strip -> inner_len + * 8) Write verdict to bd->act, inner_len to bd->len + * + * Dispatch geometry: + * workgroup =3D (256, 1, 1) - 256 threads =3D 1 AES block per thre= ad + * grid =3D (256, nr_pkts, 1) + * workgroup_id_y =3D=3D packet index in the batch + * + * Anti-replay is NOT in the shader - CPU-side sliding window in NIC NAPI. + * + * Verdict encoding in bd->act high32: + * 0..NR_SA-1 - SA hit + ICV pass, value is slot_idx + * 0xFFFFFFFF - SA miss (no entry for this SPI) + * 0xFFFFFFFE - non-IPv4/IPv6 bypass (unknown L3 protocol) + * 0xFFFFFFFD - ICV mismatch (decrypt succeeded but tag wrong) + * + * bd->len is set to inner_len on success (decrypted payload minus ESP + * trailer and padding). On miss/bypass/ICV-fail, bd->len is left as-is. + */ + +#ifndef KNOD_HELPERS_IPSEC_FUSED_GFX10_H_ +#define KNOD_HELPERS_IPSEC_FUSED_GFX10_H_ + +#include +#include "knod_amdgpu_insn.h" +/* Provide AESGCM_MAX_DIM_Y so aesgcm_shader.h compiles (OFF_T0 macro). + * Only emit_aes_encrypt_block_gfx10 / emit_gfmul_128_gfx10 are used here; + * the full aesgcm_gen_shader_* functions are unreferenced. + */ +#ifndef AESGCM_MAX_DIM_Y +#define AESGCM_MAX_DIM_Y 1024 +#endif +#include "aesgcm_shader.h" + +/* SA entry constants - must match knod_ipsec.h */ +#define KNOD_IPSEC_SHADER_NR_SA 256 +#define KNOD_IPSEC_SHADER_SA_ENTRY_SZ 104 + +/* SA entry field offsets (struct knod_ipsec_sa_entry) */ +#define SA_OFF_SPI 0 +#define SA_OFF_KEY_ADDR 16 +#define SA_OFF_HTABLE_ADDR 24 +#define SA_OFF_T_TABLES_ADDR 32 +#define SA_OFF_SALT 40 +#define SA_OFF_KEY_LEN 44 +#define SA_OFF_NR_ROUNDS 48 +#define SA_OFF_MODE 52 /* XFRM_MODE_TRANSPORT=3D0, TUNNEL=3D1 */ +#define SA_OFF_STATS_ADDR 88 /* per-SA GPU stats (u64 gpu addr) */ + +/* ESP packet geometry (ETH=3D14, IPv4=3D20 / IPv6=3D40, no VLAN/opts). + * IPv4: ESP header starts at offset 34 (14+20). + * IPv6: ESP header starts at offset 54 (14+40). + * Within ESP header: SPI+0, seq+4, IV+8, ctext+16. + * The shader dynamically computes offsets based on IP version. + */ +#define ESP_HDR_OFF_V4 34 /* ETH(14) + IPv4(20) */ +#define ESP_HDR_OFF_V6 54 /* ETH(14) + IPv6(40) */ +#define ESP_REL_SPI 0 +#define ESP_REL_SEQ 4 +#define ESP_REL_IV 8 +#define ESP_REL_CTEXT 16 /* SPI(4)+seq(4)+IV(8) */ +#define ESP_ICV_LEN 16 + +/* Fixed IPv4 layout offsets used by the crypto KAT */ +#define ESP_SPI_OFF 34 +#define ESP_SEQ_OFF 38 +#define ESP_IV_OFF 42 +#define ESP_CTEXT_OFF 50 + +/* Fused sub[] offsets within kernarg (sub[i] =3D kernarg + 40 + i*32) */ +#define SUB_BASE_OFF 40 +#define SUB_STRIDE 32 +#define SUB_OFF_PKT_ADDR 0 +#define SUB_OFF_OUT_ADDR 8 +#define SUB_OFF_BD_ADDR 16 +#define SUB_OFF_PKT_LEN 24 +#define SUB_OFF_RESULT_SEQ 28 + +/* ICV-fail sentinel (distinct from MISS=3D0xFFFFFFFF and BYPASS=3D0xFFFFF= FFE) */ +#define VERDICT_ICV_FAIL 0xFFFFFFFDu + +/* High VGPRs for saving pre-crypto IPsec state (above AES v0-v22 range) */ +#define VR_SAVE_SLOT 30 +#define VR_SAVE_BD_LO 31 +#define VR_SAVE_BD_HI 32 +#define VR_SAVE_PKT_LO 33 +#define VR_SAVE_PKT_HI 34 +#define VR_SAVE_PKTLEN 35 +#define VR_SAVE_OUT_LO 36 +#define VR_SAVE_OUT_HI 37 +#define VR_SAVE_SEQ 38 +#define VR_SAVE_SPI 39 +#define VR_SAVE_STATS_LO 40 /* per-SA stats GPU addr low */ +#define VR_SAVE_STATS_HI 41 /* per-SA stats GPU addr high */ +/* ESP header offset: 34(v4) or 54(v6) */ +#define VR_SAVE_ESP_OFF 42 +/* Ciphertext prefetch destination - free v23-v27, outside AES v0-v22 rang= e */ +#define VR_PREFETCH0 23 +#define VR_PREFETCH1 24 +#define VR_PREFETCH2 25 +#define VR_PREFETCH3 26 +/* extra dword for GFX10 unaligned fix */ +#define VR_PREFETCH4 27 + +/* Extra SGPRs for IPsec-specific state that survives into AES phases. + * These must NOT collide with SR_* from aesgcm_shader.h (s18-s49, s56-s59= ). + * s50-s55 are IPsec-specific. s56-s59 =3D SR_RK2 (AES round key double-bu= ffer). + */ +#define SR_CTEXT_LEN 50 /* ciphertext length in bytes */ +#define SR_NBLOCKS_GCM 51 /* ceil(ctext_len/16) */ +#define SR_HTABLE_LO 52 /* H-power table GPU addr */ +#define SR_HTABLE_HI 53 +#define SR_TOTAL_GHASH_BLK 54 /* nblocks + 2 (AAD + ctext + len) */ +#define SR_SA_MODE 55 /* XFRM_MODE_TRANSPORT=3D0, TUNNEL=3D1 */ + +/* File-local emit helpers */ +#ifndef _KNOD_IPSEC_EMIT +#define _KNOD_IPSEC_EMIT +#define _E(fn, ...) (n +=3D fn(__VA_ARGS__) / 4) +#define _BR(fn, ...) ({ int _p =3D n; n +=3D fn(__VA_ARGS__) / 4; _p; }) +#endif + +/* + * KNOD_IPSEC_GFX10_DIAG_STUB - graduated diagnostic stubs. + * Uncomment exactly one level to bisect the SQC inst-fault: + * + * Level 1: bare s_endpgm - tests KD, entry offset, BO mapping. + * Level 2: compute bd_addr from kernarg, store 0xDEAD0001, endpgm. + * Tests SGPR layout, VOP2/VOP1 ALU, GLOBAL load/store. + * Level 3: full Phase 0 (parse + SA scan) + write result, endpgm. + * Tests branches, patch_branch, VOP3B carry, SOPC. + * + * Leave all commented out for the real shader. + */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 1 */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 7 */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 2 */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 3 */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 4 */ +/* #define KNOD_IPSEC_GFX10_DIAG_STUB 5 */ + +static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf) +{ + int br_skip12, br_skip14, br_skip15, br_skip18, br_skip16, br_skip13; + int br_skip_aad, br_skip_ctext, br_skip_len; + int loop_top, br_match, br_loop, br_end; + int br_no_sdma, br_no_copy, br_not_last; + int br_ipv4, br_bypass, br_crypto_end; + int br_ipv6, br_v6_to_common; + int br_not_transport, li; + const int L3_TMP_BASE =3D 14; /* v14..v23 */ + u32 *buf =3D (u32 *)vbuf; + int br_crypto_done; + int br_execz_ctr; + int br_execz2; + int br_icv_bad; + int br_icv_ok; + int br_tid0; + int br_skip; + int br_ok; + int level; + int n =3D 0; + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 1 + /* LEVEL 1: bare s_endpgm - if this faults, KD or BO mapping is + * wrong + */ + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 1 (bare endpgm), %d bytes\n", + n * 4); + return n * 4; +#endif + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 2 + /* LEVEL 2: load bd_addr from kernarg, write 0xDEAD0001 to bd+8. + * Tests: s8/s9 kernarg ptr, s16 workgroup_id_y, VOP1/VOP2 ALU, + * GLOBAL_LOAD_DWORDX2, GLOBAL_STORE_DWORD, s_waitcnt. + */ + _E(emit_gfx10_s_waitcnt_vmcnt_lgkmcnt, I10(buf, n)); + /* v1 =3D sub offset =3D 40 + wg_id_y * 32 */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(1), P_S(16)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(1), P_I(5), P_V(1)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(1), P_L(SUB_BASE_OFF), + P_V(1)); + /* v[3:4] =3D &sub[wg_id_y] */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(2), P_S(9)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(3), P_S(8), P_V(1)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(4), P_I(0), P_V(2)); + /* v[5:6] =3D sub[].bd_addr */ + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(5), P_V(3), + SUB_OFF_BD_ADDR); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + /* v0 =3D 0 for lane check: only lane 0 writes */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + /* write 0xDEAD0001 to bd->act (offset +8) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0001u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(5), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 2 (bd write), %d bytes\n", + n * 4); + return n * 4; +#endif + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 3 + /* LEVEL 3: s_dcache_inv (SMEM 8-byte) + Level 2 body. + * If this faults but Level 2 passed, SMEM encoding is the culprit. + */ + _E(emit_gfx10_s_dcache_inv, I10(buf, n)); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + /* --- Level 2 body below --- */ + _E(emit_gfx10_s_waitcnt_vmcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(1), P_S(16)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(1), P_I(5), P_V(1)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(1), P_L(SUB_BASE_OFF), + P_V(1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(2), P_S(9)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(3), P_S(8), P_V(1)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(4), P_I(0), P_V(2)); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(5), P_V(3), + SUB_OFF_BD_ADDR); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0003u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(5), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 3 (SMEM + bd write), %d bytes\= n", + n * 4); + return n * 4; +#endif + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 4 + /* LEVEL 4: SOP1 + SOP2 + SOPC + SMEM + Level 2 body. + * Tests the three scalar encoding formats not covered by Levels 1-3. + * SOP1: s_mov_b32 (encoding 0x17D) + * SOP2: s_add_u32, s_lshr_b32 (encoding 0x2) + * SOPC: s_cmp_eq_u32 + s_cbranch_scc1 (encoding 0x17E) + */ + _E(emit_gfx10_s_dcache_inv, I10(buf, n)); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + + /* SOP1: s_mov_b32 s28, 0xCAFE0004 (literal) */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(28), P_L(0xCAFE0004u)); + /* SOP2: s_add_u32 s28, s28, 1 (inline const) */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(28), P_S(28), P_I(1)); + /* SOP2: s_lshr_b32 s28, s28, 0 (nop shift) */ + _E(emit_gfx10_s_lshr_b32, I10(buf, n), P_S(28), P_S(28), P_I(0)); + /* SOPC: s_cmp_eq_u32 s28, 0xCAFE0005 - should set SCC=3D1 */ + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(28), P_L(0xCAFE0005u)); + br_ok =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + /* SCC=3D0 path: write 0xBAD00004 as error marker */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(28), P_L(0xBAD00004u)); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + patch_branch(buf, br_ok, n); + + /* --- Level 2 body: bd write --- */ + _E(emit_gfx10_s_waitcnt_vmcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(1), P_S(16)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(1), P_I(5), P_V(1)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(1), P_L(SUB_BASE_OFF), + P_V(1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(2), P_S(9)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(3), P_S(8), P_V(1)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(4), P_I(0), P_V(2)); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(5), P_V(3), + SUB_OFF_BD_ADDR); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0004u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(5), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 4 (SOP1/SOP2/SOPC + bd), %d by= tes\n", + n * 4); + return n * 4; +#endif + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 0: Parse ESP header + SA table lookup + * + * s_dcache_inv: flush K$ so s_load reads fresh round keys. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_s_dcache_inv, I10(buf, n)); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_BSWAP), P_L(0x00010203)); + + /* v1 =3D 40 + wg_id_y*32 =3D offset of sub[wg_id_y] within kernarg */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(1), P_S(16)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(1), P_I(5), P_V(1)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(1), P_L(SUB_BASE_OFF), + P_V(1)); + + /* v[3:4] =3D kernarg_ptr + v1 =3D &sub[wg_id_y] */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(2), P_S(9)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(3), P_S(8), P_V(1)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(4), P_I(0), P_V(2)); + + /* Load sub[].pkt_addr -> v[9:10], sub[].bd_addr -> v[5:6] */ + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(9), P_V(3), + SUB_OFF_PKT_ADDR); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(5), P_V(3), + SUB_OFF_BD_ADDR); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* Seed v[11:12] with pkt_addr as a safe default BEFORE the IP + * version branch. The bypass path (non-v4/v6 packets like ARP) + * unconditionally branches past the v[11:12] setup at line ~195 + * and later Phase 1 does a global_load at v[11:12]+ESP_REL_SEQ to + * read the ESP sequence number. Without this seed v[11:12] would + * hold uninitialised VGPR state (wave launch garbage), producing + * a fault at ~0x{random}_00000000. For valid v4/v6 packets the + * common path below overwrites v[11:12] with pkt+esp_hdr_off so + * this seed is harmless. + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(11), P_V(9)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(12), P_V(10)); + + /* IP version gate: load dword at pkt+12 to get first byte of L3 + * header (byte[14]). Extract version nibble -> s28. + */ + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(15), P_V(9), 12); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 28, 15); + _E(emit_gfx10_s_lshr_b32, I10(buf, n), P_S(28), P_S(28), P_I(20)); + _E(emit_gfx10_s_and_b32_p, I10(buf, n), P_S(28), P_I(0xF), P_S(28)); + + /* Check IPv4 (version=3D=3D4) */ + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(28), P_I(4)); + br_ipv4 =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + + /* Check IPv6 (version=3D=3D6) */ + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(28), P_I(6)); + br_ipv6 =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + + /* Bypass: neither IPv4 nor IPv6 */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(26), P_L(0xFFFFFFFEu)); + br_bypass =3D _BR(emit_gfx10_s_branch, I10(buf, n), 0); + + /* IPv6 landing: esp_hdr_off =3D 54 */ + patch_branch(buf, br_ipv6, n); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_ESP_OFF), + P_L(ESP_HDR_OFF_V6)); + br_v6_to_common =3D _BR(emit_gfx10_s_branch, I10(buf, n), 0); + + /* IPv4 landing: esp_hdr_off =3D 34 */ + patch_branch(buf, br_ipv4, n); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_ESP_OFF), + P_L(ESP_HDR_OFF_V4)); + + /* Common path: both IPv4 and IPv6 converge here */ + patch_branch(buf, br_v6_to_common, n); + + /* v[11:12] =3D pkt_addr + esp_hdr_off (dynamic) */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(11), + P_V(VR_SAVE_ESP_OFF), P_V(9)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(12), P_I(0), + P_V(10)); + + /* v13 =3D *(u32*)(pkt + esp_hdr_off) - SPI in big-endian. + * + * GFX10 (RDNA2) quirk: global_load_dword silently clears EA's + * low 2 bits, so loading at v[11:12] =3D pkt + 34 (v4) or pkt + 54 + * (v6) would actually read pkt + 32 / pkt + 52. Both ESP offsets + * are 2 mod 4, so the shift to reconstruct the target dword is + * always 16 bits. Use dwordx2 (HW still clears low 2 bits, but + * we get enough data) + v_alignbit_b32 to extract bytes + * [addr..addr+3] from the 8-byte window. + */ + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(20), P_V(11), 0); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + /* v_alignbit_b32 D, HIGH, LOW, shift: D =3D ({HIGH, LOW} >> shift)[31:0]. + * tmp_lo =3D v20 (memory-low dword), tmp_hi =3D v21 (memory-high dword). + */ + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(13), P_V(21), P_V(20), + P_I(16)); + + /* Byteswap SPI: v13 =3D bswap32(v13) via v_perm_b32 */ + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(13), P_V(13), P_V(13), + P_S(SR_BSWAP)); + + /* SA table linear scan (VMEM path for K$ coherence). + * s22 =3D target SPI, s[24:25] =3D sa_table_addr, s23 =3D counter, + * s26 =3D result (slot_idx or 0xFFFFFFFF), v[16:17] =3D running ptr. + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(16), P_S(8)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(17), P_S(9)); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(18), P_V(16), 0); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 24, 18); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 25, 19); + + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 22, 13); + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(23), P_I(0)); + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(26), P_L(0xFFFFFFFFu)); + + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(16), P_S(24)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(17), P_S(25)); + + loop_top =3D n; + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(20), P_V(16), + SA_OFF_SPI); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 27, 20); + + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(16), + P_L(KNOD_IPSEC_SHADER_SA_ENTRY_SZ), P_V(16)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(17), P_I(0), + P_V(17)); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(23), P_I(1), P_S(23)); + _E(emit_gfx10_s_nop, I10(buf, n)); + + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(27), P_S(22)); + br_match =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + + _E(emit_gfx10_s_cmp_lt_u32, I10(buf, n), P_S(23), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_loop =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + br_end =3D _BR(emit_gfx10_s_branch, I10(buf, n), 0); + + /* Match: s26 =3D s23 - 1 */ + patch_branch(buf, br_match, n); + _E(emit_gfx10_s_sub_u32_p, I10(buf, n), P_S(26), P_S(23), P_I(1)); + + patch_branch(buf, br_end, n); + + patch_branch(buf, br_bypass, n); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 5 + /* LEVEL 5: early exit after real Phase 0. + * s26 =3D slot_idx (or 0xFFFFFFFF if no match). + * Write s26 to bd->act via v[5:6] (bd_addr loaded during Phase 0). + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0005u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(5), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 5 (Phase 0 + exit), %d bytes\n= ", + n * 4); + return n * 4; +#endif + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 1: Save pre-crypto state + load extra sub[] fields + * + * Move IPsec-specific values to v30+ so v1-v22 and s18-s49 are + * free for AES-GCM helpers from aesgcm_shader.h. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_SLOT), P_S(26)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_BD_LO), P_V(5)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_BD_HI), P_V(6)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_PKT_LO), P_V(9)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_PKT_HI), P_V(10)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_SPI), P_V(13)); + + /* Load sub[].out_addr -> v[VR_SAVE_OUT_LO:VR_SAVE_OUT_HI] */ + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(VR_SAVE_OUT_LO), + P_V(3), SUB_OFF_OUT_ADDR); + /* Load sub[].pkt_len -> v[VR_SAVE_PKTLEN] */ + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_SAVE_PKTLEN), + P_V(3), SUB_OFF_PKT_LEN); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* Load ESP seq number: pkt + esp_hdr_off + 4, BE -> bswap -> + * v[VR_SAVE_SEQ]. + * v[11:12] still holds pkt_addr + esp_hdr_off from Phase 0. + * + * GFX10 unaligned load fix: esp_hdr_off is 34(v4) or 54(v6), + * both =3D=3D 2 mod 4. EA =3D pkt+38 clips to pkt+36. Load dwordx2 + * from the clipped addr, then v_alignbit_b32 shift=3D16 to + * reconstruct the target dword. v[20:21] are free scratch. + */ + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(20), P_V(11), + ESP_REL_SEQ); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_SAVE_SEQ), P_V(21), + P_V(20), P_I(16)); + + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_SAVE_SEQ), + P_V(VR_SAVE_SEQ), P_V(VR_SAVE_SEQ), P_S(SR_BSWAP)); + + /* Write bswapped seq back into sub[].result_seq for CPU finish worker. + * v[3:4] still points to &sub[wg_id_y]. + */ + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(3), + P_V(VR_SAVE_SEQ), SUB_OFF_RESULT_SEQ); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 2: Branch on miss/bypass - skip crypto entirely + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 26, VR_SAVE_SLOT); + _E(emit_gfx10_s_cmp_ge_u32, I10(buf, n), P_S(26), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_crypto_end =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 3: Load SA entry fields for the matched slot + * + * entry_addr =3D sa_table_addr + slot_idx * SA_ENTRY_SIZE + * Load: key_gpu_addr, salt, nr_rounds, t_tables_gpu_addr, + * htable_gpu_addr + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* s27 =3D slot_idx * SA_ENTRY_SIZE (scalar mul) */ + _E(emit_gfx10_s_mul_i32, I10(buf, n), P_S(27), P_S(26), + P_L(KNOD_IPSEC_SHADER_SA_ENTRY_SZ)); + /* s[24:25] =3D sa_table_addr (already there from Phase 0 scan) */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(24), P_S(24), P_S(27)); + _E(emit_gfx10_s_addc_u32, I10(buf, n), P_S(25), P_S(25), P_I(0)); + + /* Use VMEM for coherence: stage entry addr into VGPR pair */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_LO), P_S(24)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_HI), P_S(25)); + + /* Batched SA entry loads: issue all 4 loads to different + * VGPR destinations, then single waitcnt. v1-v8 (S0-S3, + * D0-D3) are free at Phase 3 - not used until Phase 7. + * + * Layout: + * dwordx4 @+16 -> v[1:4]: key_lo, key_hi, htable_lo, htable_hi + * dwordx4 @+32 -> v[5:8]: ttables_lo, ttables_hi, salt, key_len + * dwordx2 @+48 -> v[14:15]: nr_rounds, mode + * dwordx2 @+88 -> v[16:17]: stats_lo, stats_hi + */ + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_S0), + P_V(VR_GA_LO), SA_OFF_KEY_ADDR); + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_D0), + P_V(VR_GA_LO), SA_OFF_T_TABLES_ADDR); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), SA_OFF_NR_ROUNDS); + _E(emit_gfx10_global_load_dwordx2, I10(buf, n), P_V(VR_DATA2), + P_V(VR_GA_LO), SA_OFF_STATS_ADDR); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* key_addr: v1=3Dlo, v2=3Dhi */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_KEYS, VR_S0); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_KEYS + 1, VR_S1); + /* htable_addr: v3=3Dlo, v4=3Dhi */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_HTABLE_LO, VR_S2); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_HTABLE_HI, VR_S3); + /* t_tables_addr: v5=3Dlo, v6=3Dhi */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_T_ADDR, VR_D0); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_T_ADDR + 1, VR_D1); + /* salt: v7 */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_IV0, VR_D2); + /* nr_rounds: v14, mode: v15 */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_NR_ROUNDS, VR_DATA0); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_SA_MODE, VR_DATA1); + /* stats_addr: v16=3Dlo, v17=3Dhi -> save VGPRs for Phase 10 */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_STATS_LO), + P_V(VR_DATA2)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_STATS_HI), + P_V(VR_DATA3)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 4: Build AES-GCM nonce + * + * nonce[12] =3D salt[4] || IV[8] + * salt is already in s[SR_IV0]. Load IV from pkt + esp_hdr_off + 8. + * Recompute ESP base from saved pkt_addr + VR_SAVE_ESP_OFF since + * v[11:12] were clobbered by Phase 3 SA loads (VR_GA_LO=3D12). + * IV goes to s[SR_IV1] (bytes 4-7) and s[SR_IV2] (bytes 8-11). + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + /* GFX10 unaligned load fix: EA =3D pkt+42/62 =3D=3D 2 mod 4, clips + * to pkt+40/60. Load dwordx4 (16B from clipped addr), then + * alignbit shift=3D16 to reconstruct IV[0:3] and IV[4:7]. + */ + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), ESP_REL_IV); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA1), P_V(VR_DATA0), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_DATA2), P_V(VR_DATA1), P_I(16)); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_IV1, VR_DATA0); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_IV2, VR_DATA1); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 5: Compute ciphertext bounds + * + * ctext_off =3D esp_hdr_off + 16 (SPI+seq+IV) + * ctext_len =3D pkt_len - ctext_off - ICV_LEN + * nblocks =3D (ctext_len + 15) >> 4 + * + * s28 is free here (last used in version gate) - use as scratch. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 28, VR_SAVE_ESP_OFF); + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(28), + P_I(ESP_REL_CTEXT + ESP_ICV_LEN), + P_S(28)); /* s28 =3D ctext_off + ICV_LEN =3D overhead to subtract */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_CTEXT_LEN, + VR_SAVE_PKTLEN); + _E(emit_gfx10_s_sub_u32_p, I10(buf, n), P_S(SR_CTEXT_LEN), + P_S(SR_CTEXT_LEN), P_S(28)); + /* s[SR_NBLOCKS_GCM] =3D (ctext_len + 15) >> 4 */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_NBLOCKS_GCM), P_I(15), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx10_s_lshr_b32, I10(buf, n), P_S(SR_NBLOCKS_GCM), + P_S(SR_NBLOCKS_GCM), P_I(4)); + /* total GHASH blocks =3D 1(AAD) + nblocks(ctext) + 1(len) =3D nblocks + 2 + */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_TOTAL_GHASH_BLK), + P_I(2), P_S(SR_NBLOCKS_GCM)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 6: Cooperative T-table load (VRAM -> LDS) + * + * All 256 threads load from SA's t_tables_gpu_addr. Each thread + * loads one u32 per table (4 tables x 256 entries =3D 4KB). + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_MASK), P_L(0xFF)); + + /* v[VR_TMP] =3D tid * 4 (byte offset within each 1KB table) */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_TMP), P_I(2), + P_V(VR_TID)); + + /* T0: VRAM[t_tables + tid*4] -> LDS[tid*4] */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_LO), + P_S(SR_T_ADDR)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_HI), + P_S(SR_T_ADDR + 1)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), P_V(VR_GA_LO), + P_V(VR_TMP)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), P_I(0), + P_V(VR_GA_HI)); + /* GFX10 GLOBAL offset is 12-bit signed (-2048..+2047). + * Offsets 2048 and 3072 overflow, so advance the base VGPR + * after the first two loads. + */ + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_DATA1), + P_V(VR_GA_LO), 1024); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), P_L(2048), + P_V(VR_GA_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), P_I(0), + P_V(VR_GA_HI)); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_DATA2), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_DATA3), + P_V(VR_GA_LO), 1024); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* Write to LDS: T0 at +0, T1 at +1024, T2 at +2048, T3 at +3072 */ + _E(emit_gfx10_ds_write_b32, I10(buf, n), VR_TMP, VR_DATA0); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_ADDR), P_L(1024), + P_V(VR_TMP)); + _E(emit_gfx10_ds_write_b32, I10(buf, n), VR_ADDR, VR_DATA1); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_ADDR), P_L(2048), + P_V(VR_TMP)); + _E(emit_gfx10_ds_write_b32, I10(buf, n), VR_ADDR, VR_DATA2); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_ADDR), P_L(3072), + P_V(VR_TMP)); + _E(emit_gfx10_ds_write_b32, I10(buf, n), VR_ADDR, VR_DATA3); + + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_barrier, I10(buf, n)); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 6 + /* LEVEL 6: early exit after Phase 6 (T-table -> LDS + barrier). + * Tests Phase 1-6: SA loads, nonce build, DS writes, s_barrier. + * bd_addr is in v[VR_SAVE_BD_LO:VR_SAVE_BD_HI] =3D v[31:32]. + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0006u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 6 (Phase 0-6 + exit), %d bytes= \n", + n * 4); + return n * 4; +#endif + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 7: AES-CTR decrypt + * + * Each thread handles block_id =3D tid. Only threads with tid < + * nblocks are active. Counter =3D nonce[12] || bswap32(tid+2). + * AES-encrypt the counter -> keystream. XOR with ciphertext -> + * plaintext. Store to out_addr + tid*16. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* VCC =3D (nblocks > tid) i.e. tid < nblocks - selects active CTR lanes + */ + _E(emit_gfx10_v_cmp_gt_u32, I10(buf, n), P_S(SR_NBLOCKS_GCM), + P_V(VR_TID)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_EXEC_SAVE, + 106 /* VCC_LO */); + br_execz_ctr =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + /* Save SR_KEYS for reload after this block encrypt */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_T_ADDR), P_S(SR_KEYS)); + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_T_ADDR + 1), + P_S(SR_KEYS + 1)); + + /* Prefetch ciphertext into v[23:27] before AES. + * GFX10 unaligned fix: ctext addr =3D=3D 2 mod 4, so load + * dwordx4 (clips to aligned) + extra dword at +16. + * The ~200+ cycle AES encrypt hides the VMEM latency. + * v23-v27 are not touched by AES rounds (which use + * v1-v10 only). VR_GA/VR_BLK are also AES-safe. + */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_BLK), P_I(4), + P_V(VR_TID)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_I(ESP_REL_CTEXT), P_V(VR_GA_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_BLK)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_PREFETCH0), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_PREFETCH4), + P_V(VR_GA_LO), 16); + + /* Build AES counter block in v[VR_S0:VR_S3]: + * VR_S0 =3D nonce[0:3] =3D salt (SR_IV0) + * VR_S1 =3D nonce[4:7] =3D IV[0:3] (SR_IV1) + * VR_S2 =3D nonce[8:11] =3D IV[4:7] (SR_IV2) + * VR_S3 =3D bswap32(tid + 2) + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S0), P_S(SR_IV0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S1), P_S(SR_IV1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S2), P_S(SR_IV2)); + + /* v[VR_S3] =3D bswap32(tid + 2) */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_S3), P_I(2), + P_V(VR_TID)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_S3), P_S(SR_BSWAP)); + + /* AES encrypt the counter block -> result in v[VR_S0:VR_S3] */ +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 7 + /* LEVEL 7: exit just before emit_aes_encrypt_block_gfx10. + * If this passes but Level 8 (after encrypt) faults, + * the AES block cipher GFX10 code is the culprit. + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0007u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(VR_SAVE_BD_LO), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 7 (before AES encrypt), %d byt= es\n", + n * 4); + return n * 4; +#endif + n =3D emit_aes_encrypt_block_gfx10(buf, n); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 8 + /* LEVEL 8: exit right after first emit_aes_encrypt_block_gfx10. + * If this faults, the illegal insn is inside the AES block cipher. + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0008u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(VR_SAVE_BD_LO), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 8 (after AES encrypt), %d byte= s\n", + n * 4); + return n * 4; +#endif + + /* Ciphertext arrived during AES - drain vmcnt */ + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* GFX10 unaligned fix: 4x alignbit on prefetched data */ + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_PREFETCH0), + P_V(VR_PREFETCH1), P_V(VR_PREFETCH0), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_PREFETCH1), + P_V(VR_PREFETCH2), P_V(VR_PREFETCH1), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_PREFETCH2), + P_V(VR_PREFETCH3), P_V(VR_PREFETCH2), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_PREFETCH3), + P_V(VR_PREFETCH4), P_V(VR_PREFETCH3), P_I(16)); + + /* XOR keystream with ciphertext -> plaintext */ + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_PREFETCH0), + P_V(VR_S0), P_V(VR_PREFETCH0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_PREFETCH1), + P_V(VR_S1), P_V(VR_PREFETCH1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_PREFETCH2), + P_V(VR_S2), P_V(VR_PREFETCH2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_PREFETCH3), + P_V(VR_S3), P_V(VR_PREFETCH3)); + + /* Store plaintext to out_addr + tid*16 */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_BLK), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_OUT_HI)); + _E(emit_gfx10_global_store_dwordx4, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_PREFETCH0), 0); + + patch_branch(buf, br_execz_ctr, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 7.5: Compute AES(K, J0) for ICV finalization + * + * J0 =3D nonce[12] || 0x00000001 (BE). Only thread 0 needs this + * but all active lanes can compute it; we just save the result. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Restore SR_KEYS (consumed by encrypt_block) */ + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_KEYS), P_S(SR_T_ADDR)); + _E(emit_gfx10_s_mov_b32, I10(buf, n), P_S(SR_KEYS + 1), + P_S(SR_T_ADDR + 1)); + + /* Restore full EXEC for J0 encrypt (all 256 threads) */ + _E(emit_gfx10_s_or_b64, I10(buf, n), 126 /* EXEC_LO */, SR_EXEC_SAVE, + SR_EXEC_SAVE); + + /* J0 block: nonce || bswap32(1) =3D nonce || 0x01000000 */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S0), P_S(SR_IV0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S1), P_S(SR_IV1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S2), P_S(SR_IV2)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_S3), P_L(0x01000000u)); + + n =3D emit_aes_encrypt_block_gfx10(buf, n); + + /* Save AES(K, J0) -> v[VR_J0_0:VR_J0_3] */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_J0_0), P_V(VR_S0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_J0_1), P_V(VR_S1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_J0_2), P_V(VR_S2)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_J0_3), P_V(VR_S3)); + + _E(emit_gfx10_s_barrier, I10(buf, n)); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 9 + /* LEVEL 9: exit after Phase 7.5 (J0 encrypt + barrier). + * Tests Phase 7 ctext XOR+store, Phase 7.5 second AES encrypt, + * s_or_b64 EXEC restore, s_barrier. + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), + P_L(0xDEAD0009u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(VR_SAVE_BD_LO), P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + pr_info("knod_ipsec: GFX10 DIAG STUB level 9 (Phase 7.5 + exit), %d bytes= \n", + n * 4); + return n * 4; +#endif + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 8: Parallel GHASH + * + * GHASH input blocks (total_blocks =3D nblocks + 2): + * tid 0 -> AAD: SPI(4B,BE)||seq(4B,BE)||0s (16B) + * tid 1..nblocks -> ciphertext block (tid-1) + * tid nblocks+1 -> len: AAD_bitlen(64b)||ctext_bitlen(64b) + * tid > nblocks+1 -> zero (does not participate) + * + * Each thread loads its block -> v[VR_DATA0:VR_DATA3] (big-endian + * for GF multiply), loads H^(total-tid) -> v[VR_D0:VR_D3], runs + * GF multiply -> v[VR_S0:VR_S3], then tree-reduces via LDS XOR. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Prefetch H^(total-tid) from H-power table before data selection. + * The ~60 ALU instructions in the data selection block below + * cover the VMEM latency. Result lands in VR_D0:D3, which + * data selection does not touch. VR_TMP/VR_GA are consumed + * here then free for reuse by the ctext section. + */ + _E(emit_gfx10_v_sub_nc_u32, I10(buf, n), P_V(VR_TMP), + P_S(SR_TOTAL_GHASH_BLK), P_V(VR_TID)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFFF), P_V(VR_TMP)); + _E(emit_gfx10_v_max_i32, I10(buf, n), P_V(VR_TMP), P_I(0), P_V(VR_TMP)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_TMP), P_I(4), + P_V(VR_TMP)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_LO), + P_S(SR_HTABLE_LO)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_HI), + P_S(SR_HTABLE_HI)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_TMP)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_D0), + P_V(VR_GA_LO), 0); + + /* EXEC-based per-lane data selection. Default =3D zero, then + * each case narrows EXEC to matching lanes and writes data. + * This avoids scalar VCC branching (s_cbranch_vccnz) which + * makes the entire wave take one path, not individual lanes. + */ + /* Default: all threads get zero (non-participating) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA2), P_I(0)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA3), P_I(0)); + + /* ---- AAD: tid =3D=3D 0 ---- */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_aad =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + /* VR_SAVE_SPI/SEQ are already in BE register convention: + * raw LE load from packet (BE wire bytes) + bswap =3D byte[0] + * in bits[31:24]. No second bswap needed - use directly. + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_SAVE_SPI)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SEQ)); + /* DATA2, DATA3 already 0 */ + + patch_branch(buf, br_skip_aad, n); + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 12 + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip12 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Cu)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip12, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* ---- Ctext: 1 <=3D tid <=3D nblocks ---- */ + /* block_idx =3D tid - 1 (unsigned; tid=3D=3D0 -> 0xFFFFFFFF > nblocks) */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP), P_L(0xFFFFFFFF), + P_V(VR_TID)); + _E(emit_gfx10_v_cmp_gt_u32, I10(buf, n), P_S(SR_NBLOCKS_GCM), + P_V(VR_TMP)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_ctext =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 14 + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip14 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Eu)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip14, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* Load ctext block: pkt + esp_hdr_off + 16 + block_idx*16 */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_TMP), P_I(4), + P_V(VR_TMP)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_I(ESP_REL_CTEXT), P_V(VR_GA_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_TMP)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + /* GFX10 unaligned load fix: ctext addr =3D=3D 2 mod 4. + * dwordx4 + extra dword + 4x alignbit. VR_BLK is free. + */ + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_BLK), + P_V(VR_GA_LO), 16); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA1), P_V(VR_DATA0), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_DATA2), P_V(VR_DATA1), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA2), + P_V(VR_DATA3), P_V(VR_DATA2), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA3), + P_V(VR_BLK), P_V(VR_DATA3), P_I(16)); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 15 + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip15 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Fu)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip15, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* Zero trailing dwords in the last partial ctext block. + * The load above reads 16 raw bytes, but for the last block + * only (ctext_len % 16) bytes are ciphertext - the rest are + * ICV bytes which must NOT enter GHASH. ESP ctext is always + * 4-byte aligned so the partial count is 4, 8 or 12 - pure + * dword-level zeroing suffices, no byte masking needed. + * + * VR_TMP still holds block_idx * 16 from the address calc. + * remaining =3D ctext_len - block_idx*16. For full blocks + * (remaining >=3D 16) every v_cmp evaluates true -> no change. + */ + _E(emit_gfx10_v_sub_nc_u32, I10(buf, n), P_V(VR_TMP), + P_S(SR_CTEXT_LEN), P_V(VR_TMP)); + /* VR_TMP =3D remaining bytes in this block */ + + /* DATA3 (bytes 12-15): keep only if remaining > 12 */ + _E(emit_gfx10_v_cmp_lt_u32, I10(buf, n), P_I(12), P_V(VR_TMP)); +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 18 + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip18 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD0012u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip18, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), + P_V(VR_DATA3), P_I(0), P_V(VR_DATA3)); + + /* DATA2 (bytes 8-11): keep only if remaining > 8 */ + _E(emit_gfx10_v_cmp_lt_u32, I10(buf, n), P_I(8), P_V(VR_TMP)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), + P_V(VR_DATA2), P_I(0), P_V(VR_DATA2)); + + /* DATA1 (bytes 4-7): keep only if remaining > 4 */ + _E(emit_gfx10_v_cmp_lt_u32, I10(buf, n), P_I(4), P_V(VR_TMP)); + _E(emit_gfx10_v_cndmask_b32_e32, I10(buf, n), + P_V(VR_DATA1), P_I(0), P_V(VR_DATA1)); + + /* DATA0 (bytes 0-3): always valid (ESP 4-byte alignment) */ + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 16 + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip16 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD0010u)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip16, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* bswap each dword for GHASH (big-endian GF arithmetic) */ + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA0), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_DATA1), P_V(VR_DATA1), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_DATA2), + P_V(VR_DATA2), P_V(VR_DATA2), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_DATA3), + P_V(VR_DATA3), P_V(VR_DATA3), P_S(SR_BSWAP)); + + patch_branch(buf, br_skip_ctext, n); + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 13 + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip13 =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Du)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip13, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* ---- Len block: tid =3D=3D nblocks + 1 ---- */ + /* Compute nblocks+1 in s42 (scratch) */ + _E(emit_gfx10_s_add_u32, I10(buf, n), P_S(42), P_I(1), + P_S(SR_NBLOCKS_GCM)); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_S(42), + P_V(VR_TID)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_len =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + /* Length block format (GCM big-endian): + * DATA0 =3D AAD_bits[63:32] =3D 0 + * DATA1 =3D AAD_bits[31:0] =3D 64 (8 bytes AAD x 8) + * DATA2 =3D ctext_bits[63:32] =3D 0 + * DATA3 =3D ctext_bits[31:0] =3D ctext_len * 8 + * + * No bswap32: length values are computed integers already + * in the correct big-endian register representation. + * bswap32 is only needed for data loaded from LE memory. + */ + /* DATA0, DATA2 already 0 from default init */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), + P_L(0x00000040u)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA3), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_DATA3), P_I(3), + P_V(VR_DATA3)); + + patch_branch(buf, br_skip_len, n); + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 11 + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Bu)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + /* H-table data was prefetched before data selection; drain + bswap */ + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_D0), + P_V(VR_D0), P_V(VR_D0), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_D1), + P_V(VR_D1), P_V(VR_D1), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_D2), + P_V(VR_D2), P_V(VR_D2), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_D3), + P_V(VR_D3), P_V(VR_D3), P_S(SR_BSWAP)); + + /* ---- GF(2^128) multiply: Z =3D DATA * H^k ---- */ +#if defined(KNOD_IPSEC_GFX10_DIAG_STUB) && KNOD_IPSEC_GFX10_DIAG_STUB =3D= =3D 10 + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(0)); + br_skip =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_L(0xDEAD000Au)); + _E(emit_gfx10_global_store_dword, I10(buf, n), P_V(VR_SAVE_BD_LO), + P_V(7), 8); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + patch_branch(buf, br_skip, n); + _E(emit_gfx10_s_endpgm, I10(buf, n)); + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + return n * 4; +#endif + n =3D emit_gfmul_128_gfx10(buf, n); + /* Result in v[VR_S0:VR_S3] */ + + /* ---- Tree reduction via LDS XOR (8 levels for 256 threads) ---- */ + /* Write v[VR_S0:VR_S3] to LDS at tid * 16 */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_ADDR), P_I(4), + P_V(VR_TID)); + _E(emit_gfx10_ds_write_b128, I10(buf, n), VR_ADDR, VR_S0); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + _E(emit_gfx10_s_barrier, I10(buf, n)); + + for (level =3D 1; level <=3D 128; level <<=3D 1) { + /* if (tid & level) skip */ + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(VR_TMP), + P_L(level), P_V(VR_TID)); + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(VR_TMP)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + /* Compute both addresses up front */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP), + P_L(level), P_V(VR_TID)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_TMP), + P_I(4), P_V(VR_TMP)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_ADDR), + P_I(4), P_V(VR_TID)); + + /* Issue both reads, single wait */ + _E(emit_gfx10_ds_read_b128, I10(buf, n), VR_D0, VR_TMP); + _E(emit_gfx10_ds_read_b128, I10(buf, n), VR_S0, VR_ADDR); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + + /* XOR */ + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S0), + P_V(VR_S0), P_V(VR_D0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S1), + P_V(VR_S1), P_V(VR_D1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S2), + P_V(VR_S2), P_V(VR_D2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S3), + P_V(VR_S3), P_V(VR_D3)); + + /* Write back */ + _E(emit_gfx10_ds_write_b128, I10(buf, n), VR_ADDR, VR_S0); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + + patch_branch(buf, br_skip, n); + /* Restore EXEC */ + _E(emit_gfx10_s_mov_b64, I10(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx10_s_barrier, I10(buf, n)); + } + + /* Thread 0 now has the final GHASH in LDS[0..15]. Read it. */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_TMP), P_I(0)); + _E(emit_gfx10_ds_read_b128, I10(buf, n), VR_S0, VR_TMP); + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 9: ICV verify (thread 0 only) + * + * computed_tag =3D bswap(GHASH) XOR AES(K, J0) + * received_tag =3D last 16 bytes of ESP packet + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_EXEC_SAVE, 106); + br_tid0 =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + /* bswap GHASH from big-endian to little-endian */ + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_S0), P_V(VR_S0), + P_V(VR_S0), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_S1), P_V(VR_S1), + P_V(VR_S1), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_S2), P_V(VR_S2), + P_V(VR_S2), P_S(SR_BSWAP)); + _E(emit_gfx10_v_perm_b32, I10(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_S3), P_S(SR_BSWAP)); + + /* computed_tag =3D GHASH XOR AES(K, J0) */ + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S0), + P_V(VR_S0), P_V(VR_J0_0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S1), + P_V(VR_S1), P_V(VR_J0_1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S2), + P_V(VR_S2), P_V(VR_J0_2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_S3), + P_V(VR_S3), P_V(VR_J0_3)); + + /* Load received ICV: pkt + pkt_len - 16. + * GFX10 unaligned load fix: ICV addr =3D=3D 2 mod 4. + * dwordx4 + extra dword + 4x alignbit. VR_TMP is + * free after address calc; use VR_BLK for 5th dword. + */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFF0u), P_V(VR_SAVE_PKTLEN)); /* pkt_len - 16 */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_BLK), + P_V(VR_GA_LO), 16); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA1), P_V(VR_DATA0), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_DATA2), P_V(VR_DATA1), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA2), + P_V(VR_DATA3), P_V(VR_DATA2), P_I(16)); + _E(emit_gfx10_v_alignbit_b32, I10(buf, n), P_V(VR_DATA3), + P_V(VR_BLK), P_V(VR_DATA3), P_I(16)); + + /* Compare: XOR each dword, OR together; if any non-zero -> fail */ + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_S0)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_DATA1), P_V(VR_S1)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_DATA2), + P_V(VR_DATA2), P_V(VR_S2)); + _E(emit_gfx10_v_xor_b32_e32, I10(buf, n), P_V(VR_DATA3), + P_V(VR_DATA3), P_V(VR_S3)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA1)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA2)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA3)); + + /* If VR_DATA0 !=3D 0 -> ICV fail: overwrite verdict with sentinel */ + _E(emit_gfx10_v_cmp_ne_u32, I10(buf, n), P_I(0), P_V(VR_DATA0)); + br_icv_ok =3D _BR(emit_gfx10_s_cbranch_vccz, I10(buf, n), 0); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_SAVE_SLOT), + P_L(VERDICT_ICV_FAIL)); + patch_branch(buf, br_icv_ok, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 10: ESP trailer strip + write verdict (thread 0) + * + * Decrypted tail: pad_len at out + ctext_len - 2 + * next_hdr at out + ctext_len - 1 + * inner_len =3D ctext_len - pad_len - 2 + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Only strip if ICV passed (slot < NR_SA) */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 27, VR_SAVE_SLOT); + _E(emit_gfx10_s_cmp_ge_u32, I10(buf, n), P_S(27), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_icv_bad =3D _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0); + + /* Load last 4 bytes of decrypted payload: out + ctext_len - 4. + * VOP2 src1 must be VGPR, so move SGPR to VR_TMP first. + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_TMP), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFFC), P_V(VR_TMP)); /* ctext_len - 4 */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_OUT_HI)); + _E(emit_gfx10_global_load_dword, I10(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* On LE: loaded dword has byte layout [b0,b1,b2,b3]. + * We loaded from (ctext_len - 4), so: + * b2 =3D pad_len (at ctext_len - 2) + * b3 =3D next_hdr (at ctext_len - 1) + * pad_len =3D (dword >> 16) & 0xFF + */ + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(VR_TMP), + P_I(16), P_V(VR_DATA0)); + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), P_V(VR_TMP), + P_L(0xFF), P_V(VR_TMP)); + + /* inner_len =3D ctext_len - pad_len - 2 */ + _E(emit_gfx10_v_sub_nc_u32, I10(buf, n), P_V(VR_DATA1), + P_S(SR_CTEXT_LEN), P_V(VR_TMP)); + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_DATA1), + P_L(0xFFFFFFFE), P_V(VR_DATA1)); /* -2 */ + + /* Write bd->len =3D inner_len (u16 at bd + 18) */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_L(18), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx10_global_store_short, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA1), 0); + + /* Write bd->off =3D mode | (next_hdr << 8) (u16 at bd + 16). + * next_hdr =3D byte[3] of the ESP trailer dword (VR_DATA0). + * mode from s[SR_SA_MODE]. + */ + _E(emit_gfx10_v_lshrrev_b32, I10(buf, n), P_V(VR_TMP), + P_I(24), P_V(VR_DATA0)); + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), P_V(VR_TMP), + P_I(8), P_V(VR_TMP)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA2), + P_S(SR_SA_MODE)); + _E(emit_gfx10_v_or_b32_e32, I10(buf, n), P_V(VR_TMP), + P_V(VR_DATA2), P_V(VR_TMP)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_L(16), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx10_global_store_short, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), 0); + + /* Per-SA GPU stats: atomically increment rx_packets + * and rx_bytes at stats_addr. VR_DATA1 still holds + * inner_len from the bd->len computation above. + * + * global_atomic_add_x2 uses v[data:data+1] as u64. + * Save inner_len to VR_TMP before clobbering DATA1. + * + * stats layout (knod_ipsec_sa_gpu_stats): + * +0: rx_packets (u64, LE) + * +8: rx_bytes (u64, LE) + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_TMP), + P_V(VR_DATA1)); /* save inner_len */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_STATS_LO)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_GA_HI), + P_V(VR_SAVE_STATS_HI)); + + /* rx_packets +=3D 1: v[DATA0:DATA1] =3D {1, 0} */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), P_I(1)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), P_I(0)); + _E(emit_gfx10_global_atomic_add_x2, I10(buf, n), + P_V(VR_DATA2), P_V(VR_GA_LO), P_V(VR_DATA0), 0, 0); + + /* rx_bytes +=3D inner_len: v[DATA0:DATA1] =3D {inner_len, 0} */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_V(VR_TMP)); + /* DATA1 already 0 from above */ + _E(emit_gfx10_global_atomic_add_x2, I10(buf, n), + P_V(VR_DATA2), P_V(VR_GA_LO), P_V(VR_DATA0), 8, 0); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * L3 header passthrough (transport mode only). + * + * Copy 20 B of the outer IPv4 L3 header from + * pkt + 14 (skip ETH) to out_addr - 20. In shader-GTT + * direct mode (knod_ipsec_sdma=3D0), out_addr - 20 is + * pass_buf_slot + 0 so the host-side finalise can skip + * the per-packet L3 SDMA copy entirely - the only + * remaining SDMA call on the transport IPv4 fast path. + * + * Tunnel-mode SAs (SR_SA_MODE !=3D 0) skip this write + * because for tunnel the destination at slot+0 wants + * the *inner* packet, not an outer IP header. + * + * The VRAM staging path (knod_ipsec_sdma=3D1) also runs + * this copy, but the destination is the 20-byte + * headroom knod_ipsec.c reserves at the front of + * the decrypt pool; CPU finalise still SDMAs the real + * L3 header from the raw packet into the GTT pass_buf + * so the shader's write is harmless wasted work. + * + * Alignment: pkt + 14 is only 2-byte aligned (ETH hdr + * =3D 14 bytes !=3D 4-byte multiple), so dword / dwordx4 + * loads would fault. Use 10 x global_load_ushort at + * offsets 14,16,...,32, paired with 10 x store_short + * at slot + 0,2,...,18. 10 scratch VGPRs (v14..v23), + * all free by Phase 10 since AES-GCM / GHASH state is + * done. One waitcnt between loads and stores. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), + P_S(SR_SA_MODE), P_I(0)); + br_not_transport =3D _BR(emit_gfx10_s_cbranch_scc0, + I10(buf, n), 0); + + /* src =3D pkt_addr + 14 */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), + P_V(VR_GA_LO), P_I(14), + P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), + P_V(VR_GA_HI), P_I(0), + P_V(VR_SAVE_PKT_HI)); + + /* 10 x 2-byte loads from src+0..+18 */ + for (li =3D 0; li < 10; li++) { + _E(emit_gfx10_global_load_ushort, + I10(buf, n), + P_V(L3_TMP_BASE + li), + P_V(VR_GA_LO), li * 2); + } + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* dst =3D out_addr - 20. + * + * Two GFX9 landmines here: + * + * 1. v_add_co_u32 can't take a 32-bit + * literal src together with implicit VCC + * - same class as the v_cndmask literal + * restriction. + * 2. P_I(n) is a raw initializer that always + * sets type=3DINTEGER_0 and stores n in .v. + * For negative inline constants the + * encoder must flip to INTEGER_MINUS_1 + * with v=3D~n, which P_I does NOT do. + * P_I(-1) therefore encodes as + * GFX9_SRC_INTEGER_0 + (-1) =3D 127, a + * bogus register that gave us random + * high-32 bits and page-faulted stores. + * + * Dodge both by materialising -20 into VR_TMP + * and -1 into VR_TMP2 via v_mov_b32 literals + * (VOP1, no VCC, literals fine), then pure + * VGPR+VGPR add_co / addc_co. Borrow flows + * through VCC as the carry-in to addc_co. + */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), + P_V(VR_TMP), P_L(0xFFFFFFECu)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), + P_V(VR_TMP2), P_L(0xFFFFFFFFu)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), + P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), + P_V(VR_GA_HI), + P_V(VR_TMP2), P_V(VR_SAVE_OUT_HI)); + + /* 10 x 2-byte stores to dst+0..+18 */ + for (li =3D 0; li < 10; li++) { + _E(emit_gfx10_global_store_short, + I10(buf, n), + P_V(VR_GA_LO), + P_V(L3_TMP_BASE + li), + li * 2); + } + + patch_branch(buf, br_not_transport, n); + + patch_branch(buf, br_icv_bad, n); + + /* Write bd->act (u64 at bd + 8): + * high32 =3D slot_idx (or sentinel), + * low32 =3D KNOD_IPSEC_INFLIGHT so NIC NAPI recognises + * this slot as in-flight until the finish worker stamps + * the final PASS/DROP. + */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_L(8), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_L(KNOD_IPSEC_INFLIGHT)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SLOT)); + _E(emit_gfx10_global_store_dwordx2, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA0), 0); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + patch_branch(buf, br_tid0, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 11: Miss/bypass path verdict - thread 0 only + * + * If we skipped crypto (Phase 2 branch), write the miss/bypass + * sentinel that's still in v[VR_SAVE_SLOT]. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + br_crypto_done =3D _BR(emit_gfx10_s_branch, I10(buf, n), 0); + + patch_branch(buf, br_crypto_end, n); + + /* Thread 0 writes bd->act with miss/bypass sentinel */ + _E(emit_gfx10_v_cmp_eq_u32, I10(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx10_s_and_saveexec_b64, I10(buf, n), SR_EXEC_SAVE, 106); + br_execz2 =3D _BR(emit_gfx10_s_cbranch_execz, I10(buf, n), 0); + + _E(emit_gfx10_v_add_co_u32, I10(buf, n), P_V(VR_GA_LO), + P_L(8), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0), + P_L(KNOD_IPSEC_INFLIGHT)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SLOT)); + _E(emit_gfx10_global_store_dwordx2, I10(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA0), 0); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + patch_branch(buf, br_execz2, n); + + patch_branch(buf, br_crypto_done, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 12: GPU-initiated SDMA dispatch (thread 0 only) + * + * Identical logic to GFX9 Phase 12. See ipsec_fused_gfx9.h + * for the full protocol description. + * + * GPU writes SDMA COPY_LINEAR packets only. FENCE, wptr update, + * and doorbell are left to the CPU. + * + * GFX10 differences: + * - v_add_nc_u32 / v_sub_nc_u32 (no-carry variants) + * - v_add_co_ci_u32_e32 for carry-in addition + * - s_or_b64 for 64-bit zero test (no s_or_b32 helper) + * - GFX10 global offset is 12-bit signed (all offsets <=3D56 OK) + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* --- kernarg loads ---------------------------------------- */ + _E(emit_gfx10_s_load_dwordx2, I10(buf, n), + P_S(24), P_S(8), 16); /* s[24:25] =3D sdma_ring_addr */ + _E(emit_gfx10_s_load_dwordx2, I10(buf, n), + P_S(26), P_S(8), 32); /* s[26:27] =3D sdma_ctl_addr */ + _E(emit_gfx10_s_waitcnt_lgkmcnt, I10(buf, n)); + + /* s_or_b64 sets SCC =3D (s[26:27] !=3D 0) */ + _E(emit_gfx10_s_or_b64, I10(buf, n), 28, 26, 26); + br_no_sdma =3D _BR(emit_gfx10_s_cbranch_scc0, I10(buf, n), 0); + + /* --- load sdma_ctl into VGPRs ----------------------------- */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(1), P_S(26)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(2), P_S(27)); + + /* ctl+28: wptr_base_dw(4) ring_mask(4) nr_total_wg(4) copy_hdr(4) + * -> v[3:6] + */ + _E(emit_gfx10_global_load_dwordx4, I10(buf, n), + P_V(3), P_V(1), 28); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* v3=3Dwptr_base_dw v4=3Dring_mask v5=3Dnr_total_wg v6=3Dcopy_hdr */ + + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), + 28, 5); /* s28 =3D nr_total_wg */ + + /* --- verdict check ---------------------------------------- */ + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), + 29, VR_SAVE_SLOT); /* s29 =3D verdict */ + _E(emit_gfx10_s_cmp_ge_u32, I10(buf, n), + P_S(29), P_L(0xFFFFFFFEu)); /* MISS|BYPASS? */ + br_no_copy =3D _BR(emit_gfx10_s_cbranch_scc0, I10(buf, n), 0); + + /* =3D=3D=3D This WG needs SDMA copy =3D=3D=3D */ + + /* atomic_add(&ctl->claim_counter, 1, GLC=3D1) -> my_idx */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_I(1)); + _E(emit_gfx10_global_atomic_add, I10(buf, n), + P_V(7), P_V(1), P_V(7), 0, 1); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + /* v7 =3D my_idx (old claim_counter) */ + + /* ring dword position: (wptr_base_dw + my_idx*7) & ring_mask */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), + P_V(8), P_I(3), P_V(7)); /* my_idx * 8 */ + _E(emit_gfx10_v_sub_nc_u32, I10(buf, n), + P_V(8), P_V(8), P_V(7)); /* my_idx * 7 */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), + P_V(8), P_V(3), P_V(8)); /* + wptr_base_dw */ + _E(emit_gfx10_v_and_b32_e32, I10(buf, n), + P_V(8), P_V(4), P_V(8)); /* & ring_mask */ + _E(emit_gfx10_v_lshlrev_b32, I10(buf, n), + P_V(8), P_I(2), P_V(8)); /* * 4 -> byte offset */ + + /* v[9:10] =3D sdma_ring_addr + byte_offset */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(14), P_S(25)); + _E(emit_gfx10_v_add_co_u32, I10(buf, n), + P_V(9), P_S(24), P_V(8)); + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), + P_V(10), P_I(0), P_V(14)); + + /* DW0: copy_hdr */ + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(6), 0); + + /* DW1: nbytes - 1 */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), + P_V(14), P_L(0xFFFFFFFFu), P_V(VR_SAVE_PKTLEN)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(14), 4); + + /* DW2: 0 (sub-op parameter) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(14), P_I(0)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(14), 8); + + /* DW3-4: src =3D pkt_addr (VRAM) */ + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(VR_SAVE_PKT_LO), 12); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(VR_SAVE_PKT_HI), 16); + + /* DW5-6: dst =3D out_addr - 20 (GTT slot start) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), + P_V(14), P_L(0xFFFFFFECu)); /* -20 */ + _E(emit_gfx10_v_add_co_u32, I10(buf, n), + P_V(15), P_V(14), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), + P_V(14), P_L(0xFFFFFFFFu)); /* -1 */ + _E(emit_gfx10_v_add_co_ci_u32_e32, I10(buf, n), + P_V(16), P_V(14), P_V(VR_SAVE_OUT_HI)); + + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(15), 20); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(9), P_V(16), 24); + + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* --- done counter (all WGs) ------------------------------- */ + patch_branch(buf, br_no_copy, n); + + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_I(1)); + _E(emit_gfx10_global_atomic_add, I10(buf, n), + P_V(7), P_V(1), P_V(7), 4, 1); /* ctl+4 =3D done_counter */ + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* Last WG check: my_done + 1 =3D=3D nr_total_wg? */ + _E(emit_gfx10_v_add_nc_u32, I10(buf, n), + P_V(7), P_I(1), P_V(7)); + _E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), 29, 7); + _E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(29), P_S(28)); + br_not_last =3D _BR(emit_gfx10_s_cbranch_scc0, I10(buf, n), 0); + + /* =3D=3D=3D Last WG - publish counters for CPU =3D=3D=3D */ + + /* Read final claim_counter (atomic add 0, GLC=3D1) */ + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_I(0)); + _E(emit_gfx10_global_atomic_add, I10(buf, n), + P_V(7), P_V(1), P_V(7), 0, 1); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + /* Store final_sdma_count (ctl+52) and gpu_sdma_ready (ctl+48) */ + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(1), P_V(7), 52); + _E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(7), P_I(1)); + _E(emit_gfx10_global_store_dword, I10(buf, n), + P_V(1), P_V(7), 48); + _E(emit_gfx10_s_waitcnt_vmcnt, I10(buf, n)); + + patch_branch(buf, br_not_last, n); + patch_branch(buf, br_no_sdma, n); + + _E(emit_gfx10_s_endpgm, I10(buf, n)); + + /* GFX10 RDNA2 prefetches instructions aggressively past s_endpgm. + * Without an s_code_end cushion the SQC fetches garbage beyond the + * shader and raises an SQC(inst) page fault at a seemingly random + * address. Pad to a 256-dword boundary - same pattern the BPF GFX10 + * emitter (knod_bpf.c) uses on working shaders. + */ + while (n % 256) + _E(emit_gfx10_s_code_end, I10(buf, n)); + + return n * 4; +} + +#endif /* KNOD_HELPERS_IPSEC_FUSED_GFX10_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h b/drivers/g= pu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h new file mode 100644 index 000000000000..5121ece1f2eb --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h @@ -0,0 +1,1349 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +/* + * KNOD IPsec fused RX shader - GFX9 (Vega10/20). + * + * Full AES-GCM decrypt pipeline for inbound ESP packets: + * 1) ESP header parse -> SPI + seq extract + * 2) SA table linear scan -> resolve SPI to slot index + * 3) Cooperative T-table load (VRAM -> LDS, 256 threads) + * 4) AES-CTR decrypt ciphertext -> out_addr + * 5) Parallel GHASH over (AAD || ciphertext || len) + * 6) ICV verify (GHASH ^ AES(K,J0) vs received tag) + * 7) ESP trailer strip -> inner_len + * 8) Write verdict to bd->act, inner_len to bd->len + * + * Dispatch geometry: + * workgroup =3D (256, 1, 1) - 256 threads =3D 1 AES block per thre= ad + * grid =3D (256, nr_pkts, 1) + * workgroup_id_y =3D=3D packet index in the batch + * + * Anti-replay is NOT in the shader - CPU-side sliding window in NIC NAPI. + * + * Verdict encoding in bd->act high32: + * 0..NR_SA-1 - SA hit + ICV pass, value is slot_idx + * 0xFFFFFFFF - SA miss (no entry for this SPI) + * 0xFFFFFFFE - non-IPv4/IPv6 bypass (unknown L3 protocol) + * 0xFFFFFFFD - ICV mismatch (decrypt succeeded but tag wrong) + * + * bd->len is set to inner_len on success (decrypted payload minus ESP + * trailer and padding). On miss/bypass/ICV-fail, bd->len is left as-is. + */ + +#ifndef KNOD_HELPERS_IPSEC_FUSED_GFX9_H_ +#define KNOD_HELPERS_IPSEC_FUSED_GFX9_H_ + +#include +#include "knod_amdgpu_insn.h" +/* Provide AESGCM_MAX_DIM_Y so aesgcm_shader.h compiles (OFF_T0 macro). + * Only emit_aes_encrypt_block_gfx9 / emit_gfmul_128_gfx9 are used here; + * the full aesgcm_gen_shader_* functions are unreferenced. + */ +#ifndef AESGCM_MAX_DIM_Y +#define AESGCM_MAX_DIM_Y 1024 +#endif +#include "aesgcm_shader.h" + +/* SA entry constants - must match knod_ipsec.h */ +#define KNOD_IPSEC_SHADER_NR_SA 256 +#define KNOD_IPSEC_SHADER_SA_ENTRY_SZ 104 + +/* SA entry field offsets (struct knod_ipsec_sa_entry) */ +#define SA_OFF_SPI 0 +#define SA_OFF_KEY_ADDR 16 +#define SA_OFF_HTABLE_ADDR 24 +#define SA_OFF_T_TABLES_ADDR 32 +#define SA_OFF_SALT 40 +#define SA_OFF_KEY_LEN 44 +#define SA_OFF_NR_ROUNDS 48 +#define SA_OFF_MODE 52 /* XFRM_MODE_TRANSPORT=3D0, TUNNEL=3D1 */ +#define SA_OFF_STATS_ADDR 88 /* per-SA GPU stats (u64 gpu addr) */ + +/* ESP packet geometry (ETH=3D14, IPv4=3D20 / IPv6=3D40, no VLAN/opts). + * IPv4: ESP header starts at offset 34 (14+20). + * IPv6: ESP header starts at offset 54 (14+40). + * Within ESP header: SPI+0, seq+4, IV+8, ctext+16. + * The shader dynamically computes offsets based on IP version. + */ +#define ESP_HDR_OFF_V4 34 /* ETH(14) + IPv4(20) */ +#define ESP_HDR_OFF_V6 54 /* ETH(14) + IPv6(40) */ +#define ESP_REL_SPI 0 +#define ESP_REL_SEQ 4 +#define ESP_REL_IV 8 +#define ESP_REL_CTEXT 16 /* SPI(4)+seq(4)+IV(8) */ +#define ESP_ICV_LEN 16 + +/* Fixed IPv4 layout offsets used by the crypto KAT */ +#define ESP_SPI_OFF 34 +#define ESP_SEQ_OFF 38 +#define ESP_IV_OFF 42 +#define ESP_CTEXT_OFF 50 + +/* Fused sub[] offsets within kernarg (sub[i] =3D kernarg + 40 + i*32) */ +#define SUB_BASE_OFF 40 +#define SUB_STRIDE 32 +#define SUB_OFF_PKT_ADDR 0 +#define SUB_OFF_OUT_ADDR 8 +#define SUB_OFF_BD_ADDR 16 +#define SUB_OFF_PKT_LEN 24 +#define SUB_OFF_RESULT_SEQ 28 + +/* ICV-fail sentinel (distinct from MISS=3D0xFFFFFFFF and BYPASS=3D0xFFFFF= FFE) */ +#define VERDICT_ICV_FAIL 0xFFFFFFFDu + +/* High VGPRs for saving pre-crypto IPsec state (above AES v0-v22 range) */ +#define VR_SAVE_SLOT 30 +#define VR_SAVE_BD_LO 31 +#define VR_SAVE_BD_HI 32 +#define VR_SAVE_PKT_LO 33 +#define VR_SAVE_PKT_HI 34 +#define VR_SAVE_PKTLEN 35 +#define VR_SAVE_OUT_LO 36 +#define VR_SAVE_OUT_HI 37 +#define VR_SAVE_SEQ 38 +#define VR_SAVE_SPI 39 +#define VR_SAVE_STATS_LO 40 /* per-SA stats GPU addr low */ +#define VR_SAVE_STATS_HI 41 /* per-SA stats GPU addr high */ +/* ESP header offset: 34(v4) or 54(v6) */ +#define VR_SAVE_ESP_OFF 42 +/* Ciphertext prefetch destination - free v23-v26, inside AES v0-v22 gap */ +#define VR_PREFETCH0 23 +#define VR_PREFETCH1 24 +#define VR_PREFETCH2 25 +#define VR_PREFETCH3 26 + +/* Extra SGPRs for IPsec-specific state that survives into AES phases. + * These must NOT collide with SR_* from aesgcm_shader.h (s18-s49, s56-s59= ). + * s50-s55 are IPsec-specific. s56-s59 =3D SR_RK2 (AES round key double-bu= ffer). + */ +#define SR_CTEXT_LEN 50 /* ciphertext length in bytes */ +#define SR_NBLOCKS_GCM 51 /* ceil(ctext_len/16) */ +#define SR_HTABLE_LO 52 /* H-power table GPU addr */ +#define SR_HTABLE_HI 53 +#define SR_TOTAL_GHASH_BLK 54 /* nblocks + 2 (AAD + ctext + len) */ +#define SR_SA_MODE 55 /* XFRM_MODE_TRANSPORT=3D0, TUNNEL=3D1 */ + +/* File-local emit helpers */ +#ifndef _KNOD_IPSEC_EMIT +#define _KNOD_IPSEC_EMIT +#define _E(fn, ...) (n +=3D fn(__VA_ARGS__) / 4) +#define _BR(fn, ...) ({ int _p =3D n; n +=3D fn(__VA_ARGS__) / 4; _p; }) +#endif + +static inline int kfd_ipsec_gen_fused_shader_gfx9(void *vbuf) +{ + int br_skip_aad, br_skip_ctext, br_skip_len; + int loop_top, br_match, br_loop, br_end; + int br_no_sdma, br_no_copy, br_not_last; + int br_ipv4, br_bypass, br_crypto_end; + const int L3_TMP_BASE =3D 14; /* v14..v23 */ + int br_ipv6, br_v6_to_common; + int br_not_transport, li; + u32 *buf =3D (u32 *)vbuf; + int br_crypto_done; + int br_execz_ctr; + int br_execz2; + int br_icv_bad; + int br_icv_ok; + int br_tid0; + int level; + int n =3D 0; + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 0: Parse ESP header + SA table lookup + * + * s_dcache_inv: flush K$ so s_load reads fresh round keys. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_s_dcache_inv, I9(buf, n)); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + + /* Cache bswap32 selector in SR_BSWAP for reuse across all phases */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_BSWAP), P_L(0x00010203)); + + /* v1 =3D 40 + wg_id_y*32 =3D offset of sub[wg_id_y] within kernarg */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(1), P_S(16)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(1), P_I(5), P_V(1)); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(1), P_L(SUB_BASE_OFF), P_V(1)); + + /* v[3:4] =3D kernarg_ptr + v1 =3D &sub[wg_id_y] */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(2), P_S(9)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(3), P_S(8), P_V(1)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(4), P_I(0), P_V(2)); + + /* Load sub[].pkt_addr -> v[9:10], sub[].bd_addr -> v[5:6] */ + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(9), P_V(3), + SUB_OFF_PKT_ADDR); + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(5), P_V(3), + SUB_OFF_BD_ADDR); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Seed v[11:12] with pkt_addr as a safe default BEFORE the IP + * version branch. The bypass path (non-v4/v6 packets like ARP) + * unconditionally branches past the v[11:12] setup at line ~195 + * and later Phase 1 does a global_load at v[11:12]+ESP_REL_SEQ to + * read the ESP sequence number. Without this seed v[11:12] would + * hold uninitialised VGPR state (wave launch garbage), producing + * a fault at ~0x{random}_00000000. For valid v4/v6 packets the + * common path below overwrites v[11:12] with pkt+esp_hdr_off so + * this seed is harmless. + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(11), P_V(9)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(12), P_V(10)); + + /* IP version gate: load dword at pkt+12 to get first byte of L3 + * header (byte[14]). Extract version nibble -> s28. + */ + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(15), P_V(9), 12); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 28, 15); + _E(emit_gfx9_s_lshr_b32, I9(buf, n), P_S(28), P_S(28), P_I(20)); + _E(emit_gfx9_s_and_b32_p, I9(buf, n), P_S(28), P_I(0xF), P_S(28)); + + /* Check IPv4 (version=3D=3D4) */ + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), P_S(28), P_I(4)); + br_ipv4 =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + /* Check IPv6 (version=3D=3D6) */ + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), P_S(28), P_I(6)); + br_ipv6 =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + /* Bypass: neither IPv4 nor IPv6 */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(26), P_L(0xFFFFFFFEu)); + br_bypass =3D _BR(emit_gfx9_s_branch, I9(buf, n), 0); + + /* IPv6 landing: esp_hdr_off =3D 54 */ + patch_branch(buf, br_ipv6, n); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_ESP_OFF), + P_L(ESP_HDR_OFF_V6)); + br_v6_to_common =3D _BR(emit_gfx9_s_branch, I9(buf, n), 0); + + /* IPv4 landing: esp_hdr_off =3D 34 */ + patch_branch(buf, br_ipv4, n); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_ESP_OFF), + P_L(ESP_HDR_OFF_V4)); + + /* Common path: both IPv4 and IPv6 converge here */ + patch_branch(buf, br_v6_to_common, n); + + /* v[11:12] =3D pkt_addr + esp_hdr_off (dynamic) */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(11), + P_V(VR_SAVE_ESP_OFF), P_V(9)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(12), P_I(0), P_V(10)); + + /* v13 =3D *(u32*)(pkt + esp_hdr_off) - SPI in big-endian */ + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(13), P_V(11), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Byteswap SPI: v13 =3D bswap32(v13) via v_perm_b32 */ + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(13), P_V(13), P_V(13), + P_S(SR_BSWAP)); + + /* SA table linear scan (VMEM path for K$ coherence). + * s22 =3D target SPI, s[24:25] =3D sa_table_addr, s23 =3D counter, + * s26 =3D result (slot_idx or 0xFFFFFFFF), v[16:17] =3D running ptr. + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(16), P_S(8)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(17), P_S(9)); + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(18), P_V(16), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 24, 18); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 25, 19); + + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 22, 13); + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(23), P_I(0)); + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(26), P_L(0xFFFFFFFFu)); + + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(16), P_S(24)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(17), P_S(25)); + + loop_top =3D n; + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(20), P_V(16), + SA_OFF_SPI); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 27, 20); + + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(16), + P_L(KNOD_IPSEC_SHADER_SA_ENTRY_SZ), P_V(16)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(17), P_I(0), P_V(17)); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(23), P_I(1), P_S(23)); + _E(emit_gfx9_s_nop, I9(buf, n)); + + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), P_S(27), P_S(22)); + br_match =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + _E(emit_gfx9_s_cmp_lt_u32, I9(buf, n), P_S(23), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_loop =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + patch_branch(buf, br_loop, loop_top); + + br_end =3D _BR(emit_gfx9_s_branch, I9(buf, n), 0); + + /* Match: s26 =3D s23 - 1 */ + patch_branch(buf, br_match, n); + _E(emit_gfx9_s_sub_u32_p, I9(buf, n), P_S(26), P_S(23), P_I(1)); + + patch_branch(buf, br_end, n); + + patch_branch(buf, br_bypass, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 1: Save pre-crypto state + load extra sub[] fields + * + * Move IPsec-specific values to v30+ so v1-v22 and s18-s49 are + * free for AES-GCM helpers from aesgcm_shader.h. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_SLOT), P_S(26)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_BD_LO), P_V(5)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_BD_HI), P_V(6)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_PKT_LO), P_V(9)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_PKT_HI), P_V(10)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_SPI), P_V(13)); + + /* Load sub[].out_addr -> v[VR_SAVE_OUT_LO:VR_SAVE_OUT_HI] */ + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(VR_SAVE_OUT_LO), + P_V(3), SUB_OFF_OUT_ADDR); + /* Load sub[].pkt_len -> v[VR_SAVE_PKTLEN] */ + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_SAVE_PKTLEN), + P_V(3), SUB_OFF_PKT_LEN); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Load ESP seq number: pkt + esp_hdr_off + 4, BE -> bswap -> + * v[VR_SAVE_SEQ]. + * v[11:12] still holds pkt_addr + esp_hdr_off from Phase 0. + */ + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_SAVE_SEQ), P_V(11), + ESP_REL_SEQ); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_SAVE_SEQ), + P_V(VR_SAVE_SEQ), P_V(VR_SAVE_SEQ), P_S(SR_BSWAP)); + + /* Write bswapped seq back into sub[].result_seq for CPU finish worker. + * v[3:4] still points to &sub[wg_id_y]. + */ + _E(emit_gfx9_global_store_dword, I9(buf, n), P_V(3), + P_V(VR_SAVE_SEQ), SUB_OFF_RESULT_SEQ); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 2: Branch on miss/bypass - skip crypto entirely + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 26, VR_SAVE_SLOT); + _E(emit_gfx9_s_cmp_ge_u32, I9(buf, n), P_S(26), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_crypto_end =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 3: Load SA entry fields for the matched slot + * + * entry_addr =3D sa_table_addr + slot_idx * SA_ENTRY_SIZE + * Load: key_gpu_addr, salt, nr_rounds, t_tables_gpu_addr, + * htable_gpu_addr + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* s27 =3D slot_idx * SA_ENTRY_SIZE (scalar mul) */ + _E(emit_gfx9_s_mul_i32, I9(buf, n), P_S(27), P_S(26), + P_L(KNOD_IPSEC_SHADER_SA_ENTRY_SZ)); + /* s[24:25] =3D sa_table_addr (already there from Phase 0 scan) */ + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(24), P_S(24), P_S(27)); + _E(emit_gfx9_s_addc_u32, I9(buf, n), P_S(25), P_S(25), P_I(0)); + + /* Use VMEM for coherence: stage entry addr into VGPR pair */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_LO), P_S(24)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_HI), P_S(25)); + + /* Batched SA entry loads: issue all 4 loads to different + * VGPR destinations, then single waitcnt. v1-v8 (S0-S3, + * D0-D3) are free at Phase 3 - not used until Phase 7. + * + * Layout: + * dwordx4 @+16 -> v[1:4]: key_lo, key_hi, htable_lo, htable_hi + * dwordx4 @+32 -> v[5:8]: ttables_lo, ttables_hi, salt, key_len + * dwordx2 @+48 -> v[14:15]: nr_rounds, mode + * dwordx2 @+88 -> v[16:17]: stats_lo, stats_hi + */ + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_S0), + P_V(VR_GA_LO), SA_OFF_KEY_ADDR); + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_D0), + P_V(VR_GA_LO), SA_OFF_T_TABLES_ADDR); + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), SA_OFF_NR_ROUNDS); + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(VR_DATA2), + P_V(VR_GA_LO), SA_OFF_STATS_ADDR); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* key_addr: v1=3Dlo, v2=3Dhi */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_KEYS, VR_S0); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_KEYS + 1, VR_S1); + /* htable_addr: v3=3Dlo, v4=3Dhi */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_HTABLE_LO, VR_S2); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_HTABLE_HI, VR_S3); + /* t_tables_addr: v5=3Dlo, v6=3Dhi */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_T_ADDR, VR_D0); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_T_ADDR + 1, VR_D1); + /* salt: v7 */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_IV0, VR_D2); + /* nr_rounds: v14, mode: v15 */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_NR_ROUNDS, VR_DATA0); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_SA_MODE, VR_DATA1); + /* stats_addr: v16=3Dlo, v17=3Dhi -> save VGPRs for Phase 10 */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_STATS_LO), + P_V(VR_DATA2)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_STATS_HI), + P_V(VR_DATA3)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 4: Build AES-GCM nonce + * + * nonce[12] =3D salt[4] || IV[8] + * salt is already in s[SR_IV0]. Load IV from pkt + esp_hdr_off + 8. + * Recompute ESP base from saved pkt_addr + VR_SAVE_ESP_OFF since + * v[11:12] were clobbered by Phase 3 SA loads (VR_GA_LO=3D12). + * IV goes to s[SR_IV1] (bytes 4-7) and s[SR_IV2] (bytes 8-11). + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx9_global_load_dwordx2, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), ESP_REL_IV); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_IV1, VR_DATA0); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_IV2, VR_DATA1); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 5: Compute ciphertext bounds + * + * ctext_off =3D esp_hdr_off + 16 (SPI+seq+IV) + * ctext_len =3D pkt_len - ctext_off - ICV_LEN + * nblocks =3D (ctext_len + 15) >> 4 + * + * s28 is free here (last used in version gate) - use as scratch. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 28, VR_SAVE_ESP_OFF); + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(28), + P_I(ESP_REL_CTEXT + ESP_ICV_LEN), + P_S(28)); /* s28 =3D ctext_off + ICV_LEN =3D overhead to subtract */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), SR_CTEXT_LEN, + VR_SAVE_PKTLEN); + _E(emit_gfx9_s_sub_u32_p, I9(buf, n), P_S(SR_CTEXT_LEN), + P_S(SR_CTEXT_LEN), P_S(28)); + /* s[SR_NBLOCKS_GCM] =3D (ctext_len + 15) >> 4 */ + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_NBLOCKS_GCM), P_I(15), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx9_s_lshr_b32, I9(buf, n), P_S(SR_NBLOCKS_GCM), + P_S(SR_NBLOCKS_GCM), P_I(4)); + /* total GHASH blocks =3D 1(AAD) + nblocks(ctext) + 1(len) =3D nblocks + 2 + */ + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(SR_TOTAL_GHASH_BLK), + P_I(2), P_S(SR_NBLOCKS_GCM)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 6: Cooperative T-table load (VRAM -> LDS) + * + * All 256 threads load from SA's t_tables_gpu_addr. Each thread + * loads one u32 per table (4 tables x 256 entries =3D 4KB). + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_MASK), P_L(0xFF)); + + /* v[VR_TMP] =3D tid * 4 (byte offset within each 1KB table) */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_TMP), P_I(2), + P_V(VR_TID)); + + /* T0: VRAM[t_tables + tid*4] -> LDS[tid*4] */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_LO), P_S(SR_T_ADDR)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_HI), + P_S(SR_T_ADDR + 1)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), P_V(VR_GA_LO), + P_V(VR_TMP)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), P_I(0), + P_V(VR_GA_HI)); + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_DATA1), + P_V(VR_GA_LO), 1024); + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_DATA2), + P_V(VR_GA_LO), 2048); + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_DATA3), + P_V(VR_GA_LO), 3072); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Write to LDS: T0 at +0, T1 at +1024, T2 at +2048, T3 at +3072 */ + _E(emit_gfx9_ds_write_b32, I9(buf, n), VR_TMP, VR_DATA0); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_ADDR), P_L(1024), + P_V(VR_TMP)); + _E(emit_gfx9_ds_write_b32, I9(buf, n), VR_ADDR, VR_DATA1); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_ADDR), P_L(2048), + P_V(VR_TMP)); + _E(emit_gfx9_ds_write_b32, I9(buf, n), VR_ADDR, VR_DATA2); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_ADDR), P_L(3072), + P_V(VR_TMP)); + _E(emit_gfx9_ds_write_b32, I9(buf, n), VR_ADDR, VR_DATA3); + + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_s_barrier, I9(buf, n)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 7: AES-CTR decrypt + * + * Each thread handles block_id =3D tid. Only threads with tid < + * nblocks are active. Counter =3D nonce[12] || bswap32(tid+2). + * AES-encrypt the counter -> keystream. XOR with ciphertext -> + * plaintext. Store to out_addr + tid*16. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* VCC =3D (nblocks > tid) i.e. tid < nblocks - selects active CTR lanes + */ + _E(emit_gfx9_v_cmp_gt_u32, I9(buf, n), P_S(SR_NBLOCKS_GCM), + P_V(VR_TID)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_EXEC_SAVE, + 106 /* VCC_LO */); + br_execz_ctr =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* Save SR_KEYS for reload after this block encrypt */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_T_ADDR), P_S(SR_KEYS)); + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_T_ADDR + 1), + P_S(SR_KEYS + 1)); + + /* Prefetch ciphertext into v[23:26] before AES. + * The ~200+ cycle AES encrypt hides the VMEM latency. + * v23-v26 are not touched by AES rounds (which use + * v1-v18 only). VR_GA/VR_BLK are also AES-safe. + */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_BLK), P_I(4), + P_V(VR_TID)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_I(ESP_REL_CTEXT), P_V(VR_GA_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_BLK)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_PREFETCH0), + P_V(VR_GA_LO), 0); + + /* Build AES counter block in v[VR_S0:VR_S3]: + * VR_S0 =3D nonce[0:3] =3D salt (SR_IV0) + * VR_S1 =3D nonce[4:7] =3D IV[0:3] (SR_IV1) + * VR_S2 =3D nonce[8:11] =3D IV[4:7] (SR_IV2) + * VR_S3 =3D bswap32(tid + 2) + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S0), P_S(SR_IV0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S1), P_S(SR_IV1)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S2), P_S(SR_IV2)); + + /* v[VR_S3] =3D bswap32(tid + 2) */ + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_S3), P_I(2), P_V(VR_TID)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_S3), P_S(SR_BSWAP)); + + /* AES encrypt the counter block -> result in v[VR_S0:VR_S3] */ + n =3D emit_aes_encrypt_block_gfx9(buf, n); + + /* Ciphertext arrived during AES - drain vmcnt */ + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* XOR keystream with prefetched ciphertext -> plaintext */ + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_PREFETCH0), + P_V(VR_S0), P_V(VR_PREFETCH0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_PREFETCH1), + P_V(VR_S1), P_V(VR_PREFETCH1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_PREFETCH2), + P_V(VR_S2), P_V(VR_PREFETCH2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_PREFETCH3), + P_V(VR_S3), P_V(VR_PREFETCH3)); + + /* Store plaintext to out_addr + tid*16 */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_BLK), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_OUT_HI)); + _E(emit_gfx9_global_store_dwordx4, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_PREFETCH0), 0); + + patch_branch(buf, br_execz_ctr, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 7.5: Compute AES(K, J0) for ICV finalization + * + * J0 =3D nonce[12] || 0x00000001 (BE). Only thread 0 needs this + * but all active lanes can compute it; we just save the result. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Restore SR_KEYS (consumed by encrypt_block) */ + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_KEYS), P_S(SR_T_ADDR)); + _E(emit_gfx9_s_mov_b32, I9(buf, n), P_S(SR_KEYS + 1), + P_S(SR_T_ADDR + 1)); + + /* Restore full EXEC for J0 encrypt (all 256 threads) */ + _E(emit_gfx9_s_or_b64, I9(buf, n), 126 /* EXEC_LO */, SR_EXEC_SAVE, + SR_EXEC_SAVE); + + /* J0 block: nonce || bswap32(1) =3D nonce || 0x01000000 */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S0), P_S(SR_IV0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S1), P_S(SR_IV1)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S2), P_S(SR_IV2)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_S3), P_L(0x01000000u)); + + n =3D emit_aes_encrypt_block_gfx9(buf, n); + + /* Save AES(K, J0) -> v[VR_J0_0:VR_J0_3] */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_J0_0), P_V(VR_S0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_J0_1), P_V(VR_S1)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_J0_2), P_V(VR_S2)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_J0_3), P_V(VR_S3)); + + _E(emit_gfx9_s_barrier, I9(buf, n)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 8: Parallel GHASH + * + * GHASH input blocks (total_blocks =3D nblocks + 2): + * tid 0 -> AAD: SPI(4B,BE)||seq(4B,BE)||0s (16B) + * tid 1..nblocks -> ciphertext block (tid-1) + * tid nblocks+1 -> len: AAD_bitlen(64b)||ctext_bitlen(64b) + * tid > nblocks+1 -> zero (does not participate) + * + * Each thread loads its block -> v[VR_DATA0:VR_DATA3] (big-endian + * for GF multiply), loads H^(total-tid) -> v[VR_D0:VR_D3], runs + * GF multiply -> v[VR_S0:VR_S3], then tree-reduces via LDS XOR. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Prefetch H^(total-tid) from H-power table before data selection. + * The ~60 ALU instructions in the data selection block below + * cover the VMEM latency. Result lands in VR_D0:D3, which + * data selection does not touch. VR_TMP/VR_GA are consumed + * here then free for reuse by the ctext section. + */ + _E(emit_gfx9_v_sub_u32, I9(buf, n), P_V(VR_TMP), + P_S(SR_TOTAL_GHASH_BLK), P_V(VR_TID)); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFFF), P_V(VR_TMP)); + _E(emit_gfx9_v_max_i32, I9(buf, n), P_V(VR_TMP), P_I(0), P_V(VR_TMP)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_TMP), P_I(4), + P_V(VR_TMP)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_LO), + P_S(SR_HTABLE_LO)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_HI), + P_S(SR_HTABLE_HI)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_TMP)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_D0), + P_V(VR_GA_LO), 0); + + /* EXEC-based per-lane data selection. Default =3D zero, then + * each case narrows EXEC to matching lanes and writes data. + * This avoids scalar VCC branching (s_cbranch_vccnz) which + * makes the entire wave take one path, not individual lanes. + */ + /* Default: all threads get zero (non-participating) */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA2), P_I(0)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA3), P_I(0)); + + /* ---- AAD: tid =3D=3D 0 ---- */ + _E(emit_gfx9_v_cmp_eq_u32, I9(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_aad =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* VR_SAVE_SPI/SEQ are already in BE register convention: + * raw LE load from packet (BE wire bytes) + bswap =3D byte[0] + * in bits[31:24]. No second bswap needed - use directly. + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_SAVE_SPI)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SEQ)); + /* DATA2, DATA3 already 0 */ + + patch_branch(buf, br_skip_aad, n); + _E(emit_gfx9_s_mov_b64, I9(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + + /* ---- Ctext: 1 <=3D tid <=3D nblocks ---- */ + /* block_idx =3D tid - 1 (unsigned; tid=3D=3D0 -> 0xFFFFFFFF > nblocks) */ + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_TMP), P_L(0xFFFFFFFF), + P_V(VR_TID)); + _E(emit_gfx9_v_cmp_gt_u32, I9(buf, n), P_S(SR_NBLOCKS_GCM), + P_V(VR_TMP)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_ctext =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* Load ctext block: pkt + esp_hdr_off + 16 + block_idx*16 */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_TMP), P_I(4), + P_V(VR_TMP)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_ESP_OFF), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_I(ESP_REL_CTEXT), P_V(VR_GA_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_GA_LO), P_V(VR_TMP)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_GA_HI)); + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Zero trailing dwords in the last partial ctext block. + * The load above reads 16 raw bytes, but for the last block + * only (ctext_len % 16) bytes are ciphertext - the rest are + * ICV bytes which must NOT enter GHASH. ESP ctext is always + * 4-byte aligned so the partial count is 4, 8 or 12 - pure + * dword-level zeroing suffices, no byte masking needed. + * + * VR_TMP still holds block_idx * 16 from the address calc. + * remaining =3D ctext_len - block_idx*16. For full blocks + * (remaining >=3D 16) every v_cmp evaluates true -> no change. + */ + _E(emit_gfx9_v_sub_u32, I9(buf, n), P_V(VR_TMP), + P_S(SR_CTEXT_LEN), P_V(VR_TMP)); + /* VR_TMP =3D remaining bytes in this block */ + + /* DATA3 (bytes 12-15): keep only if remaining > 12 */ + _E(emit_gfx9_v_cmp_lt_u32, I9(buf, n), P_I(12), P_V(VR_TMP)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), + P_V(VR_DATA3), P_I(0), P_V(VR_DATA3)); + + /* DATA2 (bytes 8-11): keep only if remaining > 8 */ + _E(emit_gfx9_v_cmp_lt_u32, I9(buf, n), P_I(8), P_V(VR_TMP)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), + P_V(VR_DATA2), P_I(0), P_V(VR_DATA2)); + + /* DATA1 (bytes 4-7): keep only if remaining > 4 */ + _E(emit_gfx9_v_cmp_lt_u32, I9(buf, n), P_I(4), P_V(VR_TMP)); + _E(emit_gfx9_v_cndmask_b32_e32, I9(buf, n), + P_V(VR_DATA1), P_I(0), P_V(VR_DATA1)); + + /* DATA0 (bytes 0-3): always valid (ESP 4-byte alignment) */ + + /* bswap each dword for GHASH (big-endian GF arithmetic) */ + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA0), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_DATA1), + P_V(VR_DATA1), P_V(VR_DATA1), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_DATA2), + P_V(VR_DATA2), P_V(VR_DATA2), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_DATA3), + P_V(VR_DATA3), P_V(VR_DATA3), P_S(SR_BSWAP)); + + patch_branch(buf, br_skip_ctext, n); + _E(emit_gfx9_s_mov_b64, I9(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + + /* ---- Len block: tid =3D=3D nblocks + 1 ---- */ + /* Compute nblocks+1 in s42 (scratch) */ + _E(emit_gfx9_s_add_u32, I9(buf, n), P_S(42), P_I(1), + P_S(SR_NBLOCKS_GCM)); + _E(emit_gfx9_v_cmp_eq_u32, I9(buf, n), P_S(42), + P_V(VR_TID)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip_len =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* Length block format (GCM big-endian): + * DATA0 =3D AAD_bits[63:32] =3D 0 + * DATA1 =3D AAD_bits[31:0] =3D 64 (8 bytes AAD x 8) + * DATA2 =3D ctext_bits[63:32] =3D 0 + * DATA3 =3D ctext_bits[31:0] =3D ctext_len * 8 + * + * No bswap32: length values are computed integers already + * in the correct big-endian register representation. + * bswap32 is only needed for data loaded from LE memory. + */ + /* DATA0, DATA2 already 0 from default init */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), + P_L(0x00000040u)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA3), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_DATA3), P_I(3), + P_V(VR_DATA3)); + + patch_branch(buf, br_skip_len, n); + _E(emit_gfx9_s_mov_b64, I9(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + + /* H-table data was prefetched before data selection; drain + bswap */ + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_D0), + P_V(VR_D0), P_V(VR_D0), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_D1), + P_V(VR_D1), P_V(VR_D1), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_D2), + P_V(VR_D2), P_V(VR_D2), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_D3), + P_V(VR_D3), P_V(VR_D3), P_S(SR_BSWAP)); + + /* ---- GF(2^128) multiply: Z =3D DATA * H^k ---- */ + n =3D emit_gfmul_128_gfx9(buf, n); + /* Result in v[VR_S0:VR_S3] */ + + /* ---- Tree reduction via LDS XOR (8 levels for 256 threads) ---- */ + /* Write v[VR_S0:VR_S3] to LDS at tid * 16 */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_ADDR), P_I(4), + P_V(VR_TID)); + _E(emit_gfx9_ds_write_b128, I9(buf, n), VR_ADDR, VR_S0); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + _E(emit_gfx9_s_barrier, I9(buf, n)); + + for (level =3D 1; level <=3D 128; level <<=3D 1) { + int br_skip; + + /* if (tid & level) skip */ + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(VR_TMP), + P_L(level), P_V(VR_TID)); + _E(emit_gfx9_v_cmp_eq_u32, I9(buf, n), P_I(0), P_V(VR_TMP)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_GHASH_EXEC, + 106 /* VCC */); + br_skip =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* Compute both addresses up front */ + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_TMP), + P_L(level), P_V(VR_TID)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_TMP), + P_I(4), P_V(VR_TMP)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_ADDR), + P_I(4), P_V(VR_TID)); + + /* Issue both reads, single wait */ + _E(emit_gfx9_ds_read_b128, I9(buf, n), VR_D0, VR_TMP); + _E(emit_gfx9_ds_read_b128, I9(buf, n), VR_S0, VR_ADDR); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + + /* XOR */ + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S0), + P_V(VR_S0), P_V(VR_D0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S1), + P_V(VR_S1), P_V(VR_D1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S2), + P_V(VR_S2), P_V(VR_D2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S3), + P_V(VR_S3), P_V(VR_D3)); + + /* Write back */ + _E(emit_gfx9_ds_write_b128, I9(buf, n), VR_ADDR, VR_S0); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + + patch_branch(buf, br_skip, n); + /* Restore EXEC */ + _E(emit_gfx9_s_mov_b64, I9(buf, n), 126 /* EXEC */, + SR_GHASH_EXEC); + _E(emit_gfx9_s_barrier, I9(buf, n)); + } + + /* Thread 0 now has the final GHASH in LDS[0..15]. Read it. */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_TMP), P_I(0)); + _E(emit_gfx9_ds_read_b128, I9(buf, n), VR_S0, VR_TMP); + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 9: ICV verify (thread 0 only) + * + * computed_tag =3D bswap(GHASH) XOR AES(K, J0) + * received_tag =3D last 16 bytes of ESP packet + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_v_cmp_eq_u32, I9(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_EXEC_SAVE, 106); + br_tid0 =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + /* bswap GHASH from big-endian to little-endian */ + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_S0), P_V(VR_S0), + P_V(VR_S0), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_S1), P_V(VR_S1), + P_V(VR_S1), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_S2), P_V(VR_S2), + P_V(VR_S2), P_S(SR_BSWAP)); + _E(emit_gfx9_v_perm_b32, I9(buf, n), P_V(VR_S3), P_V(VR_S3), + P_V(VR_S3), P_S(SR_BSWAP)); + + /* computed_tag =3D GHASH XOR AES(K, J0) */ + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S0), + P_V(VR_S0), P_V(VR_J0_0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S1), + P_V(VR_S1), P_V(VR_J0_1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S2), + P_V(VR_S2), P_V(VR_J0_2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_S3), + P_V(VR_S3), P_V(VR_J0_3)); + + /* Load received ICV: pkt + pkt_len - 16 */ + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFF0u), P_V(VR_SAVE_PKTLEN)); /* pkt_len - 16 */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_PKT_HI)); + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Compare: XOR each dword, OR together; if any non-zero -> fail */ + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_S0)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_DATA1), + P_V(VR_DATA1), P_V(VR_S1)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_DATA2), + P_V(VR_DATA2), P_V(VR_S2)); + _E(emit_gfx9_v_xor_b32_e32, I9(buf, n), P_V(VR_DATA3), + P_V(VR_DATA3), P_V(VR_S3)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA1)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA2)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_DATA0), P_V(VR_DATA3)); + + /* If VR_DATA0 !=3D 0 -> ICV fail: overwrite verdict with sentinel */ + _E(emit_gfx9_v_cmp_ne_u32, I9(buf, n), P_I(0), P_V(VR_DATA0)); + br_icv_ok =3D _BR(emit_gfx9_s_cbranch_vccz, I9(buf, n), 0); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_SAVE_SLOT), + P_L(VERDICT_ICV_FAIL)); + patch_branch(buf, br_icv_ok, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 10: ESP trailer strip + write verdict (thread 0) + * + * Decrypted tail: pad_len at out + ctext_len - 2 + * next_hdr at out + ctext_len - 1 + * inner_len =3D ctext_len - pad_len - 2 + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* Only strip if ICV passed (slot < NR_SA) */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 27, VR_SAVE_SLOT); + _E(emit_gfx9_s_cmp_ge_u32, I9(buf, n), P_S(27), + P_L(KNOD_IPSEC_SHADER_NR_SA)); + br_icv_bad =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + /* Load last 4 bytes of decrypted payload: out + ctext_len - 4. + * VOP2 src1 must be VGPR, so move SGPR to VR_TMP first. + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_TMP), + P_S(SR_CTEXT_LEN)); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_TMP), + P_L(0xFFFFFFFC), P_V(VR_TMP)); /* ctext_len - 4 */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_OUT_HI)); + _E(emit_gfx9_global_load_dword, I9(buf, n), P_V(VR_DATA0), + P_V(VR_GA_LO), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* On LE: loaded dword has byte layout [b0,b1,b2,b3]. + * We loaded from (ctext_len - 4), so: + * b2 =3D pad_len (at ctext_len - 2) + * b3 =3D next_hdr (at ctext_len - 1) + * pad_len =3D (dword >> 16) & 0xFF + */ + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(VR_TMP), + P_I(16), P_V(VR_DATA0)); + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), P_V(VR_TMP), + P_L(0xFF), P_V(VR_TMP)); + + /* inner_len =3D ctext_len - pad_len - 2 */ + _E(emit_gfx9_v_sub_u32, I9(buf, n), P_V(VR_DATA1), + P_S(SR_CTEXT_LEN), P_V(VR_TMP)); + _E(emit_gfx9_v_add_u32, I9(buf, n), P_V(VR_DATA1), + P_L(0xFFFFFFFE), P_V(VR_DATA1)); /* -2 */ + + /* Write bd->len =3D inner_len (u16 at bd + 18) */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_L(18), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx9_global_store_short, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA1), 0); + + /* Write bd->off =3D mode | (next_hdr << 8) (u16 at bd + 16). + * next_hdr =3D byte[3] of the ESP trailer dword (VR_DATA0). + * mode from s[SR_SA_MODE]. + */ + _E(emit_gfx9_v_lshrrev_b32, I9(buf, n), P_V(VR_TMP), + P_I(24), P_V(VR_DATA0)); + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), P_V(VR_TMP), + P_I(8), P_V(VR_TMP)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA2), + P_S(SR_SA_MODE)); + _E(emit_gfx9_v_or_b32_e32, I9(buf, n), P_V(VR_TMP), + P_V(VR_DATA2), P_V(VR_TMP)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_L(16), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx9_global_store_short, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_TMP), 0); + + /* Per-SA GPU stats: atomically increment rx_packets + * and rx_bytes at stats_addr. VR_DATA1 still holds + * inner_len from the bd->len computation above. + * + * global_atomic_add_x2 uses v[data:data+1] as u64. + * Save inner_len to VR_TMP before clobbering DATA1. + * + * stats layout (knod_ipsec_sa_gpu_stats): + * +0: rx_packets (u64, LE) + * +8: rx_bytes (u64, LE) + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_TMP), + P_V(VR_DATA1)); /* save inner_len */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_SAVE_STATS_LO)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_GA_HI), + P_V(VR_SAVE_STATS_HI)); + + /* rx_packets +=3D 1: v[DATA0:DATA1] =3D {1, 0} */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), P_I(1)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), P_I(0)); + _E(emit_gfx9_global_atomic_add_x2, I9(buf, n), + P_V(VR_DATA2), P_V(VR_GA_LO), P_V(VR_DATA0), 0, 0); + + /* rx_bytes +=3D inner_len: v[DATA0:DATA1] =3D {inner_len, 0} */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_V(VR_TMP)); + /* DATA1 already 0 from above */ + _E(emit_gfx9_global_atomic_add_x2, I9(buf, n), + P_V(VR_DATA2), P_V(VR_GA_LO), P_V(VR_DATA0), 8, 0); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * L3 header passthrough (transport mode only). + * + * Copy 20 B of the outer IPv4 L3 header from + * pkt + 14 (skip ETH) to out_addr - 20. In shader-GTT + * direct mode (knod_ipsec_sdma=3D0), out_addr - 20 is + * pass_buf_slot + 0 so the host-side finalise can skip + * the per-packet L3 SDMA copy entirely - the only + * remaining SDMA call on the transport IPv4 fast path. + * + * Tunnel-mode SAs (SR_SA_MODE !=3D 0) skip this write + * because for tunnel the destination at slot+0 wants + * the *inner* packet, not an outer IP header. + * + * The VRAM staging path (knod_ipsec_sdma=3D1) also runs + * this copy, but the destination is the 20-byte + * headroom knod_ipsec.c reserves at the front of + * the decrypt pool; CPU finalise still SDMAs the real + * L3 header from the raw packet into the GTT pass_buf + * so the shader's write is harmless wasted work. + * + * Alignment: pkt + 14 is only 2-byte aligned (ETH hdr + * =3D 14 bytes !=3D 4-byte multiple), so dword / dwordx4 + * loads would fault. Use 10 x global_load_ushort at + * offsets 14,16,...,32, paired with 10 x store_short + * at slot + 0,2,...,18. 10 scratch VGPRs (v14..v23), + * all free by Phase 10 since AES-GCM / GHASH state is + * done. One waitcnt between loads and stores. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), + P_S(SR_SA_MODE), P_I(0)); + br_not_transport =3D _BR(emit_gfx9_s_cbranch_scc0, + I9(buf, n), 0); + + /* src =3D pkt_addr + 14 */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), + P_V(VR_GA_LO), P_I(14), + P_V(VR_SAVE_PKT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), + P_V(VR_GA_HI), P_I(0), + P_V(VR_SAVE_PKT_HI)); + + /* 10 x 2-byte loads from src+0..+18 */ + for (li =3D 0; li < 10; li++) { + _E(emit_gfx9_global_load_ushort, + I9(buf, n), + P_V(L3_TMP_BASE + li), + P_V(VR_GA_LO), li * 2); + } + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* dst =3D out_addr - 20. + * + * Two GFX9 landmines here: + * + * 1. v_add_co_u32 can't take a 32-bit + * literal src together with implicit VCC + * - same class as the v_cndmask literal + * restriction. + * 2. P_I(n) is a raw initializer that always + * sets type=3DINTEGER_0 and stores n in .v. + * For negative inline constants the + * encoder must flip to INTEGER_MINUS_1 + * with v=3D~n, which P_I does NOT do. + * P_I(-1) therefore encodes as + * GFX9_SRC_INTEGER_0 + (-1) =3D 127, a + * bogus register that gave us random + * high-32 bits and page-faulted stores. + * + * Dodge both by materialising -20 into VR_TMP + * and -1 into VR_TMP2 via v_mov_b32 literals + * (VOP1, no VCC, literals fine), then pure + * VGPR+VGPR add_co / addc_co. Borrow flows + * through VCC as the carry-in to addc_co. + */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), + P_V(VR_TMP), P_L(0xFFFFFFECu)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), + P_V(VR_TMP2), P_L(0xFFFFFFFFu)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), + P_V(VR_GA_LO), + P_V(VR_TMP), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), + P_V(VR_GA_HI), + P_V(VR_TMP2), P_V(VR_SAVE_OUT_HI)); + + /* 10 x 2-byte stores to dst+0..+18 */ + for (li =3D 0; li < 10; li++) { + _E(emit_gfx9_global_store_short, + I9(buf, n), + P_V(VR_GA_LO), + P_V(L3_TMP_BASE + li), + li * 2); + } + + patch_branch(buf, br_not_transport, n); + + patch_branch(buf, br_icv_bad, n); + + /* Write bd->act (u64 at bd + 8): + * high32 =3D slot_idx (or sentinel), + * low32 =3D KNOD_IPSEC_INFLIGHT so NIC NAPI recognises + * this slot as in-flight until the finish worker stamps + * the final PASS/DROP. + */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_L(8), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_L(KNOD_IPSEC_INFLIGHT)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SLOT)); + _E(emit_gfx9_global_store_dwordx2, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA0), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + patch_branch(buf, br_tid0, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 11: Miss/bypass path verdict - thread 0 only + * + * If we skipped crypto (Phase 2 branch), write the miss/bypass + * sentinel that's still in v[VR_SAVE_SLOT]. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + br_crypto_done =3D _BR(emit_gfx9_s_branch, I9(buf, n), 0); + + patch_branch(buf, br_crypto_end, n); + + /* Thread 0 writes bd->act with miss/bypass sentinel */ + _E(emit_gfx9_v_cmp_eq_u32, I9(buf, n), P_I(0), P_V(VR_TID)); + _E(emit_gfx9_s_and_saveexec_b64, I9(buf, n), SR_EXEC_SAVE, 106); + + br_execz2 =3D _BR(emit_gfx9_s_cbranch_execz, I9(buf, n), 0); + + _E(emit_gfx9_v_add_co_u32, I9(buf, n), P_V(VR_GA_LO), + P_L(8), P_V(VR_SAVE_BD_LO)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), P_V(VR_GA_HI), + P_I(0), P_V(VR_SAVE_BD_HI)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA0), + P_L(KNOD_IPSEC_INFLIGHT)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(VR_DATA1), + P_V(VR_SAVE_SLOT)); + _E(emit_gfx9_global_store_dwordx2, I9(buf, n), P_V(VR_GA_LO), + P_V(VR_DATA0), 0); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + patch_branch(buf, br_execz2, n); + + patch_branch(buf, br_crypto_done, n); + + /* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 12: GPU-initiated SDMA dispatch (thread 0 only) + * + * Both merge paths (crypto-done, bypass/miss) arrive here with + * EXEC restricted to lane 0 of wave 0; waves 1-3 have EXEC=3D0. + * Scalar instructions fire on every wave but are side-effect-free; + * vector memory ops are NOPs when EXEC=3D0. + * + * The GPU writes SDMA COPY_LINEAR packets only. FENCE, wptr + * update, and doorbell are left to the CPU - this avoids a race + * where a GPU-emitted FENCE would prematurely satisfy the CPU + * fence poll before CPU-added SDMA copies (e.g. IPv6 transport + * L3 header) complete. + * + * Protocol per work-group (=3D per packet): + * 1. Load sdma_ring_addr (kernarg+16) and sdma_ctl_addr (+32). + * 2. If sdma_ctl_addr =3D=3D 0, gpu_sdma disabled - skip. + * 3. Load sdma_ctl fields (wptr_base_dw, ring_mask, etc.). + * 4. If verdict is MISS/BYPASS, claim a ring slot via + * atomic_add(&claim_counter) and write a 7-dword + * SDMA COPY_LINEAR packet. + * 5. ALL WGs atomic_add(&done_counter). + * 6. Last WG: read final claim_counter, store + * final_sdma_count + gpu_sdma_ready for CPU. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + /* --- kernarg loads ---------------------------------------- */ + _E(emit_gfx9_s_load_dwordx2, I9(buf, n), + P_S(24), P_S(8), 16); /* s[24:25] =3D sdma_ring_addr */ + _E(emit_gfx9_s_load_dwordx2, I9(buf, n), + P_S(26), P_S(8), 32); /* s[26:27] =3D sdma_ctl_addr */ + _E(emit_gfx9_s_waitcnt_lgkmcnt, I9(buf, n)); + + /* Skip Phase 12 entirely if sdma_ctl_addr =3D=3D 0 */ + _E(emit_gfx9_s_or_b32, I9(buf, n), 28, 26, 27); + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), P_S(28), P_I(0)); + br_no_sdma =3D _BR(emit_gfx9_s_cbranch_scc1, I9(buf, n), 0); + + /* --- load sdma_ctl into VGPRs ----------------------------- */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(1), P_S(26)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(2), P_S(27)); + + /* ctl+28: wptr_base_dw(4) ring_mask(4) nr_total_wg(4) copy_hdr(4) + * -> v[3:6] + */ + _E(emit_gfx9_global_load_dwordx4, I9(buf, n), + P_V(3), P_V(1), 28); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* v3=3Dwptr_base_dw v4=3Dring_mask v5=3Dnr_total_wg v6=3Dcopy_hdr */ + + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), + 28, 5); /* s28 =3D nr_total_wg */ + + /* --- verdict check ---------------------------------------- */ + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), + 29, VR_SAVE_SLOT); /* s29 =3D verdict */ + _E(emit_gfx9_s_cmp_ge_u32, I9(buf, n), + P_S(29), P_L(0xFFFFFFFEu)); /* MISS|BYPASS? */ + br_no_copy =3D _BR(emit_gfx9_s_cbranch_scc0, I9(buf, n), 0); + + /* =3D=3D=3D This WG needs SDMA copy =3D=3D=3D */ + + /* atomic_add(&ctl->claim_counter, 1, GLC=3D1) -> my_idx */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(7), P_I(1)); + _E(emit_gfx9_global_atomic_add, I9(buf, n), + P_V(7), P_V(1), P_V(7), 0, 1); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + /* v7 =3D my_idx (old claim_counter) */ + + /* ring dword position: (wptr_base_dw + my_idx*7) & ring_mask */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), + P_V(8), P_I(3), P_V(7)); /* my_idx * 8 */ + _E(emit_gfx9_v_sub_u32, I9(buf, n), + P_V(8), P_V(8), P_V(7)); /* my_idx * 7 */ + _E(emit_gfx9_v_add_u32, I9(buf, n), + P_V(8), P_V(3), P_V(8)); /* + wptr_base_dw */ + _E(emit_gfx9_v_and_b32_e32, I9(buf, n), + P_V(8), P_V(4), P_V(8)); /* & ring_mask */ + _E(emit_gfx9_v_lshlrev_b32, I9(buf, n), + P_V(8), P_I(2), P_V(8)); /* * 4 -> byte offset */ + + /* v[9:10] =3D sdma_ring_addr + byte_offset */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(14), P_S(25)); + _E(emit_gfx9_v_add_co_u32, I9(buf, n), + P_V(9), P_S(24), P_V(8)); + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), + P_V(10), P_I(0), P_V(14)); + + /* DW0: copy_hdr */ + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(6), 0); + + /* DW1: nbytes - 1 */ + _E(emit_gfx9_v_add_u32, I9(buf, n), + P_V(14), P_L(0xFFFFFFFFu), P_V(VR_SAVE_PKTLEN)); + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(14), 4); + + /* DW2: 0 (sub-op parameter) */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(14), P_I(0)); + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(14), 8); + + /* DW3-4: src =3D pkt_addr (VRAM) */ + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(VR_SAVE_PKT_LO), 12); + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(VR_SAVE_PKT_HI), 16); + + /* DW5-6: dst =3D out_addr - 20 (GTT slot start) */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), + P_V(14), P_L(0xFFFFFFECu)); /* -20 */ + _E(emit_gfx9_v_add_co_u32, I9(buf, n), + P_V(15), P_V(14), P_V(VR_SAVE_OUT_LO)); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), + P_V(14), P_L(0xFFFFFFFFu)); /* -1 */ + _E(emit_gfx9_v_addc_co_u32, I9(buf, n), + P_V(16), P_V(14), P_V(VR_SAVE_OUT_HI)); + + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(15), 20); + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(9), P_V(16), 24); + + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* --- done counter (all WGs) ------------------------------- */ + patch_branch(buf, br_no_copy, n); + + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(7), P_I(1)); + _E(emit_gfx9_global_atomic_add, I9(buf, n), + P_V(7), P_V(1), P_V(7), 4, 1); /* ctl+4 =3D done_counter */ + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Last WG check: my_done + 1 =3D=3D nr_total_wg? */ + _E(emit_gfx9_v_add_u32, I9(buf, n), + P_V(7), P_I(1), P_V(7)); + _E(emit_gfx9_v_readfirstlane_b32, I9(buf, n), 29, 7); + _E(emit_gfx9_s_cmp_eq_u32, I9(buf, n), P_S(29), P_S(28)); + br_not_last =3D _BR(emit_gfx9_s_cbranch_scc0, I9(buf, n), 0); + + /* =3D=3D=3D Last WG - publish counters for CPU =3D=3D=3D */ + + /* Read final claim_counter (atomic add 0, GLC=3D1) */ + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(7), P_I(0)); + _E(emit_gfx9_global_atomic_add, I9(buf, n), + P_V(7), P_V(1), P_V(7), 0, 1); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + /* Store final_sdma_count (ctl+52) and gpu_sdma_ready (ctl+48) */ + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(1), P_V(7), 52); + _E(emit_gfx9_v_mov_b32_e32, I9(buf, n), P_V(7), P_I(1)); + _E(emit_gfx9_global_store_dword, I9(buf, n), + P_V(1), P_V(7), 48); + _E(emit_gfx9_s_waitcnt_vmcnt, I9(buf, n)); + + patch_branch(buf, br_not_last, n); + patch_branch(buf, br_no_sdma, n); + + _E(emit_gfx9_s_endpgm, I9(buf, n)); + + return n * 4; +} + +#endif /* KNOD_HELPERS_IPSEC_FUSED_GFX9_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c b/drivers/gpu/drm= /amd/amdkfd/knod/knod_ipsec.c new file mode 100644 index 000000000000..f67e7e4299f2 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c @@ -0,0 +1,4273 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +/* + * KNOD IPsec (xfrm) full-packet GPU offload. + * + * Accepts xfrm SAs configured with XFRM_DEV_OFFLOAD_PACKET and runs + * ESP parse / SA lookup / AES-GCM decrypt on the GPU. Anti-replay is + * handled CPU-side in the NIC dd NAPI (see knod_ipsec_sa_window_check) + * after the finish worker has SDMA-copied decrypted inner packets into + * the per-queue framework delivery pool (pass_pool). Control plane alloca= tes + * per-SA key and GHASH H-power tables in VRAM (via the AES-GCM helpers be= low) + * and mirrors SA state into a GPU-visible SA table consumed by the + * fused RX shader. + * + * RX path: the GPU shader decrypts in place in VRAM and flags per-packet + * verdicts (ICV ok / malformed / no SA). A CPU finish worker then SDMA- + * bulk-copies successfully decrypted inner packets into per-queue host pa= ges + * from the framework delivery pool (pass_pool, page_pool-backed) and push= es + * a knod_pass_desc onto the framework per-queue pass_pending ring. The NIC + * dd drains that ring from its own NAPI (knod_d2h_drain), which builds the + * zero-copy head_frag skb (knod_pass_build_skb) and runs the ipsec + * finalisation hook (knod_ipsec_post_copy): RFC 4303 sliding-window + * anti-replay, cleartext L3 fix-up and secpath attach. The page recycles + * to the pool on skb free. + * + * The original p2pdma bd ring is only used for NIC netmem lifecycle / + * recycle and is completely decoupled from verdict delivery. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kfd_priv.h" +#include "kfd_hsa.h" +#include "kfd_knod.h" +#include "knod_ipsec.h" +#include "ipsec_fused_gfx9.h" +#include "ipsec_fused_gfx10.h" + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * AES-GCM core helpers: AES T-tables in VRAM and GHASH H-power table + * precomputation. IPsec ESP is the only KNOD GCM consumer, so these + * live here rather than in the always-builtin KNOD core. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* AES T-tables: 4 tables * 256 entries * 4 bytes =3D 4KB, laid out as + * T0 | T1 | T2 | T3 in a single VRAM page. + */ +#define KNOD_GCM_T_TABLE_SIZE (256 * sizeof(u32)) +#define KNOD_GCM_T_TABLE_COUNT 4 +#define KNOD_GCM_T_TABLES_TOTAL \ + (KNOD_GCM_T_TABLE_SIZE * KNOD_GCM_T_TABLE_COUNT) + +/* GHASH H-power table: H^1 .. H^256, each 16 bytes. */ +#define KNOD_GCM_H_POWER_COUNT 256 +#define KNOD_GCM_H_POWER_SIZE 16 +#define KNOD_GCM_H_TABLE_SIZE \ + (KNOD_GCM_H_POWER_COUNT * KNOD_GCM_H_POWER_SIZE) + +static struct knod_mem *knod_gcm_alloc_tables(struct knod *knod) +{ + struct knod_mem *tables; + u32 *p; + int i; + + tables =3D knod_alloc_mem(knod, PAGE_SIZE, KFD_IOC_ALLOC_MEM_FLAGS_VRAM); + if (IS_ERR(tables)) + return tables; + + p =3D (u32 *)tables->kaddr; + + /* + * Four rotated versions of aes_enc_tab matching the kernel + * enc_quarterround() convention (lib/crypto/aes.c): + * T0[x] =3D aes_enc_tab[x] + * Tk[x] =3D rol32(T0[x], k * 8) + * The GPU shader performs one AES round as four table lookups and + * four XORs per column. + */ + for (i =3D 0; i < 256; i++) + p[i] =3D aes_enc_tab[i]; + for (i =3D 0; i < 256; i++) + p[256 + i] =3D rol32(aes_enc_tab[i], 8); + for (i =3D 0; i < 256; i++) + p[512 + i] =3D rol32(aes_enc_tab[i], 16); + for (i =3D 0; i < 256; i++) + p[768 + i] =3D rol32(aes_enc_tab[i], 24); + + return tables; +} + +static void knod_gcm_free_tables(struct knod *knod, struct knod_mem *table= s) +{ + if (tables) + knod_free_mem(knod, tables); +} + +/* + * Multiply a by b in GF(2^128) with the GCM polynomial + * x^128 + x^7 + x^2 + x + 1 + * Both inputs and the output are stored as two big-endian u64. + */ +static void gf128_mul(u64 r[2], const u64 a[2], const u64 b[2]) +{ + u64 v[2], z[2]; + int i, j; + + v[0] =3D a[0]; + v[1] =3D a[1]; + z[0] =3D 0; + z[1] =3D 0; + + for (i =3D 0; i < 2; i++) { + u64 x =3D b[i]; + + for (j =3D 63; j >=3D 0; j--) { + if ((x >> j) & 1) { + z[0] ^=3D v[0]; + z[1] ^=3D v[1]; + } + + /* v >>=3D 1 (128-bit) with GCM reduction */ + if (v[1] & 1) { + v[1] =3D (v[1] >> 1) | (v[0] << 63); + v[0] =3D (v[0] >> 1) ^ ((u64)0xe1 << 56); + } else { + v[1] =3D (v[1] >> 1) | (v[0] << 63); + v[0] =3D v[0] >> 1; + } + } + } + + r[0] =3D z[0]; + r[1] =3D z[1]; +} + +static void knod_gcm_precompute_h_table(const u8 *key, int key_len, + u8 *h_table_buf) +{ + struct aes_enckey enc_key; + u8 h_block[AES_BLOCK_SIZE] =3D {}; + u64 h[2], h_power[2]; + int i; + + /* H =3D AES_K(0^128) */ + aes_prepareenckey(&enc_key, key, key_len); + aes_encrypt(&enc_key, h_block, h_block); + + h[0] =3D get_unaligned_be64(h_block); + h[1] =3D get_unaligned_be64(h_block + 8); + + /* H^1 =3D H, then H^(i+1) =3D H^i * H */ + h_power[0] =3D h[0]; + h_power[1] =3D h[1]; + + for (i =3D 0; i < KNOD_GCM_H_POWER_COUNT; i++) { + u64 next[2]; + + put_unaligned_be64(h_power[0], + h_table_buf + i * KNOD_GCM_H_POWER_SIZE); + put_unaligned_be64(h_power[1], + h_table_buf + i * KNOD_GCM_H_POWER_SIZE + 8); + + gf128_mul(next, h_power, h); + h_power[0] =3D next[0]; + h_power[1] =3D next[1]; + } + + memzero_explicit(&enc_key, sizeof(enc_key)); + memzero_explicit(h_block, sizeof(h_block)); +} + +/* Single AQL queue on purpose: CPU parallelism is provided by per-queue + * double-buffering through `nr_aql_ring` slots inside this one queue, not + * by multiple AQL queues. Multiple AQL queues cause ordering / contention + * issues with the fused RX dispatch path - stick to one. + */ +/* + * Dispatcher idle strategy toggle. Exposed as a debugfs toggle + * (knod_ipsec/poll) so it can be flipped mid-run without reloading + * the module and without the sysfs-module-param discoverability cost. + * + * - poll_mode=3Dfalse (default): when try_rx finds no work, + * sleep via usleep_range(20, 100). Lets the CPU idle but adds up to + * ~100us latency between a packet landing in the SPSC ring and the + * dispatcher picking it up. + * + * - poll_mode=3Dtrue: busy-poll with cpu_relax() instead of sleeping. Pins + * one CPU core at 100% even when no traffic is flowing, but removes + * the handoff latency entirely. + * + * GPU completion wait (knod_ipsec_dispatch_and_wait) is always polled - + * this switch only affects the between-dispatch idle path. + */ +static bool knod_ipsec_poll_mode; + +/* + * RX decrypt output: the shader decrypts ESP payloads into a per-work VRAM + * output buffer; the finish worker SDMA-copies the result into a framework + * delivery-pool page. For BYPASS/MISS traffic, the shader's Phase 12 wri= tes + * SDMA COPY_LINEAR packets directly into the SDMA ring, and the CPU only = adds + * FENCE + doorbell. + */ + +/* + * Parallel RX dispatcher count. Each dispatcher gets its own kaql + * AQL queue on the GPU, its own SDMA queue for finalise copies, its + * own work_pool slice and pool BOs, and an exclusive contiguous range + * of NIC RX queues. Two dispatchers running on kaql[0] and + * kaql[1] can execute dispatches in true parallel on disjoint CU + * slices, which is the only way to break the single-kaql FIFO + * throughput ceiling observed at ~60 Gbps (UDP) / ~41 Gbps (TCP) on + * the current test rig. + * + * The number must be fixed at module_init time because it drives + * knod_alloc_ctx() which creates the kaql[]/sdma[] pairs when NOD + * attaches. Hot-path code reads priv->nr_dispatchers which is + * initialised from priv->knod->queue_cnt. + * + * Load-time only (0444) - flipping between 1 and N at runtime would + * require tearing down / re-creating the knod context, which means + * unbinding NOD on the NIC. Reload the module instead. + */ +static int nr_dispatch =3D 1; +module_param(nr_dispatch, int, 0444); +MODULE_PARM_DESC(nr_dispatch, + "Number of parallel KNOD IPsec dispatchers (1..4, default: 1). Each run= s on its own AQL queue for real GPU parallelism."); + +static DEFINE_STATIC_KEY_FALSE(ipsec_stats_enabled_key); + +#define IPSEC_STAT_INC(s, field) do { \ + if (static_branch_unlikely(&ipsec_stats_enabled_key)) \ + this_cpu_inc((s)->field); \ +} while (0) +#define IPSEC_STAT_ADD(s, field, val) do { \ + if (static_branch_unlikely(&ipsec_stats_enabled_key)) \ + this_cpu_add((s)->field, (val)); \ +} while (0) + +static struct knod_ipsec_priv *ipsec_priv; + +static int knod_ipsec_nod_init(struct knod_dev *knodev); +static void knod_ipsec_nod_exit(struct knod_dev *knodev); +static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *= skb, + const struct knod_pass_desc *desc, + int queue_idx); +static int knod_ipsec_init_shader_gfx9(struct knod *knod); +static int knod_ipsec_init_shader_gfx10(struct knod *knod); +static int knod_ipsec_work_pool_alloc(struct knod_ipsec_priv *priv); +static void knod_ipsec_work_pool_free(struct knod_ipsec_priv *priv); +static int knod_ipsec_dispatcher(void *arg); +static void knod_ipsec_dispatcher_drain(struct knod_ipsec_dispatcher *disp= ); +static int knod_ipsec_disp_create_all(struct knod_ipsec_priv *priv); +static void knod_ipsec_disp_destroy_all(struct knod_ipsec_priv *priv); +static void knod_ipsec_debugfs_init(struct knod_ipsec_priv *priv); +static void knod_ipsec_debugfs_exit(struct knod_ipsec_priv *priv); + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Slot management + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static int knod_ipsec_find_free_slot(struct knod_ipsec_priv *priv) +{ + int i; + + /* + * Reuse only fully-freed slots (key_mem cleared by state_free); a + * deactivated slot whose free is still pending (an in-flight skb holds + * the xfrm_state) would be memset here, leaking its BOs. + */ + for (i =3D 0; i < KNOD_IPSEC_NR_SA; i++) { + if (!priv->slots[i].active && !priv->slots[i].key_mem) + return i; + } + return -ENOSPC; +} + +static struct knod_ipsec_sa_slot * +knod_ipsec_lookup_slot_by_spi(struct knod_ipsec_priv *priv, u32 spi) +{ + return xa_load(&priv->spi_to_slot, spi); +} + +/* + * Write (or clear) a slot entry into the GPU-visible SA table via the + * kernel mapping. SDMA flush is not required - the VRAM mapping used + * here is coherent and the shader re-reads per-dispatch. + */ +static void knod_ipsec_write_sa_entry(struct knod_ipsec_priv *priv, + int slot_idx, + const struct knod_ipsec_sa_slot *slot, + struct xfrm_state *x) +{ + struct knod_ipsec_sa_entry *e; + u32 flags =3D 0; + + e =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr; + e +=3D slot_idx; + memset(e, 0, sizeof(*e)); + + if (!slot || !x) { + e->active =3D cpu_to_le32(0); + /* Zero per-SA stats on delete */ + memset((u8 *)priv->sa_table->kaddr + + KNOD_IPSEC_STATS_REGION_OFF + + slot_idx * KNOD_IPSEC_SA_STATS_SIZE, 0, + KNOD_IPSEC_SA_STATS_SIZE); + return; + } + + if (x->props.flags & XFRM_STATE_ESN) + flags |=3D 1u << 0; + + /* Store SPI in host byte order (little-endian on this platform). + * The GPU shader byteswaps the raw BE SPI from the packet header + * before comparing, so the table entry must be in the same + * host-order form. cpu_to_le32(be32_to_cpu()) converts BE->host->LE + * which on LE is a no-op for the numeric value. + */ + e->spi =3D cpu_to_le32(be32_to_cpu(x->id.spi)); + e->dir =3D cpu_to_le32(0); + e->family =3D cpu_to_le32(x->props.family); + e->flags =3D cpu_to_le32(flags); + e->key_gpu_addr =3D cpu_to_le64(slot->key_mem ? + slot->key_mem->gaddr : 0); + e->htable_gpu_addr =3D cpu_to_le64(slot->htable_mem ? + slot->htable_mem->gaddr : 0); + e->t_tables_gpu_addr =3D cpu_to_le64(priv->t_tables ? + priv->t_tables->gaddr : 0); + e->replay_bitmap_addr =3D cpu_to_le64(slot->replay_mem ? + slot->replay_mem->gaddr : 0); + e->replay_window =3D cpu_to_le32(x->replay_esn ? + x->replay_esn->replay_window : 64); + + if (x->aead) { + int key_len =3D (x->aead->alg_key_len + 7) / 8; + + /* AES-GCM: last 4 bytes of keymat are the salt. */ + if (key_len >=3D 4) { + memcpy(e->salt, + x->aead->alg_key + key_len - 4, 4); + key_len -=3D 4; + } + e->key_len =3D cpu_to_le32(key_len); + /* nr_rounds: AES-128->10, AES-192->12, AES-256->14 */ + e->nr_rounds =3D cpu_to_le32(6 + key_len / 4); + } + + e->mode =3D cpu_to_le32(x->props.mode); + e->stats_addr =3D cpu_to_le64(priv->sa_table->gaddr + + KNOD_IPSEC_STATS_REGION_OFF + + slot_idx * KNOD_IPSEC_SA_STATS_SIZE); + /* Zero per-SA stats on (re)key */ + memset((u8 *)priv->sa_table->kaddr + KNOD_IPSEC_STATS_REGION_OFF + + slot_idx * KNOD_IPSEC_SA_STATS_SIZE, 0, + KNOD_IPSEC_SA_STATS_SIZE); + e->version =3D cpu_to_le32(slot->version); + e->active =3D cpu_to_le32(1); +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * xfrmdev_ops callbacks + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static int knod_ipsec_xdo_state_add(struct knod_dev *knodev, + struct xfrm_state *x, + struct netlink_ext_ack *extack) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_sa_entry *e_dbg; + struct crypto_aes_ctx aes_ctx; + struct knod_ipsec_sa_slot *slot; + int slot_idx, key_len, err; + u32 spi; + + if (!priv || !priv->knod) { + NL_SET_ERR_MSG(extack, "knod_ipsec: not initialized"); + return -ENODEV; + } + + /* Only support AEAD AES-GCM in packet offload mode. */ + if (x->xso.type !=3D XFRM_DEV_OFFLOAD_PACKET) { + NL_SET_ERR_MSG(extack, "knod_ipsec: only packet offload supported"); + return -EINVAL; + } + if (x->xso.dir !=3D XFRM_DEV_OFFLOAD_IN) { + NL_SET_ERR_MSG(extack, + "knod_ipsec: only inbound (RX decrypt) offload supported"); + return -EINVAL; + } + if (!x->aead || + strcmp(x->aead->alg_name, "rfc4106(gcm(aes))")) { + NL_SET_ERR_MSG(extack, "knod_ipsec: only rfc4106(gcm(aes)) supported"); + return -EINVAL; + } + if (x->id.proto !=3D IPPROTO_ESP) { + NL_SET_ERR_MSG(extack, "knod_ipsec: only ESP supported"); + return -EINVAL; + } + + key_len =3D (x->aead->alg_key_len + 7) / 8; + if (key_len < 4) { + NL_SET_ERR_MSG(extack, "knod_ipsec: key too short"); + return -EINVAL; + } + key_len -=3D 4; /* strip salt */ + if (key_len !=3D 16 && key_len !=3D 24 && key_len !=3D 32) { + NL_SET_ERR_MSG(extack, "knod_ipsec: unsupported AES key length"); + return -EINVAL; + } + + spi =3D be32_to_cpu(x->id.spi); + + mutex_lock(&priv->slot_lock); + + if (knod_ipsec_lookup_slot_by_spi(priv, spi)) { + mutex_unlock(&priv->slot_lock); + NL_SET_ERR_MSG(extack, "knod_ipsec: SPI already offloaded"); + return -EEXIST; + } + + slot_idx =3D knod_ipsec_find_free_slot(priv); + if (slot_idx < 0) { + mutex_unlock(&priv->slot_lock); + NL_SET_ERR_MSG(extack, "knod_ipsec: SA table full"); + return -ENOSPC; + } + + slot =3D &priv->slots[slot_idx]; + memset(slot, 0, sizeof(*slot)); + slot->slot_idx =3D slot_idx; + slot->spi =3D spi; + + /* Expanded round key buffer (VRAM). The GPU shader loads round keys + * via s_load_dwordx4 (16 bytes per round), so we expand up front + * rather than shipping raw key material. AES-128 is 176B, + * AES-256 is 240B; a full PAGE_SIZE allocation leaves room and + * avoids the VRAM + * 7-page alloc trap. + */ + slot->key_mem =3D knod_alloc_mem(priv->knod, PAGE_SIZE, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(slot->key_mem)) { + err =3D PTR_ERR(slot->key_mem); + slot->key_mem =3D NULL; + goto err_slot; + } + err =3D aes_expandkey(&aes_ctx, x->aead->alg_key, key_len); + if (err) { + NL_SET_ERR_MSG(extack, "knod_ipsec: AES key expand failed"); + goto err_key; + } + /* Store the encryption schedule. Nr+1 round keys x 16 bytes. + * The shader reads s[SR_NR_ROUNDS] to know how many rounds. + */ + memcpy(slot->key_mem->kaddr, aes_ctx.key_enc, + (aes_ctx.key_length / 4 + 7) * 16); + memzero_explicit(&aes_ctx, sizeof(aes_ctx)); + + /* H-power table (VRAM) */ + slot->htable_mem =3D knod_alloc_mem(priv->knod, KNOD_GCM_H_TABLE_SIZE, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(slot->htable_mem)) { + err =3D PTR_ERR(slot->htable_mem); + slot->htable_mem =3D NULL; + goto err_key; + } + knod_gcm_precompute_h_table(x->aead->alg_key, key_len, + (u8 *)slot->htable_mem->kaddr); + + /* Replay bitmap (VRAM) */ + slot->replay_mem =3D knod_alloc_mem(priv->knod, PAGE_SIZE, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(slot->replay_mem)) { + err =3D PTR_ERR(slot->replay_mem); + slot->replay_mem =3D NULL; + goto err_htable; + } + memset(slot->replay_mem->kaddr, 0, PAGE_SIZE); + + slot->x =3D x; + slot->version =3D 1; + slot->active =3D true; + + knod_ipsec_write_sa_entry(priv, slot_idx, slot, x); + + err =3D xa_insert(&priv->spi_to_slot, spi, slot, GFP_KERNEL); + if (err) + goto err_replay; + + x->xso.offload_handle =3D (unsigned long)slot; + + mutex_unlock(&priv->slot_lock); + + this_cpu_inc(priv->stats->sa_add); + + e_dbg =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr + slot_idx; + + pr_info("knod_ipsec: SA added spi=3D0x%08x slot=3D%d dir=3D%d\n", + spi, slot_idx, x->xso.dir); + pr_info(" sa_entry: spi_le=3D0x%08x active=3D%u nr_rounds=3D%u key_len= =3D%u\n", + le32_to_cpu(e_dbg->spi), le32_to_cpu(e_dbg->active), + le32_to_cpu(e_dbg->nr_rounds), le32_to_cpu(e_dbg->key_len)); + pr_info(" sa_entry: key_addr=3D0x%llx htable_addr=3D0x%llx salt=3D%*ph\n= ", + le64_to_cpu(e_dbg->key_gpu_addr), + le64_to_cpu(e_dbg->htable_gpu_addr), + 4, e_dbg->salt); + return 0; + +err_replay: + knod_free_mem(priv->knod, slot->replay_mem); +err_htable: + knod_free_mem(priv->knod, slot->htable_mem); +err_key: + knod_free_mem(priv->knod, slot->key_mem); +err_slot: + memset(slot, 0, sizeof(*slot)); + mutex_unlock(&priv->slot_lock); + return err; +} + +static void knod_ipsec_xdo_state_delete(struct knod_dev *knodev, + struct xfrm_state *x) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_sa_slot *slot; + u32 spi; + + if (!priv) + return; + + spi =3D be32_to_cpu(x->id.spi); + + mutex_lock(&priv->slot_lock); + slot =3D knod_ipsec_lookup_slot_by_spi(priv, spi); + if (!slot) { + mutex_unlock(&priv->slot_lock); + return; + } + + /* Deactivate first so GPU shaders skip this slot on next dispatch. */ + slot->active =3D false; + slot->version++; + knod_ipsec_write_sa_entry(priv, slot->slot_idx, NULL, NULL); + + xa_erase(&priv->spi_to_slot, spi); + + mutex_unlock(&priv->slot_lock); + + this_cpu_inc(priv->stats->sa_del); +} + +static void knod_ipsec_xdo_state_free(struct knod_dev *knodev, + struct xfrm_state *x) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_sa_slot *slot; + + if (!priv) + return; + + slot =3D (struct knod_ipsec_sa_slot *)x->xso.offload_handle; + if (!slot) + return; + + mutex_lock(&priv->slot_lock); + + if (slot->replay_mem) + knod_free_mem(priv->knod, slot->replay_mem); + if (slot->htable_mem) + knod_free_mem(priv->knod, slot->htable_mem); + if (slot->key_mem) + knod_free_mem(priv->knod, slot->key_mem); + + memset(slot, 0, sizeof(*slot)); + x->xso.offload_handle =3D 0; + + mutex_unlock(&priv->slot_lock); +} + +static bool knod_ipsec_xdo_offload_ok(struct knod_dev *knodev, + struct sk_buff *skb, + struct xfrm_state *x) +{ + /* We support only ESP / AES-GCM / packet offload; state_add + * already filtered unsupported cases. Offload OK for matching SA. + */ + return x->xso.type =3D=3D XFRM_DEV_OFFLOAD_PACKET && + x->id.proto =3D=3D IPPROTO_ESP && x->aead; +} + +static void knod_ipsec_xdo_state_advance_esn(struct knod_dev *knodev, + struct xfrm_state *x) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_sa_slot *slot; + struct knod_ipsec_sa_entry *e; + + if (!priv || !x->replay_esn) + return; + slot =3D (struct knod_ipsec_sa_slot *)x->xso.offload_handle; + if (!slot || !slot->active) + return; + + e =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr; + e +=3D slot->slot_idx; + + /* xfrm calls advance_esn whenever the inbound ESN high-32 bits + * change. The GPU reads seq_hi/seq_last via VMEM (global_load) so + * the COHERENT VRAM mapping ensures visibility without SDMA. + */ + WRITE_ONCE(e->seq_hi, cpu_to_le32(x->replay_esn->seq_hi)); + WRITE_ONCE(e->seq_last, cpu_to_le64( + ((u64)x->replay_esn->seq_hi << 32) | + (u64)x->replay_esn->seq)); + /* Ensure both fields are visible to GPU before returning. */ + wmb(); +} + +static void knod_ipsec_xdo_state_update_stats(struct knod_dev *knodev, + struct xfrm_state *x) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_sa_slot *slot; + struct knod_ipsec_sa_gpu_stats *gs; + u32 spi; + + if (!priv || !priv->sa_table) + return; + + spi =3D be32_to_cpu(x->id.spi); + slot =3D xa_load(&priv->spi_to_slot, spi); + if (!slot || !slot->active) + return; + + /* Read GPU-side per-SA counters (atomically updated by shader). */ + gs =3D (struct knod_ipsec_sa_gpu_stats *) + ((u8 *)priv->sa_table->kaddr + KNOD_IPSEC_STATS_REGION_OFF + + slot->slot_idx * KNOD_IPSEC_SA_STATS_SIZE); + + x->curlft.packets =3D le64_to_cpu(READ_ONCE(gs->rx_packets)); + x->curlft.bytes =3D le64_to_cpu(READ_ONCE(gs->rx_bytes)); +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * RX delivery queues (host SPSC descriptor ring; pages from pass_pool) + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* + * RFC 4303 anti-replay window check + update (single-writer, lockless). + * Returns true if the packet is accepted (not a replay and in window), + * false if it must be dropped. + * + * Caller guarantees that RSS pins the SA to a single RX queue, so the + * only writer to `win` on this CPU is the NAPI for this queue. The + * function both tests and updates - NIC dd calls it exactly once per + * desc, either to accept or to drop. + */ +static inline bool knod_ipsec_sa_window_check(struct knod_ipsec_sa_window = *win, + u64 seq) +{ + const unsigned int nwords =3D KNOD_IPSEC_CPU_REPLAY_WORDS; + unsigned int diff; + unsigned int w; + u64 bit; + int i; + + if (seq =3D=3D 0) + return false; + + if (seq > win->top_seq) { + diff =3D (unsigned int)(seq - win->top_seq); + if (diff >=3D KNOD_IPSEC_CPU_REPLAY_BITS) { + memset(win->bitmap, 0, sizeof(win->bitmap)); + } else { + unsigned int word_shift =3D diff >> 6; /* diff / 64 */ + unsigned int bit_shift =3D diff & 63; /* diff % 64 */ + + /* Shift the whole bitmap left by `diff` bits. bit at + * position p in bitmap[w] moves to p + diff. Work + * from the high word downwards so we don't overwrite + * source words before reading them. + */ + if (bit_shift =3D=3D 0) { + for (i =3D (int)nwords - 1; i >=3D 0; i--) { + int src =3D i - (int)word_shift; + + win->bitmap[i] =3D (src >=3D 0) + ? win->bitmap[src] : 0; + } + } else { + for (i =3D (int)nwords - 1; i >=3D 0; i--) { + int src_hi =3D i - (int)word_shift; + int src_lo =3D src_hi - 1; + u64 hi =3D 0, lo =3D 0; + + if (src_hi >=3D 0) + hi =3D win->bitmap[src_hi] << + bit_shift; + if (src_lo >=3D 0) + lo =3D win->bitmap[src_lo] >> + (64 - bit_shift); + win->bitmap[i] =3D hi | lo; + } + } + } + win->top_seq =3D seq; + win->bitmap[0] |=3D 1ull; + return true; + } + + diff =3D (unsigned int)(win->top_seq - seq); + if (diff >=3D KNOD_IPSEC_CPU_REPLAY_BITS) + return false; /* too old */ + w =3D diff >> 6; + bit =3D 1ull << (diff & 63); + + if (win->bitmap[w] & bit) + return false; /* replay */ + win->bitmap[w] |=3D bit; + return true; +} + +static int knod_ipsec_rxq_init_all(struct knod_ipsec_priv *priv) +{ + /* One delivery queue per NIC RX queue, capped at KNOD_SPSC_MAX to + * match the rest of the NOD/offmem/bd ring infrastructure. Per-queue + * delivery now flows through the framework pass_pending ring, so this + * only records the queue count for the finish-worker bounds check. + */ + int nr =3D priv->knodev && priv->knodev->netdev + ? (int)priv->knodev->netdev->num_rx_queues : 1; + + if (nr > KNOD_SPSC_MAX) + nr =3D KNOD_SPSC_MAX; + if (nr < 1) + nr =3D 1; + + priv->nr_rxq =3D nr; + return 0; +} + +static void knod_ipsec_rxq_exit_all(struct knod_ipsec_priv *priv) +{ + priv->nr_rxq =3D 0; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * NOD init / exit - allocate SA table and shared T-tables + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static int knod_ipsec_nod_init(struct knod_dev *knodev) +{ + struct amdgpu_device *adev; + struct knod_ipsec_priv *priv; + int err; + + if (ipsec_priv) { + pr_warn("knod_ipsec: priv already initialized\n"); + return -EBUSY; + } + + /* + * Pin the module while IPsec is the selected feature: the core calls + * into these ops, so it must not be unloaded until feature->none. + * (No-op when built in - THIS_MODULE is NULL.) + */ + if (!try_module_get(THIS_MODULE)) + return -ENODEV; + + /* kvzalloc because priv has grown large (slots[NR_SA] each with + * win[KNOD_SPSC_MAX] sliding windows - a couple of MB now). kzalloc + * may succeed but kvzalloc falls back to vmalloc if kmalloc can't + * find contiguous pages, which is safer under memory pressure. + */ + priv =3D kvzalloc_obj(*priv, GFP_KERNEL); + if (!priv) { + module_put(THIS_MODULE); + return -ENOMEM; + } + + priv->stats =3D alloc_percpu(struct knod_ipsec_stats); + if (!priv->stats) { + err =3D -ENOMEM; + goto err_priv; + } + + priv->knod =3D knodev->accel->priv; + if (!priv->knod) { + pr_err("knod_ipsec: no knod context (NOD not attached?)\n"); + err =3D -ENODEV; + goto err_stats; + } + priv->knodev =3D knodev; + mutex_init(&priv->slot_lock); + xa_init(&priv->spi_to_slot); + + /* Number of parallel dispatchers =3D number of AQL/SDMA queue pairs + * that the accel was allocated with, capped at the ipsec module's + * compile-time limit. Falls back to 1 when the knod layer reports + * 0 or the clamp leaves nothing usable. + */ + priv->nr_dispatchers =3D priv->knod->queue_cnt; + if (priv->nr_dispatchers > KNOD_IPSEC_MAX_DISPATCHERS) + priv->nr_dispatchers =3D KNOD_IPSEC_MAX_DISPATCHERS; + if (priv->nr_dispatchers < 1) + priv->nr_dispatchers =3D 1; + priv->pkt_batch =3D KNOD_IPSEC_PKT_BATCH; + + priv->t_tables =3D knod_gcm_alloc_tables(priv->knod); + if (IS_ERR(priv->t_tables)) { + err =3D PTR_ERR(priv->t_tables); + priv->t_tables =3D NULL; + goto err_ctx; + } + + adev =3D priv->knod->process->pdds[0]->dev->adev; + + if (adev->asic_type =3D=3D CHIP_VEGA10 || + adev->asic_type =3D=3D CHIP_VEGA20) { + priv->isa_version =3D 9; + priv->shader_size =3D knod_ipsec_init_shader_gfx9(priv->knod); + } else { + priv->isa_version =3D 10; + priv->shader_size =3D knod_ipsec_init_shader_gfx10(priv->knod); + } + + err =3D knod_ipsec_work_pool_alloc(priv); + if (err) { + knod_ipsec_work_pool_free(priv); + goto err_ttables; + } + + /* Persistent KAT scratch BO. Sized for the largest nr + * (KNOD_IPSEC_PKT_BATCH=3D64 bd slots at 64B stride, plus a matching + * per-packet dummy area at 128B stride for IPv6 ESP minimum). + * 4 pages =3D 16KB to fit max batch. Keeping it as a single long-lived + * BO avoids the per-KAT alloc/free dance that retriggers the + * multi-BO mapping bug. + */ + priv->kat_scratch =3D knod_alloc_mem(priv->knod, PAGE_SIZE * 4, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(priv->kat_scratch)) { + err =3D PTR_ERR(priv->kat_scratch); + priv->kat_scratch =3D NULL; + goto err_works; + } + + /* + * Allocate sa_table LAST. The GTT/VRAM multi-BO mapping bug + * (memory/gtt_multi_bo_bug.md) makes some BOs allocated mid-init fail + * to bind their backing pages at the returned gaddr even though the + * CPU kaddr is valid. Empirically, BOs allocated after all other init + * BOs (knod ctx, t_tables, shader init, work pool, kat_scratch) bind + * reliably. Allocating sa_table last sidesteps this. + */ + /* DIAGNOSTIC bisect step 1b: BO size is still ENLARGED, but rounded up + * to the next power-of-two (8 pages =3D 32 KB) instead of the natural + * 7 pages =3D 28 KB. Exact-7-page (0x7000) sa_table alloc empirically + * corrupts the subsequent dispatch (kernarg TCP fault at 0x01af5000); + * 6 pages (0x6000) is fine. Suspicion: amdgpu VRAM/GTT path handles + * non-power-of-two sizes differently for mappings of this scale. + */ + /* RX delivery desc rings (host kvmalloc; payload pages come from the + * framework pass_pool, no per-queue GTT BO). Set up before sa_table, + * which must stay the last BO bound on this KFD process VM per the + * "sa_table last" constraint in memory/gtt_multi_bo_bug.md. + */ + err =3D knod_ipsec_rxq_init_all(priv); + if (err) + goto err_kat_scratch; + + priv->sa_table =3D knod_alloc_mem(priv->knod, + ALIGN(KNOD_IPSEC_SA_BO_SIZE, + 8 * PAGE_SIZE), + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(priv->sa_table)) { + err =3D PTR_ERR(priv->sa_table); + priv->sa_table =3D NULL; + goto err_rxq; + } + memset(priv->sa_table->kaddr, 0, KNOD_IPSEC_SA_BO_SIZE); + pr_info("knod_ipsec: sa_table gaddr=3D0x%llx kaddr=3D%p size=3D%u\n", + priv->sa_table->gaddr, priv->sa_table->kaddr, + (u32)KNOD_IPSEC_SA_BO_SIZE); + + err =3D knod_ipsec_disp_create_all(priv); + if (err) + goto err_sa_table; + + ipsec_priv =3D priv; + knodev->post_copy =3D knod_ipsec_post_copy; + knod_ipsec_debugfs_init(priv); + pr_info("knod_ipsec: initialized on %s (gfx%d, single dispatcher)\n", + netdev_name(knodev->netdev), priv->isa_version); + return 0; + +err_sa_table: + knod_free_mem(priv->knod, priv->sa_table); + priv->sa_table =3D NULL; +err_rxq: + knod_ipsec_rxq_exit_all(priv); +err_kat_scratch: + knod_free_mem(priv->knod, priv->kat_scratch); +err_works: + knod_ipsec_work_pool_free(priv); +err_ttables: + knod_gcm_free_tables(priv->knod, priv->t_tables); +err_ctx: + priv->knod =3D NULL; +err_stats: + free_percpu(priv->stats); +err_priv: + kvfree(priv); + module_put(THIS_MODULE); + return err; +} + +static void knod_ipsec_nod_exit(struct knod_dev *knodev) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + int i; + + if (!priv) + return; + + knod_ipsec_debugfs_exit(priv); + + /* + * Publish the NULL and wait a grace period before freeing: the + * softirq RX reader (knod_ipsec_post_copy) samples ipsec_priv and must + * not touch it once it is freed. Runs under rtnl (feature_set/detach), + * so the synchronize_net() is expedited. + */ + knodev->post_copy =3D NULL; + WRITE_ONCE(ipsec_priv, NULL); + synchronize_net(); + + knod_ipsec_disp_destroy_all(priv); + + /* Free any leftover slots (should be none if xfrm tore down SAs). */ + mutex_lock(&priv->slot_lock); + for (i =3D 0; i < KNOD_IPSEC_NR_SA; i++) { + struct knod_ipsec_sa_slot *slot =3D &priv->slots[i]; + + if (!slot->active && !slot->key_mem) + continue; + if (slot->replay_mem) + knod_free_mem(priv->knod, slot->replay_mem); + if (slot->htable_mem) + knod_free_mem(priv->knod, slot->htable_mem); + if (slot->key_mem) + knod_free_mem(priv->knod, slot->key_mem); + memset(slot, 0, sizeof(*slot)); + } + xa_destroy(&priv->spi_to_slot); + mutex_unlock(&priv->slot_lock); + + knod_ipsec_work_pool_free(priv); + + knod_ipsec_rxq_exit_all(priv); + + if (priv->kat_scratch) { + knod_free_mem(priv->knod, priv->kat_scratch); + priv->kat_scratch =3D NULL; + } + if (priv->sa_table) + knod_free_mem(priv->knod, priv->sa_table); + if (priv->t_tables) + knod_gcm_free_tables(priv->knod, priv->t_tables); + /* knod context is owned by NOD core (knod_attach), don't free */ + priv->knod =3D NULL; + if (priv->stats) + free_percpu(priv->stats); + + kvfree(priv); + module_put(THIS_MODULE); + pr_info("knod_ipsec: torn down\n"); +} + +/* True while an offloaded xfrm SA is still bound to this accel. */ +static bool knod_ipsec_nod_busy(struct knod_dev *knodev) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + + if (!priv) + return false; + return !xa_empty(&priv->spi_to_slot); +} + +static int knod_ipsec_disp_create_all(struct knod_ipsec_priv *priv) +{ + int d, i; + + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + struct task_struct *p; + + p =3D kthread_create(knod_ipsec_dispatcher, disp, + "knod_ipsec-d%d", d); + if (IS_ERR(p)) { + pr_warn("knod_ipsec: failed to create dispatcher %d: %ld\n", + d, PTR_ERR(p)); + for (i =3D 0; i < d; i++) { + kthread_stop(priv->disp[i].kthread); + priv->disp[i].kthread =3D NULL; + } + return PTR_ERR(p); + } + kthread_park(p); + disp->kthread =3D p; + } + return 0; +} + +static void knod_ipsec_disp_destroy_all(struct knod_ipsec_priv *priv) +{ + int d; + + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + if (disp->kthread) { + kthread_stop(disp->kthread); + disp->kthread =3D NULL; + } + } +} + +static void knod_ipsec_nod_start(struct knod_dev *knodev) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + int base_rx, rem_rx; + int off_rx; + struct knod *knod; + int d, nr_rxq; + + if (!priv || READ_ONCE(priv->running)) + return; + + knod =3D priv->knod; + + /* Partition RX queues across dispatchers. Each disp + * owns a contiguous range of queues so the per-SPSC single- + * consumer invariant holds without cross-dispatcher locking. + * Leftover (when the total isn't a multiple of nr_dispatchers) + * lands on the first few disps, one extra each. + */ + nr_rxq =3D knodev->netdev ? knodev->netdev->real_num_rx_queues : 0; + if (nr_rxq > KNOD_SPSC_MAX) + nr_rxq =3D KNOD_SPSC_MAX; + if (nr_rxq > knod->channels) + nr_rxq =3D knod->channels; + if (nr_rxq < 0) + nr_rxq =3D 0; + + base_rx =3D nr_rxq / priv->nr_dispatchers; + rem_rx =3D nr_rxq % priv->nr_dispatchers; + off_rx =3D 0; + + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + int n_rx =3D base_rx + (d < rem_rx ? 1 : 0); + + disp->rxq_first =3D off_rx; + disp->rxq_count =3D n_rx; + off_rx +=3D n_rx; + } + + WRITE_ONCE(priv->running, true); + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + if (!disp->kthread) + continue; + kthread_unpark(disp->kthread); + pr_info("knod_ipsec: disp[%d] started rxq[%d..%d) kaql=3D%d\n", + d, disp->rxq_first, + disp->rxq_first + disp->rxq_count, + disp->kaql_idx); + } +} + +static void knod_ipsec_nod_stop(struct knod_dev *knodev) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + int d; + + if (!priv || !READ_ONCE(priv->running)) + return; + + WRITE_ONCE(priv->running, false); + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + if (!disp->kthread) + continue; + kthread_park(disp->kthread); + } +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 3: fused RX shader load + dispatch + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * + * The fused RX shader parses ESP, resolves the SA via SPI lookup, and + * (in follow-up work) runs AES-GCM decrypt/ICV. The resulting inner + * packet lives in VRAM; the CPU finish worker SDMA-copies it into a + * per-queue delivery-pool page and publishes a knod_pass_desc onto the + * framework pass_pending ring. Anti-replay and skb build happen in the NIC + * dd NAPI consumer - the shader does neither. See knod_ipsec.h for the + * full data-flow description. + */ + +static int knod_ipsec_init_shader_gfx9(struct knod *knod) +{ + struct compute_pgm_rsrc1 rsrc1 =3D {}; + struct compute_pgm_rsrc2 rsrc2 =3D {}; + struct kernel_descriptor *kd =3D knod->kernels[0]->kaddr; + struct code_properties props =3D {}; + int shader_size; + + memset(kd, 0, sizeof(*kd)); + kd->kernel_code_entry_byte_offset =3D 1024; + kd->group_segment_fixed_size =3D KNOD_GCM_T_TABLES_TOTAL; + + /* VGPRs: (gran+1)*4. Need v0-v42 (43 VGPRs) -> gran=3D12 -> 52. + * SGPRs: (gran+1)*8. Need s0-s59 (SR_RK2) -> gran=3D7 -> 64. + */ + rsrc1.granulated_workitem_vgpr_count =3D 12; + rsrc1.granulated_wavefront_sgpr_count =3D 7; + rsrc1.float_denorm_mode_32 =3D 3; + rsrc1.float_denorm_mode_16_64 =3D 3; + rsrc1.enable_dx10_clamp =3D 1; + rsrc1.enable_ieee_mode =3D 1; + + rsrc2.user_sgpr_count =3D 15; + rsrc2.enable_sgpr_workgroup_id_x =3D 1; + rsrc2.enable_sgpr_workgroup_id_y =3D 1; + rsrc2.enable_sgpr_workgroup_id_z =3D 1; + rsrc2.granulated_lds_size =3D 8; + + props.enable_sgpr_private_segment_buffer =3D 1; + props.enable_sgpr_dispatch_ptr =3D 1; + props.enable_sgpr_queue_ptr =3D 1; + props.enable_sgpr_kernarg_segment_ptr =3D 1; + props.enable_sgpr_dispatch_id =3D 1; + props.enable_sgpr_flat_scratch_init =3D 1; + props.enable_sgpr_private_segment_size =3D 1; + + memcpy(&kd->compute_pgm_rsrc1, &rsrc1, sizeof(rsrc1)); + memcpy(&kd->compute_pgm_rsrc2, &rsrc2, sizeof(rsrc2)); + memcpy(&kd->code_properties, &props, sizeof(props)); + + memset(knod->kernels[0]->kaddr + kd->kernel_code_entry_byte_offset, + 0, (PAGE_SIZE << 4) - kd->kernel_code_entry_byte_offset); + shader_size =3D kfd_ipsec_gen_fused_shader_gfx9( + knod->kernels[0]->kaddr + kd->kernel_code_entry_byte_offset); + pr_debug("knod_ipsec: GFX9 RX shader generated, %d bytes\n", + shader_size); + + return shader_size; +} + +static int knod_ipsec_init_shader_gfx10(struct knod *knod) +{ + struct compute_pgm_rsrc1 rsrc1 =3D {}; + struct compute_pgm_rsrc2 rsrc2 =3D {}; + struct kernel_descriptor *kd =3D knod->kernels[0]->kaddr; + struct code_properties props =3D {}; + int shader_size; + u32 rsrc1_raw; + + memset(kd, 0, sizeof(*kd)); + kd->kernel_code_entry_byte_offset =3D 1024; + kd->group_segment_fixed_size =3D KNOD_GCM_T_TABLES_TOTAL; + + /* VGPRs: Wave64, granularity=3D4. (12+1)*4 =3D 52 VGPRs. + * Shader uses v0-v42 (VR_SAVE_ESP_OFF). + * + * SGPRs: RDNA2 ignores granulated_wavefront_sgpr_count - + * SGPRs come from a flat 106-entry pool. Field is reserved + * and must be 0 (non-zero corrupts RSRC1 interpretation on + * some RDNA2 steppings, causing SQC inst-fetch faults). + */ + rsrc1.granulated_workitem_vgpr_count =3D 12; + rsrc1.granulated_wavefront_sgpr_count =3D 0; + rsrc1.float_denorm_mode_32 =3D 3; + rsrc1.float_denorm_mode_16_64 =3D 3; + rsrc1.enable_dx10_clamp =3D 1; + rsrc1.enable_ieee_mode =3D 1; + rsrc1.wgp_mode =3D 0; + rsrc1.mem_ordered =3D 1; + + rsrc2.user_sgpr_count =3D 15; + rsrc2.enable_sgpr_workgroup_id_x =3D 1; + rsrc2.enable_sgpr_workgroup_id_y =3D 1; + rsrc2.enable_sgpr_workgroup_id_z =3D 1; + rsrc2.granulated_lds_size =3D 8; + + props.enable_sgpr_private_segment_buffer =3D 1; + props.enable_sgpr_dispatch_ptr =3D 1; + props.enable_sgpr_queue_ptr =3D 1; + props.enable_sgpr_kernarg_segment_ptr =3D 1; + props.enable_sgpr_dispatch_id =3D 1; + props.enable_sgpr_flat_scratch_init =3D 1; + props.enable_sgpr_private_segment_size =3D 1; + + memcpy(&kd->compute_pgm_rsrc1, &rsrc1, sizeof(rsrc1)); + memcpy(&kd->compute_pgm_rsrc2, &rsrc2, sizeof(rsrc2)); + memcpy(&kd->code_properties, &props, sizeof(props)); + + memset(knod->kernels[0]->kaddr + kd->kernel_code_entry_byte_offset, + 0, (PAGE_SIZE << 4) - kd->kernel_code_entry_byte_offset); + shader_size =3D kfd_ipsec_gen_fused_shader_gfx10( + knod->kernels[0]->kaddr + kd->kernel_code_entry_byte_offset); + memcpy(&rsrc1_raw, &kd->compute_pgm_rsrc1, 4); + pr_info("knod_ipsec: GFX10 shader %d bytes, RSRC1=3D0x%08x (vgpr=3D%u sgp= r=3D%u wgp=3D%u mem=3D%u)\n", + shader_size, rsrc1_raw, + rsrc1_raw & 0x3F, + (rsrc1_raw >> 6) & 0xF, + (rsrc1_raw >> 29) & 1, + (rsrc1_raw >> 30) & 1); + + return shader_size; +} + +/* + * Single-dispatcher AQL machinery. + * + * One AQL queue (kaql[0]), one in-flight dispatch, one kthread that owns + * everything. The dispatcher drains per-queue NIC RX SPSC bd rings, build= s one + * dispatch, kicks the GPU, spins on the completion signal, finalises, and + * loops. No lock on the hot path. + * + * KAT paths park the dispatcher with kthread_park() for exclusive ownersh= ip + * of the queue and the work slot. + */ + +static void knod_ipsec_fill_dispatch(struct knod *knod, + struct knod_ipsec_work *work, + struct knod_dispatch_params *p) +{ + p->workgroup_size_x =3D 256; + p->grid_size_x =3D 256; + p->grid_size_y =3D max(work->nr_packets, 1); + p->private_segment_size =3D 0; + p->group_segment_size =3D KNOD_GCM_T_TABLES_TOTAL; + p->kernel_object =3D (u64)knod->kernels[0]->gaddr; + p->kernarg_address =3D work->param.gaddr; +} + +/* + * Prepare the dispatcher's single work slot to run the fused RX shader + * over `sub[0..nr)`. Must be called from dispatcher context only (or while + * the dispatcher is parked - e.g. from KAT). `bds` may be NULL for the + * in-kernel KAT path; in that case the KAT owns out_addr and this helper + * leaves it untouched. + */ +static void knod_ipsec_prepare_rx_dispatch(struct knod_ipsec_priv *priv, + struct knod_ipsec_work *work, + struct knod_ipsec_fused_sub *sub, + struct spsc_bd **bds, int nr, + struct napi_struct *napi, + int queue_idx) +{ + struct knod_ipsec_fused_param *param; + struct amd_signal *signal; + + if (nr > (int)READ_ONCE(priv->pkt_batch)) + nr =3D (int)READ_ONCE(priv->pkt_batch); + + param =3D (struct knod_ipsec_fused_param *)work->param.kaddr; + memset(param, 0, sizeof(*param)); + param->sa_table_addr =3D cpu_to_le64(priv->sa_table->gaddr); + param->t_tables_addr =3D cpu_to_le64(priv->t_tables->gaddr); + param->nr_sa =3D cpu_to_le32(KNOD_IPSEC_NR_SA); + memcpy(param->sub, sub, sizeof(sub[0]) * nr); + + /* Patch out_addr to point to the per-work decrypt output buffer + * instead of pkt_addr. AES-CTR writes plaintext here; GHASH reads + * the original ciphertext from pkt_addr. In-place would corrupt + * the ciphertext before GHASH could read it. + * + * Skip for KAT (bds=3D=3DNULL): the KAT manages out_addr itself. + */ + if (bds) { + int pi; + + for (pi =3D 0; pi < nr; pi++) + param->sub[pi].out_addr =3D cpu_to_le64( + work->rx_out_gaddr + + (u64)pi * KNOD_IPSEC_DECRYPT_PKT_SIZE); + } + + work->nr_packets =3D nr; + work->rx_napi =3D napi; + work->rx_queue_idx =3D queue_idx; + if (bds) { + int bi; + + for (bi =3D 0; bi < nr; bi++) { + work->rx_bds[bi] =3D bds[bi]; + /* Single-queue legacy/KAT path: all packets belong + * to queue_idx. Multi-queue dispatches populate + * rx_pkt_queue[] directly in try_rx and never call + * this function. + */ + work->rx_pkt_queue[bi] =3D (u8)queue_idx; + } + } else { + memset(work->rx_bds, 0, sizeof(work->rx_bds[0]) * nr); + memset(work->rx_pkt_queue, (u8)queue_idx, + sizeof(work->rx_pkt_queue[0]) * nr); + } + signal =3D (struct amd_signal *)priv->knod->kaql[0].queue_signal->kaddr; + work->sigval =3D READ_ONCE(signal->value); + if (static_branch_unlikely(&ipsec_stats_enabled_key)) + work->dispatch_ts =3D ktime_to_ns(ktime_get()); +} + +/* + * Legacy exported entry points. With the single-dispatcher architecture, + * NICs publish bds into knodev->wpriv[].spsc_bds and the dispatcher polls + * them directly - no NIC driver actually calls these anymore, but keep the + * exports so external out-of-tree builds don't break while they transitio= n. + */ +int knod_ipsec_rx_submit(struct knod_ipsec_fused_sub *sub, int nr, + struct napi_struct *napi, int queue_idx) +{ + return 0; +} +EXPORT_SYMBOL_GPL(knod_ipsec_rx_submit); + +int knod_ipsec_rx_submit_bds(struct knod_ipsec_fused_sub *sub, + struct spsc_bd **bds, int nr, + struct napi_struct *napi, int queue_idx) +{ + return 0; +} +EXPORT_SYMBOL_GPL(knod_ipsec_rx_submit_bds); + +/* + * Shader verdict sentinels (low-side) - kept in lockstep with + * ipsec_fused_gfx9.h. `bd->act` is packed as (low32=3Dsnapshot, high32=3D + * slot_idx|sentinel); we only read the high half here. + */ +#define KNOD_IPSEC_SHADER_VERDICT_MISS 0xFFFFFFFFu +#define KNOD_IPSEC_SHADER_VERDICT_BYPASS 0xFFFFFFFEu +#define KNOD_IPSEC_SHADER_VERDICT_ICV_FAIL 0xFFFFFFFDu + +/* + * The dispatcher fences each RX batch at the SDMA ring position that + * knod_sdma_submit() returned (a 32-bit dword cursor that wraps), so the + * completion test is a signed-32 compare against the signal's low word. + */ +static bool knod_ipsec_fence_passed(struct knod_ipsec_work *w) +{ + return (s32)((u32)READ_ONCE(*w->sdma_fence_ptr) - + w->sdma_fence_target) >=3D 0; +} + +/* Bounded spin until @w's RX-batch SDMA fence fires (teardown path only).= */ +static void knod_ipsec_fence_wait(struct knod_ipsec_work *w) +{ + int timeout =3D 100000; + + while (!knod_ipsec_fence_passed(w)) { + if (--timeout <=3D 0) + break; + cpu_relax(); + } +} + +/* + * RX finish: GPU is done writing verdicts for this batch. For every + * packet that the shader flagged as a hit, SDMA-copy the decrypted + * inner payload from VRAM to a per-queue delivery-pool (pass_pool) page, + * then push a knod_pass_desc onto the framework pass_pending ring so the + * NIC dd NAPI picks it up via knod_d2h_drain() + knod_ipsec_post_copy(). + * + * Packets with a miss/bypass/malformed verdict never reach the pending + * ring; they just get accounted here and the bd is recycled by the + * NIC dd through its normal bd ring path (unchanged). + */ +static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *dis= p, + struct knod_ipsec_work *work) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + struct knod_ipsec_fused_param *param; + struct knod_ipsec_stats *s =3D NULL; + struct knod *knod =3D priv->knod; + int sdma_idx =3D disp->kaql_idx; + struct knod_sdma *sdma_q =3D &knod->sdma[sdma_idx]; + struct knod_sdma_copy_desc copies[2]; + u32 batch_fence =3D 0; + int i; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + int n_sdma =3D 0; + u32 sdma_copies =3D 0; + u32 sdma_bytes =3D 0; + int pkt_idx_of[KNOD_IPSEC_PKT_BATCH]; + struct knod_ipsec_rx_pending *pending =3D work->rx_pending; + + if (stats_on) + s =3D this_cpu_ptr(priv->stats); + + param =3D (struct knod_ipsec_fused_param *)work->param.kaddr; + + /* Step 1: classify + schedule SDMA for hits. Each packet in the + * batch may belong to a different NIC RX queue, so rxq routing + * happens per-packet via work->rx_pkt_queue[i] -> priv->rxq[]. + */ + for (i =3D 0; i < work->nr_packets; i++) { + struct knod_ipsec_sa_slot *sa_slot; + struct spsc_bd *bd =3D work->rx_bds[i]; + struct page_pool *pool; + netmem_ref netmem; + u64 pkt_gaddr, out_gaddr; + u64 dst_base; + u8 pkt_next_hdr; + u32 verdict_hi; + u32 inner_len; + u32 pend_inner_len; + u32 fv; + int ncopy; + u8 pkt_family; + u8 pkt_mode; + unsigned int rxq_idx =3D work->rx_pkt_queue[i]; + + if (!bd) + continue; + if ((int)rxq_idx >=3D priv->nr_rxq) + continue; + pool =3D READ_ONCE(priv->knodev->wpriv[rxq_idx].pass_pool); + + verdict_hi =3D (u32)(bd->act >> 32); + + if (verdict_hi =3D=3D KNOD_IPSEC_SHADER_VERDICT_MISS || + verdict_hi =3D=3D KNOD_IPSEC_SHADER_VERDICT_BYPASS) { + u32 raw_len =3D bd->len; + u64 pkt_gaddr; + u64 dst_gaddr; + + if (raw_len =3D=3D 0 || raw_len > PAGE_SIZE) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-A bypass raw_len=3D%u pkt[%d]= act=3D0x%llx q%u\n", + raw_len, i, bd->act, rxq_idx); + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + netmem =3D pool ? page_pool_dev_alloc_netmems(pool) : 0; + if (!netmem) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-bypass-nomem pkt[%d] raw_len= =3D%u q%u\n", + i, raw_len, rxq_idx); + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + pkt_gaddr =3D le64_to_cpu(param->sub[i].pkt_addr); + dst_gaddr =3D page_pool_get_dma_addr_netmem(netmem); + copies[0].dst =3D dst_gaddr; + copies[0].src =3D pkt_gaddr; + copies[0].len =3D raw_len; + fv =3D knod_sdma_submit(knod, sdma_idx, copies, 1); + if (!fv) { + page_pool_put_full_netmem(pool, netmem, false); + if (stats_on) + s->rx_drop_sdma_full++; + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + batch_fence =3D fv; + sdma_copies++; + sdma_bytes +=3D raw_len; + + pending[n_sdma].inner_len =3D raw_len; + pending[n_sdma].netmem =3D netmem; + pending[n_sdma].sa_slot =3D KNOD_IPSEC_NR_SA; + pending[n_sdma].rxq_idx =3D (u16)rxq_idx; + pending[n_sdma].mode =3D 0; + pending[n_sdma].next_hdr =3D 0; + pending[n_sdma].family =3D 0; + pending[n_sdma].inner_off =3D 0; + pkt_idx_of[n_sdma] =3D i; + n_sdma++; + + bd->act =3D KNOD_IPSEC_PASS; + continue; + } + if (verdict_hi =3D=3D KNOD_IPSEC_SHADER_VERDICT_ICV_FAIL) { + if (stats_on) + s->rx_drop_icv++; + pr_warn_ratelimited("knod_ipsec: RX ICV fail pkt[%d] act=3D0x%llx len= =3D%u off=3D%u\n", + i, bd->act, bd->len, bd->off); + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + if (verdict_hi >=3D KNOD_IPSEC_NR_SA) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-B verdict_hi=3D%u pkt[%d] act= =3D0x%llx len=3D%u q%u\n", + verdict_hi, i, bd->act, bd->len, rxq_idx); + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + + inner_len =3D bd->len; + if (inner_len =3D=3D 0 || inner_len > PAGE_SIZE) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-C inner_len=3D%u pkt[%d] act= =3D0x%llx slot=3D%u q%u\n", + inner_len, i, bd->act, verdict_hi, rxq_idx); + continue; + } + + pkt_mode =3D bd->off & 0xFF; + pkt_next_hdr =3D (bd->off >> 8) & 0xFF; + + sa_slot =3D &priv->slots[verdict_hi]; + pkt_family =3D sa_slot->x ? sa_slot->x->props.family : AF_INET; + + netmem =3D pool ? page_pool_dev_alloc_netmems(pool) : 0; + if (!netmem) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-decrypt-nomem pkt[%d] slot=3D%= u inner_len=3D%u q%u\n", + i, verdict_hi, inner_len, rxq_idx); + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + pkt_gaddr =3D le64_to_cpu(param->sub[i].pkt_addr); + out_gaddr =3D le64_to_cpu(param->sub[i].out_addr); + dst_base =3D page_pool_get_dma_addr_netmem(netmem); + + if (pkt_mode =3D=3D XFRM_MODE_TRANSPORT) { + u32 l3_hdr_len =3D (pkt_family =3D=3D AF_INET6) ? 40 : 20; + + copies[0].dst =3D dst_base + l3_hdr_len; + copies[0].src =3D out_gaddr; + copies[0].len =3D inner_len; + copies[1].dst =3D dst_base; + copies[1].src =3D pkt_gaddr + 14; + copies[1].len =3D l3_hdr_len; + ncopy =3D 2; + pend_inner_len =3D l3_hdr_len + inner_len; + } else { + copies[0].dst =3D dst_base + KNOD_IPSEC_GTT_OUT_L3_OFF; + copies[0].src =3D out_gaddr; + copies[0].len =3D inner_len; + ncopy =3D 1; + pend_inner_len =3D inner_len; + } + + fv =3D knod_sdma_submit(knod, sdma_idx, copies, ncopy); + if (!fv) { + page_pool_put_full_netmem(pool, netmem, false); + if (stats_on) + s->rx_drop_sdma_full++; + bd->act =3D KNOD_IPSEC_DROP; + continue; + } + batch_fence =3D fv; + sdma_copies +=3D ncopy; + sdma_bytes +=3D pend_inner_len; + + pending[n_sdma].inner_len =3D pend_inner_len; + pending[n_sdma].netmem =3D netmem; + pending[n_sdma].sa_slot =3D verdict_hi; + pending[n_sdma].rxq_idx =3D (u16)rxq_idx; + pending[n_sdma].mode =3D pkt_mode; + pending[n_sdma].next_hdr =3D pkt_next_hdr; + pending[n_sdma].family =3D pkt_family; + pending[n_sdma].inner_off =3D + (pkt_mode !=3D XFRM_MODE_TRANSPORT) ? + KNOD_IPSEC_GTT_OUT_L3_OFF : 0; + pkt_idx_of[n_sdma] =3D i; + n_sdma++; + } + + if (stats_on) { + work->rx_sdma_copies =3D sdma_copies; + work->rx_sdma_bytes =3D sdma_bytes; + } + + /* Step 2: FENCE + doorbell. knod_sdma_submit() emitted the copies + * above; kick fences the ring position it returned (batch_fence). + */ + if (n_sdma =3D=3D 0) + return; + knod_sdma_kick(knod, sdma_idx); + work->sdma_fence_target =3D batch_fence; + + work->sdma_fence_ptr =3D (s64 *)&((struct amd_signal *) + sdma_q->queue_signal->kaddr)->value; + work->sdma_submit_ns =3D stats_on ? ktime_get_ns() : 0; + work->n_sdma_pending =3D n_sdma; + for (i =3D 0; i < n_sdma; i++) + work->sdma_pkt_idx_of[i] =3D (u16)pkt_idx_of[i]; +} + +/* + * Deferred SDMA completion: publish to pass_pending and mark bds PASS. + * + * Called from the dispatcher loop once work->sdma_fence_ptr shows the + * SDMA fence has fired. All SDMA copies have landed in the delivery pages + * so the drain path can now read from them. + */ +static void knod_ipsec_finish_rx_complete(struct knod_ipsec_dispatcher *di= sp, + struct knod_ipsec_work *work) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + struct knod_ipsec_fused_param *param =3D + (struct knod_ipsec_fused_param *)work->param.kaddr; + struct knod_ipsec_rx_pending *pending =3D work->rx_pending; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + struct knod_ipsec_stats *s =3D NULL; + int n_sdma =3D work->n_sdma_pending; + int i; + + if (stats_on) { + s =3D this_cpu_ptr(priv->stats); + if (work->sdma_submit_ns) + work->sdma_wait_ns =3D + ktime_get_ns() - work->sdma_submit_ns; + } + + /* Publish each decrypted packet onto its RX queue's common pending + * ring; knod_d2h_drain delivers it through knod_ipsec_post_copy. The + * SDMA fence already fired (the dispatcher waited), so the descriptor + * is tagged already-landed and the drain's fence check is a no-op. + */ + for (i =3D 0; i < n_sdma; i++) { + u16 rxq =3D pending[i].rxq_idx; + struct knod_work_priv *wpriv =3D &priv->knodev->wpriv[rxq]; + int src_j =3D work->sdma_pkt_idx_of[i]; + struct knod_pass_desc *d; + + if (spsc_produce(&wpriv->pass_pending, (void **)&d)) { + struct page_pool *pool =3D READ_ONCE(wpriv->pass_pool); + + /* ring full: return the undelivered page */ + if (pool) + page_pool_put_full_netmem(pool, + pending[i].netmem, + false); + if (stats_on) + s->rx_drop_desc_full++; + continue; + } + d->netmem =3D pending[i].netmem; + d->src =3D 0; /* NIC act handler recycles the RX bd */ + d->len =3D pending[i].inner_len; + d->off =3D pending[i].inner_off; + d->fence_val =3D work->sdma_fence_target; + d->sdma_idx =3D disp->kaql_idx; + d->feat.sa_slot =3D pending[i].sa_slot; + if (pending[i].sa_slot < KNOD_IPSEC_NR_SA) { + struct knod_ipsec_sa_slot *sa =3D + &priv->slots[pending[i].sa_slot]; + + d->feat.seq_lo =3D + le32_to_cpu(param->sub[src_j].result_seq); + d->feat.seq_hi =3D (sa->x && sa->x->replay_esn) ? + sa->x->replay_esn->seq_hi : 0; + } else { + d->feat.seq_lo =3D 0; + d->feat.seq_hi =3D 0; + } + d->feat.mode =3D pending[i].mode; + d->feat.next_hdr =3D pending[i].next_hdr; + d->feat.family =3D pending[i].family; + spsc_produce_commit(&wpriv->pass_pending); + } + + if (stats_on) { + s->rx_packets +=3D n_sdma; + s->finish_produced +=3D n_sdma; + } + + /* Mark all bds as KNOD_IPSEC_PASS so the NIC act_handler recycles + * the netmem on the next NAPI poll. + */ + for (i =3D 0; i < work->nr_packets; i++) { + if (work->rx_bds[i]) + work->rx_bds[i]->act =3D KNOD_IPSEC_PASS; + } + memset(work->rx_bds, 0, + sizeof(work->rx_bds[0]) * work->nr_packets); + + work->n_sdma_pending =3D 0; +} + +/* + * NIC dd NAPI consumer: drain the per-queue host desc ring, run the + * RFC 4303 sliding window check on each descriptor, and deliver the + * inner packet up the stack as a zero-copy head_frag skb wrapping the + * delivery-pool page (knod_pass_build_skb); the page recycles to the + * pool on skb free. + * + * Safe to call before knod_ipsec_priv has been created: returns 0. + * NIC drivers can therefore wire this call unconditionally from their + * NAPI poll without extra gating. + * + * Ordering w.r.t. the bd recycle ring: the NIC driver should drain its + * bd ring *first* (to recycle netmem back into the page_pool) and then + * call drain_rx. The two rings are independent - bd ring delivers no + * verdicts, desc ring carries only PASS candidates. + */ +/* + * Feature delivery hook (knod_dev->post_copy) for the ipsec RX path. + * knod_d2h_drain has already built the head_frag skb wrapping the delivery + * page; here we run the SA lookup, RFC 4303 anti-replay window, cleartext + * L3 fix-up and secpath attach. Returns false to drop the packet (the + * drain frees the skb, which recycles the page). + */ +static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *= skb, + const struct knod_pass_desc *desc, + int queue_idx) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + struct knod_ipsec_stats *s =3D NULL; + struct knod_ipsec_sa_slot *slot; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + u32 sa_slot =3D desc->feat.sa_slot; + u64 seq; + + if (stats_on) + s =3D this_cpu_ptr(priv->stats); + + /* Raw bypass: packet was not IPsec (ARP, non-ESP IP, ESP with an + * unknown SPI). The page holds the full Ethernet frame; deliver it + * with no IPsec state. + */ + if (sa_slot =3D=3D KNOD_IPSEC_NR_SA) { + skb->dev =3D knodev->netdev; + skb->protocol =3D eth_type_trans(skb, knodev->netdev); + skb_reset_network_header(skb); + return true; + } + if (sa_slot > KNOD_IPSEC_NR_SA) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-pc-saslot sa_slot=3D%u len=3D%u= q%d\n", + sa_slot, desc->len, queue_idx); + return false; + } + + slot =3D &priv->slots[sa_slot]; + if (!slot->active) { + if (stats_on) + s->rx_drop_no_sa++; + return false; + } + + seq =3D ((u64)desc->feat.seq_hi << 32) | (u64)desc->feat.seq_lo; + if (!knod_ipsec_sa_window_check(&slot->win[queue_idx], seq)) { + if (stats_on) + s->rx_drop_replay++; + return false; + } + + /* skb->data is the decrypted inner packet (head_frag at inner_off). */ + if (desc->feat.mode =3D=3D XFRM_MODE_TRANSPORT) { + /* Transport: the page holds the outer L3 header (20B IPv4 or + * 40B IPv6) followed by the decrypted payload; patch the + * next-header and length fields to describe the cleartext. + */ + if (desc->feat.family =3D=3D AF_INET6) { + struct ipv6hdr *ip6h; + + if (desc->len < 40) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-pc-v6len len=3D%u q%d\n", + desc->len, queue_idx); + return false; + } + ip6h =3D (struct ipv6hdr *)skb->data; + ip6h->nexthdr =3D desc->feat.next_hdr; + ip6h->payload_len =3D htons(desc->len - 40); + skb->protocol =3D htons(ETH_P_IPV6); + } else { + struct iphdr *iph; + + if (desc->len < 20) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-pc-v4len len=3D%u q%d\n", + desc->len, queue_idx); + return false; + } + iph =3D (struct iphdr *)skb->data; + iph->protocol =3D desc->feat.next_hdr; + iph->tot_len =3D htons(desc->len); + iph->check =3D 0; + iph->check =3D ip_fast_csum((u8 *)iph, iph->ihl); + skb->protocol =3D htons(ETH_P_IP); + } + } else { + /* Tunnel: decrypted payload is a bare inner L3 packet; infer + * v4/v6 from the IP version nibble. + */ + u8 first =3D *(const u8 *)skb->data; + + if ((first >> 4) =3D=3D 4) { + skb->protocol =3D htons(ETH_P_IP); + } else if ((first >> 4) =3D=3D 6) { + skb->protocol =3D htons(ETH_P_IPV6); + } else { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-pc-tunver first=3D0x%02x len= =3D%u off=3D%u q%d\n", + first, desc->len, desc->off, queue_idx); + return false; + } + } + + skb->dev =3D knodev->netdev; + skb_reset_mac_header(skb); + skb_reset_network_header(skb); + + /* The inner L4 checksum was computed by the sender before encryption + * and ESP authentication guarantees the decrypted payload is byte- + * identical, so trust it instead of recomputing (saves ~1.8% CPU at + * 45+ Gbps UDP). + */ + skb->ip_summed =3D CHECKSUM_UNNECESSARY; + + /* Attach secpath so xfrm_policy_check() recognises this packet as + * decrypted by an offload engine; without it the inbound policy + * drops the cleartext. + */ + if (slot->x) { + struct sec_path *sp; + struct xfrm_offload *xo; + + sp =3D secpath_set(skb); + if (unlikely(!sp)) { + if (stats_on) + s->rx_drop_malformed++; + pr_warn_ratelimited("knod_ipsec: malform-pc-secpath len=3D%u q%d\n", + desc->len, queue_idx); + return false; + } + xfrm_state_hold(slot->x); + sp->xvec[sp->len++] =3D slot->x; + sp->olen++; + + xo =3D xfrm_offload(skb); + xo->flags =3D CRYPTO_DONE; + xo->status =3D CRYPTO_SUCCESS; + } + + if (stats_on) { + s->rx_bytes +=3D desc->len; + s->drain_delivered++; + } + return true; +} + +/* + * After the GPU completion signal has fired for `work`, finalise it: + * SDMA-copy decrypted RX inner packets into per-queue delivery-pool pages + * and publish them on the host desc ring. Called only from dispatcher + * context. No lock needed: the dispatcher is the sole accessor. + */ +static void knod_ipsec_finalise_work(struct knod_ipsec_dispatcher *disp, + struct knod_ipsec_work *work) +{ + knod_ipsec_finish_rx_deliver(disp, work); + + if (work->rx_napi) { + napi_schedule(work->rx_napi); + work->rx_napi =3D NULL; + } + /* rx_bds[] clear is deferred to finish_rx_complete / + * finalise_sdma_done for the SDMA path, because the bd + * pointers are needed to mark bd->act =3D PASS after SDMA + * completes. For the no-SDMA path (n_sdma_pending =3D=3D 0), + * finalise_inflight clears them inline before returning + * EMPTY. + */ + if (work->n_sdma_pending =3D=3D 0) + memset(work->rx_bds, 0, + sizeof(work->rx_bds[0]) * work->nr_packets); + + /* Timing stats are now accumulated by the dispatcher + * (try_rx) which has visibility into all phases - + * build / gpu / sdma / finalise. This function no longer + * touches *_gpu_ns. + */ +} + +/* + * Kick the GPU using kaql[disp->kaql_idx]. Assigns a forward-looking + * sigval by decrementing disp->dispatch_sigval_next so each in-flight + * slot has a distinct target - required for depth-N pipelining where + * multiple dispatches are queued on the same AQL queue and GPU + * decrements signal->value by 1 per completion. + * + * Submit-only: does not wait. The caller is responsible for polling + * knod_ipsec_dispatch_done() or blocking via knod_ipsec_dispatch_wait(). + */ +static void knod_ipsec_dispatch_submit(struct knod_ipsec_dispatcher *disp, + struct knod_ipsec_work *work) +{ + struct knod_dispatch_params p; + + disp->dispatch_sigval_next--; + work->sigval =3D disp->dispatch_sigval_next; + knod_ipsec_fill_dispatch(disp->priv->knod, work, &p); + knod_setup_header(disp->priv->knod, &p, disp->kaql_idx); +} + +/* + * Non-blocking completion check. Returns true if the kernel dispatch + * for `work` has finished. Used by the pipelined dispatcher loop to + * poll in-flight slots without stalling the build side. + */ +static bool knod_ipsec_dispatch_done(struct knod_ipsec_dispatcher *disp, + struct knod_ipsec_work *work) +{ + struct amd_signal *signal =3D + (struct amd_signal *)disp->priv->knod->kaql[disp->kaql_idx] + .queue_signal->kaddr; + + return READ_ONCE(signal->value) <=3D work->sigval; +} + +/* + * Blocking wait. Synchronous path used by the KAT selftest where we + * don't want pipelining. Spin-waits with cpu_relax and a best-effort + * timeout. In production, the dispatcher uses the non-blocking + * dispatch_done() check instead. + */ +static void knod_ipsec_dispatch_wait(struct knod_ipsec_dispatcher *disp, + struct knod_ipsec_work *work) +{ + struct amd_signal *signal =3D + (struct amd_signal *)disp->priv->knod->kaql[disp->kaql_idx] + .queue_signal->kaddr; + int timeout =3D 1000000; + + while (READ_ONCE(signal->value) > work->sigval) { + if (--timeout <=3D 0) { + pr_warn_ratelimited("knod_ipsec: GPU signal timeout nr=3D%d\n", + work->nr_packets); + return; + } + cpu_relax(); + } +} + +/* + * Convenience wrapper: submit + block until done. Keeps the existing + * KAT / selftest call sites that expect a synchronous dispatch. + */ +static void knod_ipsec_dispatch_and_wait(struct knod_ipsec_dispatcher *dis= p, + struct knod_ipsec_work *work) +{ + knod_ipsec_dispatch_submit(disp, work); + knod_ipsec_dispatch_wait(disp, work); +} + +/* + * Multi-queue RX drain into one GPU dispatch. + * + * For each NIC RX queue round-robin'd from *rx_rr, peek up to + * `cap_per_q =3D KNOD_IPSEC_PKT_BATCH / nr_queues` packets and stage them + * directly into the fused-shader kernarg sub[] array. This gives strict + * per-queue fairness: no single queue can monopolise the batch even if + * it has 10k packets backed up. When cap_per_q < 1 we fall back to 1 so + * that PKT_BATCH < nr_queues setups still drain something per queue. + * + * All queues drained in one dispatch must share the same SA table, but + * their packets may be for different SAs (each sub[].pkt_addr is an + * independent ESP frame with its own SPI); the shader performs the SPI + * scan per workgroup. Per-packet rx_pkt_queue[i] is recorded so + * finish_rx_deliver() routes each decrypted packet to the matching + * priv->rxq[q] delivery pool + desc_ring. + * + * NAPIs for every queue we touched are scheduled at the end so the + * driver-side act_handler sees the INFLIGHT -> PASS/DROP transition and + * recycles the netmem pages. + */ +static bool knod_ipsec_dispatcher_try_rx(struct knod_ipsec_dispatcher *dis= p, + struct knod_ipsec_work *work) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + struct knod_dev *knodev =3D priv->knodev; + struct knod_ipsec_fused_param *param; + struct { + u16 q; + u16 count; + } per_q[KNOD_SPSC_MAX]; + int nr_queues, nr_disp_queues, i, q =3D -1, start; + int active, pass, pi; + unsigned int total_n =3D 0; + unsigned int cap_per_q; + int per_q_n =3D 0; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + u64 t_start =3D 0, t_build_end =3D 0; + + if (!knodev || !priv->knod || !priv->knod->buf) + return false; + + if (stats_on) { + t_start =3D ktime_get_ns(); + work->sdma_wait_ns =3D 0; + work->rx_sdma_copies =3D 0; + work->rx_sdma_bytes =3D 0; + } + + nr_queues =3D knodev->netdev ? knodev->netdev->real_num_rx_queues : 0; + if (nr_queues > KNOD_SPSC_MAX) + nr_queues =3D KNOD_SPSC_MAX; + if (nr_queues > priv->knod->channels) + nr_queues =3D priv->knod->channels; + if (nr_queues <=3D 0) + return false; + + /* This dispatcher's window into the global queue set. Everything + * below indexes with `disp->rxq_first + (rr_offset % nr_disp_queues)` + * so dispatcher N only ever touches queues it owns. + */ + nr_disp_queues =3D disp->rxq_count; + if (nr_disp_queues > nr_queues - disp->rxq_first) + nr_disp_queues =3D nr_queues - disp->rxq_first; + if (nr_disp_queues <=3D 0) + return false; + + /* Count active (non-empty) queues within this dispatcher's range + * so the fair-share cap reflects actual demand. Without this, a + * single-flow iperf3 that only populates 1 out of N RX queues + * would be limited to BATCH/N per dispatch. + */ + active =3D 0; + for (i =3D 0; i < nr_disp_queues; i++) { + int qi =3D disp->rxq_first + + ((disp->rx_rr + i) % nr_disp_queues); + struct knod_work_priv *wp =3D &knodev->wpriv[qi]; + struct spsc_ring *rr =3D &wp->spsc_bds; + + if (!wp->napi || !rr->slots || rr->mask =3D=3D 0) + continue; + if (!priv->knod->buf[qi]) + continue; + if (spsc_count(rr) > 0) + active++; + } + if (active =3D=3D 0) + return false; + cap_per_q =3D DIV_ROUND_UP(READ_ONCE(priv->pkt_batch), active); + + /* Build fused_param directly into kernarg. Zero the entire struct + * so stale sub[batch_n..BATCH-1] entries from previous dispatches + * cannot be picked up by a GPU kernarg prefetch - the GFX9 CP may + * speculatively read beyond grid_size_y into the kernarg buffer, + * and a stale sub[].pkt_addr pointing at valid VRAM could cause + * the shader to process garbage packets (observed as ICV failures + * when the memset was removed). + */ + param =3D (struct knod_ipsec_fused_param *)work->param.kaddr; + memset(param, 0, sizeof(*param)); + param->sa_table_addr =3D cpu_to_le64(priv->sa_table->gaddr); + param->t_tables_addr =3D cpu_to_le64(priv->t_tables->gaddr); + param->nr_sa =3D cpu_to_le32(KNOD_IPSEC_NR_SA); + + /* Two-pass fair drain. + * + * Pass 1 caps each queue at cap_per_q (=3D BATCH / active_queues) so + * one queue cannot monopolise the batch when several queues have + * traffic. + * + * Pass 2 fills any remaining batch budget greedily from queues + * that still have packets. This matters for single-flow iperf3 + * where only one RX queue is active: pass 1 fills that queue up + * to its fair share (=3D BATCH for active=3D1) and pass 2 is a no-op, + * but with a few active queues pass 2 picks up the slack left by + * queues that had fewer than their fair share. + * + * IMPORTANT: we advance the SPSC acquired cursor IMMEDIATELY after + * each per-queue peek so that a subsequent pass re-visiting the + * same queue never re-reads the same packets. Earlier naive 2-pass + * without this acquire double-processed packets and caused ~50% + * anti-replay drops downstream. + */ + start =3D disp->rx_rr; + for (pass =3D 0; pass < 2; pass++) { + unsigned int per_q_cap =3D (pass =3D=3D 0) ? + cap_per_q : READ_ONCE(priv->pkt_batch); + + for (i =3D 0; i < nr_disp_queues; i++) { + struct knod_work_priv *wpriv; + struct spsc_ring *r; + u64 base_gaddr; + unsigned int cnt =3D 0; + unsigned int acq_start; + unsigned int remaining, budget; + int err, k, pi; + bool found; + + q =3D disp->rxq_first + + ((start + i) % nr_disp_queues); + wpriv =3D &knodev->wpriv[q]; + r =3D &wpriv->spsc_bds; + + if (!wpriv->napi || !r->slots || r->mask =3D=3D 0) + continue; + if (!priv->knod->buf[q]) + continue; + + remaining =3D READ_ONCE(priv->pkt_batch) - total_n; + if (remaining =3D=3D 0) + break; + budget =3D min(per_q_cap, remaining); + + base_gaddr =3D priv->knod->buf[q]->gaddr; + acq_start =3D r->acquired; + + err =3D spsc_peek(r, (void **)&work->rx_bds[total_n], + budget, &cnt); + if (err || cnt =3D=3D 0) + continue; + + /* Delivery pages are allocated lazily at finish time + * from the framework page_pool; no per-packet slot + * reservation here. + */ + for (k =3D 0; k < (int)cnt; k++) { + struct spsc_bd *bd =3D work->rx_bds[total_n + k]; + u32 ring_idx =3D (acq_start + k) & r->mask; + u64 pkt_addr, bd_gaddr, out_gaddr; + unsigned int si =3D total_n + k; + + pkt_addr =3D base_gaddr + + ((u64)bd->page_idx << PAGE_SHIFT) + + bd->off; + bd_gaddr =3D wpriv->spsc_pool_gaddr + + (u64)ring_idx * r->elem_stride; + + out_gaddr =3D work->rx_out_gaddr + + (u64)si * + KNOD_IPSEC_DECRYPT_PKT_SIZE; + + param->sub[si].pkt_addr =3D + cpu_to_le64(pkt_addr); + param->sub[si].out_addr =3D + cpu_to_le64(out_gaddr); + param->sub[si].bd_addr =3D + cpu_to_le64(bd_gaddr); + param->sub[si].pkt_len =3D + cpu_to_le32(bd->len); + param->sub[si].result_seq =3D 0; + + work->rx_pkt_queue[si] =3D (u8)q; + bd->act =3D KNOD_IPSEC_INFLIGHT; + } + + /* Advance r->acquired now so pass 2 never + * re-peeks the same entries. + */ + spsc_acquire(r, NULL, cnt, NULL); + + found =3D false; + for (pi =3D 0; pi < per_q_n; pi++) { + if (per_q[pi].q =3D=3D (u16)q) { + per_q[pi].count +=3D (u16)cnt; + found =3D true; + break; + } + } + if (!found) { + per_q[per_q_n].q =3D (u16)q; + per_q[per_q_n].count =3D (u16)cnt; + per_q_n++; + } + total_n +=3D cnt; + } + + if (total_n >=3D READ_ONCE(priv->pkt_batch)) + break; + } + + if (total_n =3D=3D 0) + return false; + + if (static_branch_unlikely(&ipsec_stats_enabled_key)) { + struct knod_ipsec_stats *cs =3D this_cpu_ptr(priv->stats); + + cs->rx_peek_total +=3D total_n; + } + + /* spsc_acquire was already called per-queue inside the drain loop + * to prevent the two-pass logic from re-peeking the same entries, + * so there is nothing to commit here. + */ + + work->nr_packets =3D (int)total_n; + /* Multi-queue: no single napi. finalise_work will skip its + * rx_napi kick and we schedule per-queue napis below. + */ + work->rx_napi =3D NULL; + work->rx_queue_idx =3D per_q[0].q; /* used only by single-queue KAT */ + + if (stats_on) { + t_build_end =3D ktime_get_ns(); + work->t_build_end =3D t_build_end; + } + + if (stats_on) { + struct knod_ipsec_stats *s =3D this_cpu_ptr(priv->stats); + + s->rx_dispatches++; + s->rx_batch_total +=3D total_n; + if ((u64)total_n > s->rx_batch_max) + s->rx_batch_max =3D total_n; + } + + /* Stash per-queue drain bookkeeping so the deferred finalise + * path (called when the dispatch completes, possibly in a later + * dispatcher iteration) can wake the touched NAPIs and attribute + * stats back to this slot. + */ + work->per_q_n =3D per_q_n; + for (pi =3D 0; pi < per_q_n; pi++) + work->per_q_touched[pi] =3D per_q[pi].q; + work->t_build_start =3D t_start; + work->t_submit =3D stats_on ? ktime_get_ns() : 0; + + param->sdma_ring_addr =3D 0; + param->sdma_ctl_addr =3D 0; + + /* Submit-only: do not wait, do not finalise. The dispatcher loop + * polls knod_ipsec_dispatch_done() and runs finalise via + * knod_ipsec_finalise_inflight() once the completion signal fires. + */ + knod_ipsec_dispatch_submit(disp, work); + + disp->rx_rr =3D nr_disp_queues > 0 + ? ((q - disp->rxq_first + 1) % nr_disp_queues) + : 0; + return true; +} + +/* + * Deferred finalise for a slot whose GPU dispatch has completed (signal + * fired). Runs finish_rx_deliver(), wakes per-queue + * napis tracked by try_rx, and accumulates per-phase timing stats. + * Called from the pipelined dispatcher loop once + * knod_ipsec_dispatch_done() returns true for the slot. + */ +/* + * Called when GPU dispatch completes. For RX with SDMA copies, this + * transitions to SDMA_PENDING (the dispatcher loop polls the fence + * and calls knod_ipsec_finalise_sdma_done when ready). For RX without + * SDMA, completes everything inline and returns true so the + * caller can transition directly to EMPTY. + * + * Returns true if fully done (-> EMPTY), false if -> SDMA_PENDING. + */ +static bool knod_ipsec_finalise_inflight(struct knod_ipsec_dispatcher *dis= p, + struct knod_ipsec_work *work) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + struct knod_dev *knodev =3D priv->knodev; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + u64 t_gpu_end =3D 0, t_end =3D 0; + int i; + + if (stats_on) + t_gpu_end =3D ktime_get_ns(); + + work->t_finalise_start =3D t_gpu_end; + knod_ipsec_finalise_work(disp, work); + + /* RX path with SDMA copies pending - defer completion until the + * SDMA fence fires. The dispatcher loop will poll sdma_fence_ptr + * and call knod_ipsec_finalise_sdma_done(). + */ + if (work->n_sdma_pending > 0) { + if (stats_on) { + struct knod_ipsec_stats *s =3D this_cpu_ptr(priv->stats); + u64 build_ns =3D work->t_build_end - work->t_build_start; + u64 gpu_ns =3D t_gpu_end > work->t_submit + ? t_gpu_end - work->t_submit : 0; + + s->rx_build_ns +=3D build_ns; + s->rx_gpu_ns +=3D gpu_ns; + s->rx_sdma_copies_total +=3D work->rx_sdma_copies; + s->rx_sdma_bytes_total +=3D work->rx_sdma_bytes; + if (work->rx_sdma_copies > s->rx_sdma_copies_max) + s->rx_sdma_copies_max =3D work->rx_sdma_copies; + } + return false; /* -> SDMA_PENDING */ + } + + /* No SDMA copies - mark bds PASS and schedule NAPIs. */ + for (i =3D 0; i < work->nr_packets; i++) { + if (work->rx_bds[i]) + work->rx_bds[i]->act =3D KNOD_IPSEC_PASS; + } + if (knodev) { + for (i =3D 0; i < work->per_q_n; i++) { + struct knod_work_priv *wpriv =3D + &knodev->wpriv[work->per_q_touched[i]]; + + knod_napi_kick(wpriv); + } + } + + if (stats_on) { + struct knod_ipsec_stats *s =3D this_cpu_ptr(priv->stats); + + t_end =3D ktime_get_ns(); + + /* RX no-SDMA: all stats fit here. The SDMA path + * records build/gpu stats in the early-return above + * and sdma/finalise stats in finalise_sdma_done(). + */ + u64 build_ns =3D work->t_build_end - work->t_build_start; + u64 gpu_ns =3D t_gpu_end > work->t_submit + ? t_gpu_end - work->t_submit : 0; + u64 finalise_ns =3D t_end > t_gpu_end + ? t_end - t_gpu_end : 0; + + s->rx_build_ns +=3D build_ns; + s->rx_gpu_ns +=3D gpu_ns; + s->rx_finalise_ns +=3D finalise_ns; + s->rx_total_ns +=3D t_end - work->t_build_start; + } + + /* Clear slot state so the dispatcher can reuse this work entry. */ + work->per_q_n =3D 0; + work->nr_packets =3D 0; + work->sdma_wait_ns =3D 0; + work->rx_sdma_copies =3D 0; + work->rx_sdma_bytes =3D 0; + work->n_sdma_pending =3D 0; + + return true; /* -> EMPTY */ +} + +/* + * SDMA fence has fired for a slot in SDMA_PENDING. Publish desc_ring + * entries, mark bds PASS, schedule NAPIs, accumulate remaining stats, + * and clear the slot for reuse. + */ +static void knod_ipsec_finalise_sdma_done(struct knod_ipsec_dispatcher *di= sp, + struct knod_ipsec_work *work) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + bool stats_on =3D static_branch_unlikely(&ipsec_stats_enabled_key); + + knod_ipsec_finish_rx_complete(disp, work); + + /* Schedule NAPIs now that bds are marked PASS. */ + if (priv->knodev) { + int i; + + for (i =3D 0; i < work->per_q_n; i++) { + struct knod_work_priv *wpriv =3D + &priv->knodev->wpriv[work->per_q_touched[i]]; + + knod_napi_kick(wpriv); + } + } + + if (stats_on) { + struct knod_ipsec_stats *s =3D this_cpu_ptr(priv->stats); + u64 t_end =3D ktime_get_ns(); + u64 finalise_ns =3D t_end > work->t_finalise_start + ? t_end - work->t_finalise_start : 0; + + if (finalise_ns > work->sdma_wait_ns) + finalise_ns -=3D work->sdma_wait_ns; + else + finalise_ns =3D 0; + + s->rx_sdma_ns +=3D work->sdma_wait_ns; + s->rx_finalise_ns +=3D finalise_ns; + s->rx_total_ns +=3D t_end - work->t_build_start; + } + + work->per_q_n =3D 0; + work->nr_packets =3D 0; + work->sdma_wait_ns =3D 0; + work->rx_sdma_copies =3D 0; + work->rx_sdma_bytes =3D 0; + work->n_sdma_pending =3D 0; +} + +/* + * Pipelined dispatcher kthread. Each dispatcher owns one kaql[i] / + * sdma[i] pair, a private slice of the work_pool, and a contiguous + * range of RX queues. Up to KNOD_IPSEC_NR_WORK dispatches + * may be in-flight on this AQL queue at once. Each iteration: + * + * 1) Scan INFLIGHT slots within this disp's slice, finalise any + * whose completion signal fired. + * 2) Scan EMPTY slots in the same slice, try to build+submit one RX + * batch into it. + * 3) If nothing happened (no work to finalise, nothing to build), + * idle (usleep or cpu_relax spins). + * + * With nr_dispatchers > 1 the AQL queues run independently on the + * GPU - disjoint CU allocations per kaql - so two dispatchers get + * real parallel execution on the GPU. Within a single dispatcher, + * kaql[i] is FIFO: dispatches complete in submission order, and each + * slot's work->sigval is assigned a distinct target by + * dispatch_submit() so dispatch_done() can distinguish completions + * of different in-flight slots. + * + * No cross-dispatcher synchronisation on the hot path: each disp + * only touches its own rxq range and its own work slice. + * The shared SA table, slot array and per-CPU stats are read-mostly + * or percpu respectively. + */ +static int knod_ipsec_dispatcher(void *arg) +{ + struct knod_ipsec_dispatcher *disp =3D arg; + struct knod_ipsec_priv *priv =3D disp->priv; + int i; + + while (!kthread_should_stop()) { + bool did_work =3D false; + + if (kthread_should_park()) { + knod_ipsec_dispatcher_drain(disp); + kthread_parkme(); + continue; + } + + /* Phase 1a: finalise any INFLIGHT slot whose GPU work done. */ + for (i =3D 0; i < disp->work_count; i++) { + struct knod_ipsec_work *w =3D + &priv->work_pool[disp->work_first + i]; + + if (w->state !=3D KNOD_WORK_INFLIGHT) + continue; + if (!knod_ipsec_dispatch_done(disp, w)) + continue; + + if (knod_ipsec_finalise_inflight(disp, w)) + w->state =3D KNOD_WORK_EMPTY; + else + w->state =3D KNOD_WORK_SDMA_PENDING; + did_work =3D true; + } + + /* + * Phase 1b: complete any SDMA_PENDING slot whose fence + * fired. + */ + for (i =3D 0; i < disp->work_count; i++) { + struct knod_ipsec_work *w =3D + &priv->work_pool[disp->work_first + i]; + + if (w->state !=3D KNOD_WORK_SDMA_PENDING) + continue; + if (!knod_ipsec_fence_passed(w)) + continue; + + knod_ipsec_finalise_sdma_done(disp, w); + w->state =3D KNOD_WORK_EMPTY; + did_work =3D true; + } + + /* Phase 2: try to build + submit into at most one EMPTY slot + * per iteration. Limiting to one per iteration keeps phase-1 + * polling responsive so completions don't queue up while we + * greedily fill every empty slot. + */ + for (i =3D 0; i < disp->work_count; i++) { + int idx =3D (disp->build_cursor + i) % disp->work_count; + struct knod_ipsec_work *w =3D + &priv->work_pool[disp->work_first + idx]; + + if (w->state !=3D KNOD_WORK_EMPTY) + continue; + + if (knod_ipsec_dispatcher_try_rx(disp, w)) { + w->state =3D KNOD_WORK_INFLIGHT; + did_work =3D true; + disp->build_cursor =3D + (idx + 1) % disp->work_count; + break; + } + } + + if (priv->knodev) { + int q; + + for (q =3D disp->rxq_first; + q < disp->rxq_first + disp->rxq_count; + q++) { + struct knod_work_priv *wp =3D + &priv->knodev->wpriv[q]; + + if (!spsc_empty(&wp->pass_pending)) + knod_napi_kick(wp); + } + } + + if (!did_work) { + u64 idle_t0 =3D 0; + bool stats_on =3D + static_branch_unlikely(&ipsec_stats_enabled_key); + + if (stats_on) + idle_t0 =3D ktime_get_ns(); + + if (READ_ONCE(knod_ipsec_poll_mode)) { + /* Busy-poll: spin a small number of times with + * cpu_relax() so newly-produced SPSC entries or + * completions are picked up in <1us. + */ + int spins =3D 64; + + while (spins-- > 0) + cpu_relax(); + } else { + usleep_range(20, 100); + } + + if (stats_on) + this_cpu_ptr(priv->stats)->rx_idle_ns +=3D + ktime_get_ns() - idle_t0; + } + } + + knod_ipsec_dispatcher_drain(disp); + return 0; +} + +static void knod_ipsec_dispatcher_drain(struct knod_ipsec_dispatcher *disp) +{ + struct knod_ipsec_priv *priv =3D disp->priv; + int i; + + for (i =3D 0; i < disp->work_count; i++) { + struct knod_ipsec_work *w =3D + &priv->work_pool[disp->work_first + i]; + + if (w->state =3D=3D KNOD_WORK_INFLIGHT) { + knod_ipsec_dispatch_wait(disp, w); + if (!knod_ipsec_finalise_inflight(disp, w)) { + /* + * SDMA started - spin-wait, we can't + * defer. + */ + knod_ipsec_fence_wait(w); + knod_ipsec_finalise_sdma_done(disp, w); + } + w->state =3D KNOD_WORK_EMPTY; + } else if (w->state =3D=3D KNOD_WORK_SDMA_PENDING) { + knod_ipsec_fence_wait(w); + knod_ipsec_finalise_sdma_done(disp, w); + w->state =3D KNOD_WORK_EMPTY; + } + } +} + +/* + * Work slot pool allocator. + * + * `work_pool[0..KNOD_IPSEC_NR_WORK-1]` are pipelined work slots owned by + * the dispatcher kthread. Each slot gets its own slice of the param / + * decrypt pool BOs - single large BOs avoid the multi-BO GPU-VA + * mapping bug. KAT / selftest code paths always run against slot 0. + * + */ +static int knod_ipsec_work_pool_alloc(struct knod_ipsec_priv *priv) +{ + /* Round the kernarg pool up to a power-of-two page count so we never + * hit the 7-page (or other non-pow2) VRAM alloc bug the KNOD + * allocator triggers on certain sizes. At PKT_BATCH=3D512 fused_param + * is ~16 KB =3D 4 pages (already pow2). At BATCH=3D1024 it is ~32 KB =3D= 8 + * pages. Harmless extra slack otherwise. + */ + const size_t param_raw_bytes =3D + ALIGN(sizeof(struct knod_ipsec_fused_param), PAGE_SIZE); + const unsigned long param_raw_pages =3D param_raw_bytes >> PAGE_SHIFT; + const unsigned long param_pages =3D + param_raw_pages <=3D 1 ? 1 : roundup_pow_of_two(param_raw_pages); + const size_t param_stride =3D param_pages << PAGE_SHIFT; + struct amd_signal *signal; + int nr_disp =3D priv->nr_dispatchers; + int d, i; + + if (nr_disp < 1) + nr_disp =3D 1; + if (nr_disp > KNOD_IPSEC_MAX_DISPATCHERS) + nr_disp =3D KNOD_IPSEC_MAX_DISPATCHERS; + priv->nr_dispatchers =3D nr_disp; + + memset(priv->work_pool, 0, sizeof(priv->work_pool)); + + for (d =3D 0; d < nr_disp; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + disp->priv =3D priv; + disp->kthread =3D NULL; + disp->kaql_idx =3D d; + disp->work_first =3D d * KNOD_IPSEC_NR_WORK; + disp->work_count =3D KNOD_IPSEC_NR_WORK; + disp->rx_rr =3D 0; + disp->build_cursor =3D 0; + + /* GTT (not VRAM) for param_pool so the dispatcher CPU + * can read sub[].pkt_addr / sub[].out_addr back in + * finish_rx_deliver via WB cache rather than WC PCIe. + */ + disp->param_pool =3D knod_alloc_mem(priv->knod, + ALIGN(param_stride * KNOD_IPSEC_NR_WORK, PAGE_SIZE), + KFD_IOC_ALLOC_MEM_FLAGS_GTT | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(disp->param_pool)) { + int err =3D PTR_ERR(disp->param_pool); + + disp->param_pool =3D NULL; + return err; + } + + /* +KNOD_IPSEC_GTT_OUT_L3_OFF: 20 B headroom so the + * shader's "L3 header -> out_addr - L3_OFF" write has a + * valid GPU VM target when a SLOT_NONE fallback directs + * output here. Each work slot's rx_out_gaddr includes + * this offset. + */ + disp->decrypt_pool =3D knod_alloc_mem(priv->knod, + KNOD_IPSEC_DECRYPT_WORK_SIZE * KNOD_IPSEC_NR_WORK + + KNOD_IPSEC_GTT_OUT_L3_OFF, + KFD_IOC_ALLOC_MEM_FLAGS_VRAM | + KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE | + KFD_IOC_ALLOC_MEM_FLAGS_COHERENT); + if (IS_ERR(disp->decrypt_pool)) { + int err =3D PTR_ERR(disp->decrypt_pool); + + disp->decrypt_pool =3D NULL; + return err; + } + + for (i =3D 0; i < KNOD_IPSEC_NR_WORK; i++) { + struct knod_ipsec_work *work =3D + &priv->work_pool[disp->work_first + i]; + + memset(work, 0, sizeof(*work)); + work->state =3D KNOD_WORK_EMPTY; + work->param.kaddr =3D (u8 *)disp->param_pool->kaddr + + (size_t)i * param_stride; + work->param.gaddr =3D disp->param_pool->gaddr + + (u64)i * param_stride; + work->rx_out_gaddr =3D disp->decrypt_pool->gaddr + + KNOD_IPSEC_GTT_OUT_L3_OFF + + (u64)i * KNOD_IPSEC_DECRYPT_WORK_SIZE; + } + + /* Initialise this dispatcher's forward-looking sigval + * counter from its kaql[d] completion signal. Every + * submit decrements this so each in-flight slot has a + * distinct target. + */ + signal =3D (struct amd_signal *) + priv->knod->kaql[d].queue_signal->kaddr; + disp->dispatch_sigval_next =3D READ_ONCE(signal->value); + + pr_info("knod_ipsec: disp[%d] param=3D0x%llx decrypt=3D0x%llx kaql=3D%d\= n", + d, (u64)disp->param_pool->gaddr, + (u64)disp->decrypt_pool->gaddr, + disp->kaql_idx); + } + + return 0; +} + +static void knod_ipsec_work_pool_free(struct knod_ipsec_priv *priv) +{ + int d; + + for (d =3D 0; d < KNOD_IPSEC_MAX_DISPATCHERS; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + if (disp->decrypt_pool) { + knod_free_mem(priv->knod, disp->decrypt_pool); + disp->decrypt_pool =3D NULL; + } + if (disp->param_pool) { + knod_free_mem(priv->knod, disp->param_pool); + disp->param_pool =3D NULL; + } + } + memset(priv->work_pool, 0, sizeof(priv->work_pool)); +} + + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Phase 5: in-kernel KAT (randomized + dispatch smoke) + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * + * Two-layer validation: + * + * 1) CPU layer - randomized H-table check + * For each supported AES key length (128/192/256), run N iterations: + * generate a fresh random key, compute the reference H =3D AES_K(0^1= 28) + * via the kernel's verified library, then recompute t= he + * same H via our gcm_core helper (which will be used on the GPU path) + * and require a bytewise match. This exercises aes_prepareenckey / + * aes_encrypt for every round count and the downstream gf128 squaring + * chain that builds the H-power table. No hand-computed constants. + * + * 2) GPU layer - multi-packet shader dispatch smoke test + * Run the fused RX shader at several batch sizes (1, 8, 32, 64) with + * each sub[i].bd_addr pointing at a distinct fake spsc_bd slot inside + * a single scratch BO. Pre-stamp each bd->act with a UNIQUE sentinel + * (sentinel base XORed with packet index) and verify the shader + * rewrites ALL slots to XDP_PASS. This covers workgroup_id_y indexing + * in the shader + variable grid dimensions in the AQL packet. When + * the real RX crypto shader lands, the per-slot verification can be + * upgraded to "decrypted inner packet matches expected plaintext". + * + * Randomness seeds vary per run so repeated `echo 1 > selftest` covers a + * widening input space over time. The shader layer is O(ms); the CPU + * layer runs in microseconds per iteration. + */ + +#define KNOD_IPSEC_KAT_RAND_ROUNDS 16 + +static int knod_ipsec_last_kat_result =3D -1; +static char knod_ipsec_last_kat_detail[256]; + +static int knod_ipsec_kat_h_one(const u8 *key, int key_len) +{ + u8 h_table[KNOD_GCM_H_TABLE_SIZE]; + struct aes_enckey enckey; + u8 zero[AES_BLOCK_SIZE] =3D {}; + u8 h_ref[AES_BLOCK_SIZE]; + int rc; + + rc =3D aes_prepareenckey(&enckey, key, key_len); + if (rc) + return rc; + aes_encrypt(&enckey, h_ref, zero); + memzero_explicit(&enckey, sizeof(enckey)); + + memset(h_table, 0, sizeof(h_table)); + knod_gcm_precompute_h_table(key, key_len, h_table); + + return memcmp(h_table, h_ref, AES_BLOCK_SIZE) =3D=3D 0 ? 0 : -EBADMSG; +} + +static int knod_ipsec_run_cpu_kat(void) +{ + static const int key_lens[] =3D { 16, 24, 32 }; + u8 key[32]; + int i, r, failed =3D 0; + + for (i =3D 0; i < ARRAY_SIZE(key_lens); i++) { + int kl =3D key_lens[i]; + int round_fail =3D 0; + + for (r =3D 0; r < KNOD_IPSEC_KAT_RAND_ROUNDS; r++) { + get_random_bytes(key, kl); + if (knod_ipsec_kat_h_one(key, kl)) { + pr_err("knod_ipsec: H-KAT aes%d round=3D%d FAIL\n", + kl * 8, r); + round_fail++; + } + } + if (round_fail) { + failed +=3D round_fail; + } else { + pr_info("knod_ipsec: H-KAT aes%d ok (%d random rounds)\n", + kl * 8, KNOD_IPSEC_KAT_RAND_ROUNDS); + } + } + memzero_explicit(key, sizeof(key)); + return failed; +} + +/* + * Shader-dispatch smoke test (post 32d.2 architecture pivot). + * + * Uses the persistent priv->kat_scratch BO as a private sandbox for both + * per-packet spsc_bd slots and synthetic ESP packet buffers: + * + * scratch[0 .. nr*64) -- spsc_bd slots (act at +i*64+8) + * scratch[pkt_region_base + i*64 ..) -- ETH+IPv4+ESP synthetic packet + * + * Each synthetic packet gets a unique SPI =3D SPI_KAT_BASE | i written in + * network byte order at ESP header offset (14 ETH + 20 IPv4 =3D 34). The + * corresponding slot in priv->sa_table is pre-populated with the same + * SPI (in host byte order) at index i. The fused RX shader reads + * sub[wg_id_y].pkt_addr, loads the BE SPI from pkt+34, byteswaps it, + * linear-scans priv->sa_table for a match, and writes the matching + * slot_idx (as u64) into bd->act. + * + * Verification scope (single pass): + * - even i (IPv4): bd->act high32 =3D=3D 0xFFFFFFFD (ICV fail - garbage + * keys mean decryption produces wrong tag, but the full crypto + * pipeline runs to completion proving SPI scan + dispatch work) + * - odd i (IPv6): bd->act high32 =3D=3D 0xFFFFFFFD (ICV fail - same as + * IPv4 but exercises the IPv6 ESP offset path; SPI at +54) + * + * The 32d.1/32d.2 in-shader replay bitmap path has been removed (the + * sliding window now lives CPU-side in the NIC dd NAPI), so the + * second dispatch pass and the replay_bitmap_addr/readback block have + * been dropped. End-to-end worker->desc_ring->NIC-dd delivery is covered + * by the userspace selftest `knod_ipsec_offload.sh`, not by this KAT. + */ +static int knod_ipsec_run_shader_kat_n(struct knod_ipsec_priv *priv, int n= r) +{ + static const u64 SENTINEL_BASE =3D 0xDEADBEEFCAFEBABEULL; + static const u32 SPI_KAT_BASE =3D 0xDECAF000u; + const size_t SLOT_STRIDE =3D 64; + /* must fit IPv6 ESP minimum (86B overhead) */ + const size_t PKT_STRIDE =3D 128; + /* Heap-backed sub[] - at BATCH=3D512 the stack copy would be 16 KB + * and overflow the dispatcher kernel stack. KAT is cold path, so + * kvmalloc is fine. + */ + struct knod_ipsec_sa_entry *e_dbg; + struct knod_ipsec_sa_entry *e; + struct knod_ipsec_work *dbg_work; + struct knod_ipsec_fused_sub *sub; + struct amd_signal *sig; + struct knod_mem *scratch; + void *sa_backup =3D NULL; + size_t bd_region; + size_t pkt_region_base; + int written_final =3D 0; + s64 sig_before; + u64 dbg16 =3D 0; + int ret =3D 0; + int tries, i; + + if (nr < 1 || nr > KNOD_IPSEC_KAT_MAX_BATCH) + return -EINVAL; + + sub =3D kvcalloc(KNOD_IPSEC_KAT_MAX_BATCH, sizeof(*sub), GFP_KERNEL); + if (!sub) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch[%d] FAIL: sub alloc", nr); + return -ENOMEM; + } + + bd_region =3D SLOT_STRIDE * nr; + pkt_region_base =3D ALIGN(bd_region, 64); + if (pkt_region_base + nr * PKT_STRIDE > PAGE_SIZE * 4) { + ret =3D -EINVAL; /* defensive; kat_scratch is 4 pages */ + goto out_free; + } + + /* Reuse the persistent per-priv scratch BO - see priv->kat_scratch */ + scratch =3D priv->kat_scratch; + if (!scratch) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch[%d] FAIL: kat_scratch not allocated", + nr); + ret =3D -ENOMEM; + goto out_free; + } + memset(scratch->kaddr, 0, PAGE_SIZE * 4); + + /* Back up the production SA table before trashing it with KAT + * fixtures. Without this, any production SA installed by the user + * (e.g. via `ip xfrm state add`) gets wiped when the KAT wipes the + * SA BO at exit, which causes a subsequent production dispatch + * to load key_gpu_addr=3D0 -> VM fault at address 0. + */ + sa_backup =3D kvmalloc(KNOD_IPSEC_SA_BO_SIZE, GFP_KERNEL); + if (!sa_backup) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch[%d] FAIL: sa_backup alloc", nr); + ret =3D -ENOMEM; + goto out_free; + } + memcpy(sa_backup, priv->sa_table->kaddr, KNOD_IPSEC_SA_BO_SIZE); + + /* Pre-populate priv->sa_table[0..nr) with one entry per synthetic + * packet. Entry i carries spi =3D SPI_KAT_BASE | i in host byte order + * (the table is CPU-side LE and the shader reads it as a host u32). + * Higher-index slots are zeroed first so a previous KAT iteration + * at larger nr cannot leak into a smaller one. + */ + BUILD_BUG_ON(sizeof(struct knod_ipsec_sa_entry) !=3D 104); + BUILD_BUG_ON(KNOD_IPSEC_NR_SA < KNOD_IPSEC_KAT_MAX_BATCH); + e =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr; + + /* Wipe entries + replay bitmap region (one BO). Size is + * ALIGNed to 8 pages in the alloc path to dodge the KNOD VRAM + * 7-page alloc bug; the logical payload is still SA_BO_SIZE. + */ + memset(e, 0, KNOD_IPSEC_SA_BO_SIZE); + for (i =3D 0; i < nr; i++) { + e[i].spi =3D cpu_to_le32(SPI_KAT_BASE | (u32)i); + e[i].active =3D cpu_to_le32(1); + /* Full crypto shader loads key/htable/t_tables from + * the SA entry after scan hit. Point them at valid + * GPU addresses to avoid NULL page faults. The data + * is garbage so decryption will produce ICV mismatch, + * but the dispatch will complete without a VM fault. + */ + e[i].key_gpu_addr =3D cpu_to_le64(scratch->gaddr); + e[i].htable_gpu_addr =3D cpu_to_le64(scratch->gaddr); + e[i].t_tables_gpu_addr =3D cpu_to_le64( + priv->t_tables ? priv->t_tables->gaddr : + scratch->gaddr); + e[i].key_len =3D cpu_to_le32(16); + e[i].nr_rounds =3D cpu_to_le32(10); + } + /* publish the key-table writes to WC VRAM before dispatch */ + wmb(); + + /* sub was zero-filled by kvcalloc at entry. */ + for (i =3D 0; i < nr; i++) { + u64 *act_i =3D (u64 *)((u8 *)scratch->kaddr + + i * SLOT_STRIDE + 8); + u8 *pkt_i =3D (u8 *)scratch->kaddr + + pkt_region_base + i * PKT_STRIDE; + __be32 spi_be =3D cpu_to_be32(SPI_KAT_BASE | (u32)i); + bool is_ipv4 =3D !(i & 1); + + /* Pre-stamp bd->act with a sentinel so an untouched slot is + * distinguishable from one that was overwritten with the SPI. + */ + *act_i =3D SENTINEL_BASE ^ (u64)i; + + /* Alternate IPv4 / IPv6 packets per slot to exercise both + * code paths in the same batch. Even i: standard IPv4 header + * (version=3D4, IHL=3D5 -> byte 0x45), SPI at offset 34. + * Odd i: IPv6 header (version=3D6 -> byte 0x60), SPI at + * offset 54. Both hit the SA scan, both get ICV fail + * (since ciphertext is garbage). + */ + pkt_i[14] =3D is_ipv4 ? 0x45 : 0x60; + if (is_ipv4) + memcpy(pkt_i + 34, &spi_be, sizeof(spi_be)); + else + memcpy(pkt_i + 54, &spi_be, sizeof(spi_be)); + + sub[i].pkt_addr =3D cpu_to_le64(scratch->gaddr + + pkt_region_base + i * PKT_STRIDE); + sub[i].out_addr =3D sub[i].pkt_addr; + sub[i].bd_addr =3D cpu_to_le64(scratch->gaddr + + i * SLOT_STRIDE); + sub[i].pkt_len =3D cpu_to_le32(PKT_STRIDE); + } + /* publish the sub-descriptor writes to WC VRAM before dispatch */ + wmb(); + + /* Single dispatch: post-32d.2 architecture has no in-shader replay, + * so the KAT only validates the fresh scan-hit path (+ the IPv6 + * IPv6 ESP offset path). End-to-end worker->desc_ring->NAPI delivery + * is covered by the userspace selftest, not here. + */ + /* Force the dispatch onto queue 0 so the signal we observe below is + * actually the one CP updates. The public rx_submit() picks a queue + * by smp_processor_id(), which would let the KAT measure a queue it + * never submitted to and see a stale sig=3D-1 forever. + */ + sig =3D (struct amd_signal *) + priv->knod->kaql[0].queue_signal->kaddr; + sig_before =3D READ_ONCE(sig->value); + dbg_work =3D &priv->work_pool[0]; + + e_dbg =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr; + + pr_info("knod_ipsec: KAT[%d] scratch.gaddr=3D0x%llx work.param.gaddr=3D0x= %llx kernel.gaddr=3D0x%llx sa_table.gaddr=3D0x%llx cpu-readback: entry[0].s= pi=3D0x%08x entry[1].spi=3D0x%08x\n", + nr, + (u64)scratch->gaddr, + (u64)dbg_work->param.gaddr, + (u64)priv->knod->kernels[0]->gaddr, + (u64)priv->sa_table->gaddr, + le32_to_cpu(e_dbg[0].spi), + le32_to_cpu(e_dbg[1].spi)); + + if (priv->disp[0].kthread) + kthread_park(priv->disp[0].kthread); + knod_ipsec_prepare_rx_dispatch(priv, dbg_work, sub, NULL, + nr, NULL, 0); + knod_ipsec_dispatch_and_wait(&priv->disp[0], dbg_work); + if (priv->disp[0].kthread) + kthread_unpark(priv->disp[0].kthread); + ret =3D 0; + pr_info("knod_ipsec: shader-dispatch[%d] submitted, signal %lld -> (done)= \n", + nr, sig_before); + + /* bd->act is packed as + * low =3D s22 =3D scan target SPI + * high =3D s26 =3D final result + * even i (IPv4): ICV fail -> 0xFFFFFFFD + * odd i (IPv6): ICV fail -> 0xFFFFFFFD + * scan miss : 0xFFFFFFFF + * + * Both IPv4 and IPv6 paths run the full crypto pipeline. The SPI + * is placed at the correct offset (34 for v4, 54 for v6) so the + * SA scan finds a match, then decrypt + ICV check fails on garbage. + */ + written_final =3D 0; +#define KAT_EXPECT_LOW_MASK (0ULL) +/* Both IPv4 and IPv6 slots: full crypto pipeline -> ICV mismatch + * -> verdict =3D VERDICT_ICV_FAIL (0xFFFFFFFD). + */ +#define KAT_EXPECT_FOR_SLOT \ + (((u64)KNOD_IPSEC_SHADER_VERDICT_ICV_FAIL << 32) | 0ULL) + + for (tries =3D 0; tries < 500; tries++) { + int w =3D 0; + + for (i =3D 0; i < nr; i++) { + u64 expect =3D KAT_EXPECT_FOR_SLOT; + u64 mask =3D 0xFFFFFFFF00000000ULL | + KAT_EXPECT_LOW_MASK; + u64 v =3D READ_ONCE(*(u64 *)( + (u8 *)scratch->kaddr + + i * SLOT_STRIDE + 8)); + if ((v & mask) =3D=3D (expect & mask)) + w++; + } + if (w =3D=3D nr) { + written_final =3D w; + break; + } + written_final =3D w; + usleep_range(500, 1000); + } + + if (written_final !=3D nr) { + int first_ok =3D -1, first_bad =3D -1, last_ok =3D -1; + u64 first_bad_val =3D 0; + u64 first_bad_expect =3D 0; + struct amd_signal *sig =3D (struct amd_signal *) + priv->knod->kaql[0].queue_signal->kaddr; + s64 sig_after =3D READ_ONCE(sig->value); + + for (i =3D 0; i < nr; i++) { + u64 expect =3D KAT_EXPECT_FOR_SLOT; + u64 mask =3D 0xFFFFFFFF00000000ULL | + KAT_EXPECT_LOW_MASK; + u64 v =3D READ_ONCE(*(u64 *)( + (u8 *)scratch->kaddr + + i * SLOT_STRIDE + 8)); + if ((v & mask) =3D=3D (expect & mask)) { + if (first_ok < 0) + first_ok =3D i; + last_ok =3D i; + } else if (first_bad < 0) { + first_bad =3D i; + first_bad_val =3D v; + first_bad_expect =3D expect; + } + } + dbg16 =3D 0; + if (first_bad >=3D 0) { + dbg16 =3D READ_ONCE(*(u64 *)( + (u8 *)scratch->kaddr + + first_bad * SLOT_STRIDE + 16)); + } + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch[%d] FAIL written=3D%d/%d first_bad=3D%d got=3D0x%llx= (low=3D0x%08x high=3D0x%08x) expect=3D0x%llx sig_after=3D%lld first_ok=3D%= d last_ok=3D%d dbg[s27,s23]=3D0x%08x 0x%08x", + nr, written_final, nr, + first_bad, first_bad_val, + (u32)first_bad_val, + (u32)(first_bad_val >> 32), + first_bad_expect, sig_after, + first_ok, last_ok, + (u32)dbg16, + (u32)(dbg16 >> 32)); + pr_err("knod_ipsec: %s\n", knod_ipsec_last_kat_detail); + ret =3D -EIO; + goto out_free; + } +#undef KAT_EXPECT_FOR_SLOT +#undef KAT_EXPECT_LOW_MASK + + pr_info("knod_ipsec: shader-dispatch[%d] ok (%d slots, %d us poll)\n", + nr, nr, tries * 750); + +out_free: + /* Restore the production SA table we saved at entry. Never wipe - + * wiping would destroy any live xfrm SAs installed by userspace. + */ + if (sa_backup) { + memcpy(priv->sa_table->kaddr, sa_backup, KNOD_IPSEC_SA_BO_SIZE); + /* publish the restored SA table to the GPU */ + wmb(); + kvfree(sa_backup); + } + kvfree(sub); + return ret; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Crypto KAT: end-to-end AES-GCM decrypt verification via GPU shader + * + * Builds a synthetic ESP packet with known AES-128-GCM ciphertext+ICV, + * dispatches the fused RX shader, and verifies the decrypted output + * matches the expected plaintext. This exercises the full decrypt + * pipeline: T-table load, CTR decrypt, parallel GHASH, ICV verify, + * ESP trailer strip. + * + * Layout in kat_scratch (2 pages =3D 8192B): + * [0x0000..0x003F] bd slot (64B, act at +8) + * [0x0040..0x00FF] out buffer for decrypted output (192B) + * [0x0100..0x01FF] expanded AES round keys (256B) + * [0x0200..0x02FF] synthetic ESP packet (256B) + * [0x1000..0x1FFF] GHASH H-power table (4096B) + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +/* GF(2^128) multiply for reference GHASH - identical to gcm_core but + * local to avoid exporting an internal helper for test-only use. + */ +static void kat_gf128_mul(u64 r[2], const u64 a[2], const u64 b[2]) +{ + u64 v[2], z[2]; + int i, j; + + v[0] =3D a[0]; v[1] =3D a[1]; + z[0] =3D 0; z[1] =3D 0; + + for (i =3D 0; i < 2; i++) { + u64 x =3D b[i]; + + for (j =3D 63; j >=3D 0; j--) { + if ((x >> j) & 1) { + z[0] ^=3D v[0]; + z[1] ^=3D v[1]; + } + if (v[1] & 1) { + v[1] =3D (v[1] >> 1) | (v[0] << 63); + v[0] =3D (v[0] >> 1) ^ ((u64)0xe1 << 56); + } else { + v[1] =3D (v[1] >> 1) | (v[0] << 63); + v[0] =3D v[0] >> 1; + } + } + } + r[0] =3D z[0]; + r[1] =3D z[1]; +} + +/* GHASH: hash `data` (must be multiple of 16 bytes) with key H. + * Result is stored in `out` (16 bytes, big-endian). + */ +static void kat_ghash(const u8 *h, const u8 *data, int data_len, u8 *out) +{ + u64 y[2] =3D { 0, 0 }; + u64 hh[2]; + int i; + + hh[0] =3D get_unaligned_be64(h); + hh[1] =3D get_unaligned_be64(h + 8); + + for (i =3D 0; i < data_len; i +=3D 16) { + u64 d[2], r[2]; + + d[0] =3D get_unaligned_be64(data + i); + d[1] =3D get_unaligned_be64(data + i + 8); + y[0] ^=3D d[0]; + y[1] ^=3D d[1]; + kat_gf128_mul(r, y, hh); + y[0] =3D r[0]; + y[1] =3D r[1]; + } + put_unaligned_be64(y[0], out); + put_unaligned_be64(y[1], out + 8); +} + +static int knod_ipsec_run_crypto_kat(struct knod_ipsec_priv *priv) +{ + /* Fixed AES-128 key + 4-byte salt (from RFC 4106 conventions). */ + static const u8 kat_key[16] =3D { + 0xfe, 0xff, 0xe9, 0x92, 0x86, 0x65, 0x73, 0x1c, + 0x6d, 0x6a, 0x8f, 0x94, 0x67, 0x30, 0x83, 0x08 + }; + static const u8 kat_salt[4] =3D { 0xca, 0xfe, 0xba, 0xbe }; + static const u8 kat_iv[8] =3D { + 0xfa, 0xce, 0xdb, 0xad, 0xde, 0xca, 0xf8, 0x88 + }; + /* 32-byte plaintext (2 AES blocks) + ESP trailer (pad_len=3D0, + * next_hdr=3D4). + * Total decrypted payload =3D 34 bytes, but ciphertext is padded to + * block boundary: 48 bytes (3 blocks) with 14 pad bytes + trailer. + * Actually simpler: use exactly 32 bytes of payload + 2 bytes trailer + * =3D 34 bytes, which is NOT block-aligned. The shader handles this + * because nblocks =3D ceil(34/16) =3D 3, and the last partial block is + * XORed with only the relevant bytes. + * + * Simplification: use 32 bytes ciphertext (2 full blocks) where + * the last 2 bytes are ESP trailer: byte[30]=3Dpad_len=3D0, byte[31]=3D4 + * (IPv4 next header). inner_len =3D 32 - 0 - 2 =3D 30. + */ + static const u8 kat_plain[32] =3D { + 0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5, + 0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a, + 0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda, + 0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, + 0x00, /* pad_len =3D 0 */ + 0x04, /* next_hdr =3D IPPROTO_IPIP (IPv4 tunnel) */ + }; + static const u32 kat_spi =3D 0xA5A5A5A5u; + static const u32 kat_seq =3D 0x00000001u; + +#define CRYPTO_KAT_BD_OFF 0x0000 +#define CRYPTO_KAT_OUT_OFF 0x0040 +#define CRYPTO_KAT_KEY_OFF 0x0100 +#define CRYPTO_KAT_PKT_OFF 0x0200 +#define CRYPTO_KAT_HTABLE_OFF 0x1000 + + struct knod_ipsec_sa_gpu_stats *gs; + struct knod_mem *scratch =3D priv->kat_scratch; + struct knod_ipsec_sa_entry *sa_entry; + struct knod_ipsec_fused_sub sub[1]; + struct crypto_aes_ctx aes_ctx; + struct aes_enckey enckey; + void *sa_backup =3D NULL; + u8 nonce[12], ctr_blk[16], keystream[16]; + u8 ciphertext[32], icv[16]; + u8 ghash_input[80]; + u8 h_block[16], ghash_out[16], j0_enc[16]; + u8 *pkt, *out_buf, *key_buf, *htable_buf; + u64 *act_ptr; + u32 j0_ctr; + int ret, tries, i, b; + + if (!scratch) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: kat_scratch not allocated"); + return -ENOMEM; + } + if (!priv->t_tables) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: t_tables not allocated"); + return -ENOMEM; + } + + memset(scratch->kaddr, 0, PAGE_SIZE * 4); + + pkt =3D (u8 *)scratch->kaddr + CRYPTO_KAT_PKT_OFF; + out_buf =3D (u8 *)scratch->kaddr + CRYPTO_KAT_OUT_OFF; + key_buf =3D (u8 *)scratch->kaddr + CRYPTO_KAT_KEY_OFF; + htable_buf =3D (u8 *)scratch->kaddr + CRYPTO_KAT_HTABLE_OFF; + act_ptr =3D (u64 *)((u8 *)scratch->kaddr + CRYPTO_KAT_BD_OFF + 8); + + /* ---- Step 1: Expand AES key into kat_scratch ---- */ + /* GPU round-key format uses crypto_aes_ctx.key_enc + * (matches xdo_state_add) + */ + ret =3D aes_expandkey(&aes_ctx, kat_key, sizeof(kat_key)); + if (ret) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: aes_expandkey=3D%d", ret); + return ret; + } + /* AES-128: 11 round keys x 4 u32 =3D 44 u32 =3D 176B */ + memcpy(key_buf, aes_ctx.key_enc, + (aes_ctx.key_length / 4 + 7) * 16); + /* Also prepare aes_enckey for CPU-side reference encryption */ + ret =3D aes_prepareenckey(&enckey, kat_key, sizeof(kat_key)); + if (ret) { + memzero_explicit(&aes_ctx, sizeof(aes_ctx)); + return ret; + } + + /* ---- Step 2: Precompute H-power table into kat_scratch page 2 ---- */ + knod_gcm_precompute_h_table(kat_key, sizeof(kat_key), htable_buf); + + /* ---- Step 3: CPU-side AES-GCM encrypt to produce ciphertext + ICV */ + /* H =3D AES_K(0^128) */ + memset(h_block, 0, 16); + aes_encrypt(&enckey, h_block, h_block); + + /* Nonce =3D salt || IV */ + memcpy(nonce, kat_salt, 4); + memcpy(nonce + 4, kat_iv, 8); + + /* CTR encrypt: counter starts at 2 for payload blocks */ + for (i =3D 0; i < 2; i++) { + u32 ctr_val =3D cpu_to_be32(i + 2); + + memcpy(ctr_blk, nonce, 12); + memcpy(ctr_blk + 12, &ctr_val, 4); + aes_encrypt(&enckey, keystream, ctr_blk); + + /* C_i =3D P_i XOR keystream */ + for (b =3D 0; b < 16; b++) + ciphertext[i * 16 + b] =3D + kat_plain[i * 16 + b] ^ keystream[b]; + } + + /* GHASH over AAD(16) || ciphertext(32) || len_block(16) */ + memset(ghash_input, 0, sizeof(ghash_input)); + /* AAD =3D SPI(4B,BE) || seq(4B,BE) || zero-pad to 16B */ + put_unaligned_be32(kat_spi, ghash_input); + put_unaligned_be32(kat_seq, ghash_input + 4); + /* ciphertext blocks */ + memcpy(ghash_input + 16, ciphertext, 32); + /* len block: AAD_bitlen(64b) || ctext_bitlen(64b) */ + /* 8 bytes AAD =3D 64 bits */ + put_unaligned_be64(8ULL * 8, ghash_input + 48); + /* 32 bytes ctext =3D 256 bits */ + put_unaligned_be64(32ULL * 8, ghash_input + 56); + + kat_ghash(h_block, ghash_input, 64, ghash_out); + + /* ICV =3D GHASH XOR AES_K(J0), where J0 =3D nonce || 0x00000001 (BE) */ + j0_ctr =3D cpu_to_be32(1); + memcpy(ctr_blk, nonce, 12); + memcpy(ctr_blk + 12, &j0_ctr, 4); + aes_encrypt(&enckey, j0_enc, ctr_blk); + for (b =3D 0; b < 16; b++) + icv[b] =3D ghash_out[b] ^ j0_enc[b]; + memzero_explicit(&enckey, sizeof(enckey)); + memzero_explicit(&aes_ctx, sizeof(aes_ctx)); + + /* ---- Step 4: Build synthetic ESP packet ---- */ + /* ETH header (14B): dst=3D00:..., src=3D00:..., ethertype=3D0x0800 */ + pkt[12] =3D 0x08; pkt[13] =3D 0x00; + /* IPv4 header (20B): version=3D4, IHL=3D5, protocol=3D50(ESP) */ + pkt[14] =3D 0x45; + /* total_length (network order): + * 20(IP) + 8(ESP) + 8(IV) + 32(ctext) + 16(ICV) =3D 84 + */ + pkt[16] =3D 0x00; pkt[17] =3D 84; + /* protocol =3D 50 (ESP) */ + pkt[23] =3D 50; + /* SPI at +34 (network byte order) */ + put_unaligned_be32(kat_spi, pkt + ESP_SPI_OFF); + /* Seq at +38 */ + put_unaligned_be32(kat_seq, pkt + ESP_SEQ_OFF); + /* IV at +42 (8 bytes) */ + memcpy(pkt + ESP_IV_OFF, kat_iv, 8); + /* Ciphertext at +50 (32 bytes) */ + memcpy(pkt + ESP_CTEXT_OFF, ciphertext, 32); + /* ICV at +82 (16 bytes) */ + memcpy(pkt + ESP_CTEXT_OFF + 32, icv, 16); + + /* total pkt len =3D 14 + 84 =3D 98 bytes */ +#define CRYPTO_KAT_PKT_LEN 98 + + /* ---- Step 5: Set up SA entry (slot 0) in sa_table ---- */ + BUILD_BUG_ON(sizeof(struct knod_ipsec_sa_entry) !=3D 104); + + /* Back up production SA table before trashing slot 0 with the KAT SA. + * Restored at out_wipe. Without this, any live xfrm SA in slot 0 + * gets wiped and subsequent production dispatch faults on key=3D0. + */ + sa_backup =3D kvmalloc(KNOD_IPSEC_SA_BO_SIZE, GFP_KERNEL); + if (!sa_backup) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: sa_backup alloc"); + return -ENOMEM; + } + memcpy(sa_backup, priv->sa_table->kaddr, KNOD_IPSEC_SA_BO_SIZE); + memset(priv->sa_table->kaddr, 0, KNOD_IPSEC_SA_BO_SIZE); + + sa_entry =3D (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr; + sa_entry->spi =3D cpu_to_le32(kat_spi); + sa_entry->dir =3D cpu_to_le32(0); + sa_entry->family =3D cpu_to_le32(AF_INET); + sa_entry->flags =3D cpu_to_le32(0); + sa_entry->key_gpu_addr =3D cpu_to_le64(scratch->gaddr + + CRYPTO_KAT_KEY_OFF); + sa_entry->htable_gpu_addr =3D cpu_to_le64(scratch->gaddr + + CRYPTO_KAT_HTABLE_OFF); + sa_entry->t_tables_gpu_addr =3D cpu_to_le64(priv->t_tables->gaddr); + memcpy(sa_entry->salt, kat_salt, 4); + sa_entry->key_len =3D cpu_to_le32(16); + sa_entry->nr_rounds =3D cpu_to_le32(10); /* AES-128 */ + /* KAT uses tunnel mode */ + sa_entry->mode =3D cpu_to_le32(XFRM_MODE_TUNNEL); + sa_entry->stats_addr =3D cpu_to_le64(priv->sa_table->gaddr + + KNOD_IPSEC_STATS_REGION_OFF); + sa_entry->active =3D cpu_to_le32(1); + sa_entry->version =3D cpu_to_le32(1); + /* publish the SA entry to the GPU before it becomes live */ + wmb(); + + /* Pre-stamp bd->act with sentinel */ + *act_ptr =3D 0xDEADDEADDEADDEADULL; + + /* ---- Step 6: Build sub[] and dispatch ---- */ + memset(sub, 0, sizeof(sub)); + sub[0].pkt_addr =3D cpu_to_le64(scratch->gaddr + CRYPTO_KAT_PKT_OFF); + sub[0].out_addr =3D cpu_to_le64(scratch->gaddr + CRYPTO_KAT_OUT_OFF); + sub[0].bd_addr =3D cpu_to_le64(scratch->gaddr + CRYPTO_KAT_BD_OFF); + sub[0].pkt_len =3D cpu_to_le32(CRYPTO_KAT_PKT_LEN); + sub[0].result_seq =3D 0; + + pr_info("knod_ipsec: crypto-kat: dispatching AES-128-GCM decrypt (spi=3D0= x%08x, %d bytes ctext)\n", + kat_spi, 32); + + if (priv->disp[0].kthread) + kthread_park(priv->disp[0].kthread); + knod_ipsec_prepare_rx_dispatch(priv, &priv->work_pool[0], sub, NULL, + 1, NULL, 0); + knod_ipsec_dispatch_and_wait(&priv->disp[0], &priv->work_pool[0]); + if (priv->disp[0].kthread) + kthread_unpark(priv->disp[0].kthread); + ret =3D 0; + + /* ---- Step 7: Poll for completion ---- */ + for (tries =3D 0; tries < 500; tries++) { + u64 v =3D READ_ONCE(*act_ptr); + u32 verdict_hi =3D (u32)(v >> 32); + + if (verdict_hi !=3D 0xDEADDEAD) { + if (verdict_hi =3D=3D 0) { + /* + * Slot 0 =3D ICV passed! + * Verify decrypted output. + */ + if (memcmp(out_buf, kat_plain, 32) !=3D 0) { + pr_err("knod_ipsec: crypto-kat FAIL: plaintext mismatch\n"); + print_hex_dump(KERN_ERR, " expected: ", + DUMP_PREFIX_NONE, + 16, 1, kat_plain, 32, + false); + print_hex_dump(KERN_ERR, " got: ", + DUMP_PREFIX_NONE, + 16, 1, out_buf, 32, + false); + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: plaintext mismatch (verdict ok)"); + ret =3D -EBADMSG; + goto out_wipe; + } + gs =3D (struct knod_ipsec_sa_gpu_stats *) + ((u8 *)priv->sa_table->kaddr + + KNOD_IPSEC_STATS_REGION_OFF); + pr_info("knod_ipsec: crypto-kat PASS: AES-128-GCM decrypt verified (%d= us poll, gpu_stats: pkts=3D%llu bytes=3D%llu)\n", + tries * 750, + le64_to_cpu(gs->rx_packets), + le64_to_cpu(gs->rx_bytes)); + ret =3D 0; + goto out_wipe; + } else if (verdict_hi =3D=3D VERDICT_ICV_FAIL) { + pr_err("knod_ipsec: crypto-kat FAIL: ICV mismatch (verdict=3D0x%08x)\n= ", + verdict_hi); + print_hex_dump(KERN_ERR, " ref-icv: ", + DUMP_PREFIX_NONE, + 16, 1, icv, 16, false); + print_hex_dump(KERN_ERR, " out-buf: ", + DUMP_PREFIX_NONE, + 16, 1, out_buf, 32, false); + print_hex_dump(KERN_ERR, " ref-plain:", + DUMP_PREFIX_NONE, + 16, 1, kat_plain, 32, false); + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: shader ICV mismatch"); + ret =3D -EBADMSG; + goto out_wipe; + } else { + pr_err("knod_ipsec: crypto-kat FAIL: unexpected verdict=3D0x%08x (low= =3D0x%08x)\n", + verdict_hi, (u32)v); + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: verdict=3D0x%08x", + verdict_hi); + ret =3D -EIO; + goto out_wipe; + } + } + usleep_range(500, 1000); + } + + pr_err("knod_ipsec: crypto-kat FAIL: timeout (bd->act=3D0x%016llx)\n", + READ_ONCE(*act_ptr)); + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "crypto-kat FAIL: timeout"); + ret =3D -ETIMEDOUT; + +out_wipe: + if (sa_backup) { + memcpy(priv->sa_table->kaddr, sa_backup, KNOD_IPSEC_SA_BO_SIZE); + /* publish the restored SA table to the GPU */ + wmb(); + kvfree(sa_backup); + } + return ret; + +#undef CRYPTO_KAT_BD_OFF +#undef CRYPTO_KAT_OUT_OFF +#undef CRYPTO_KAT_KEY_OFF +#undef CRYPTO_KAT_PKT_OFF +#undef CRYPTO_KAT_HTABLE_OFF +#undef CRYPTO_KAT_PKT_LEN +} + +static int knod_ipsec_run_shader_kat(struct knod_ipsec_priv *priv) +{ + static const int batch_sizes[] =3D { 1, 8, 32, KNOD_IPSEC_KAT_MAX_BATCH }; + int i, failed =3D 0; + int last_ok =3D 0; + + if (!priv->work_pool[0].param.kaddr) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch SKIPPED (NOD not attached)"); + pr_info("knod_ipsec: %s\n", knod_ipsec_last_kat_detail); + return 0; + } + + for (i =3D 0; i < ARRAY_SIZE(batch_sizes); i++) { + int rc =3D knod_ipsec_run_shader_kat_n(priv, batch_sizes[i]); + + if (rc) + failed++; + else + last_ok =3D batch_sizes[i]; + } + + if (!failed) { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch PASS (batch sizes 1/8/32/%d, last_ok=3D%d)", + KNOD_IPSEC_PKT_BATCH, last_ok); + pr_info("knod_ipsec: %s\n", knod_ipsec_last_kat_detail); + } + return failed; +} + +static int knod_ipsec_run_kat(void) +{ + struct knod_ipsec_priv *priv =3D ipsec_priv; + int failed =3D 0; + + failed +=3D knod_ipsec_run_cpu_kat(); + + if (priv) { + failed +=3D knod_ipsec_run_shader_kat(priv); + if (priv->work_pool[0].param.kaddr) + failed +=3D knod_ipsec_run_crypto_kat(priv); + else + pr_info("knod_ipsec: crypto-kat SKIPPED (NOD not attached)\n"); + } else { + scnprintf(knod_ipsec_last_kat_detail, + sizeof(knod_ipsec_last_kat_detail), + "shader-dispatch SKIPPED (priv not initialized)"); + } + + knod_ipsec_last_kat_result =3D failed; + return failed; +} + +static void knod_ipsec_stats_sum(struct knod_ipsec_priv *priv, + struct knod_ipsec_stats *dst) +{ + int cpu; + + memset(dst, 0, sizeof(*dst)); + if (!priv) + return; + + for_each_possible_cpu(cpu) { + struct knod_ipsec_stats *s =3D per_cpu_ptr(priv->stats, cpu); + + dst->rx_packets +=3D s->rx_packets; + dst->rx_bytes +=3D s->rx_bytes; + dst->rx_dispatches +=3D s->rx_dispatches; + dst->rx_batch_total +=3D s->rx_batch_total; + if (s->rx_batch_max > dst->rx_batch_max) + dst->rx_batch_max =3D s->rx_batch_max; + dst->rx_sdma_copies_total +=3D s->rx_sdma_copies_total; + dst->rx_sdma_bytes_total +=3D s->rx_sdma_bytes_total; + if (s->rx_sdma_copies_max > dst->rx_sdma_copies_max) + dst->rx_sdma_copies_max =3D s->rx_sdma_copies_max; + dst->rx_drop_icv +=3D s->rx_drop_icv; + dst->rx_drop_replay +=3D s->rx_drop_replay; + dst->rx_drop_no_sa +=3D s->rx_drop_no_sa; + dst->rx_drop_malformed +=3D s->rx_drop_malformed; + dst->rx_drop_desc_full +=3D s->rx_drop_desc_full; + dst->rx_drop_sdma_full +=3D s->rx_drop_sdma_full; + dst->rx_build_ns +=3D s->rx_build_ns; + dst->rx_gpu_ns +=3D s->rx_gpu_ns; + dst->rx_sdma_ns +=3D s->rx_sdma_ns; + dst->rx_finalise_ns +=3D s->rx_finalise_ns; + dst->rx_total_ns +=3D s->rx_total_ns; + dst->rx_idle_ns +=3D s->rx_idle_ns; + dst->sa_add +=3D s->sa_add; + dst->sa_del +=3D s->sa_del; + dst->sa_rekey +=3D s->sa_rekey; + dst->drain_calls +=3D s->drain_calls; + dst->drain_found +=3D s->drain_found; + dst->drain_delivered +=3D s->drain_delivered; + dst->drain_alloc_ns +=3D s->drain_alloc_ns; + dst->drain_copy_ns +=3D s->drain_copy_ns; + dst->drain_proto_ns +=3D s->drain_proto_ns; + dst->drain_gro_ns +=3D s->drain_gro_ns; + dst->drain_total_ns +=3D s->drain_total_ns; + dst->drain_zc_ok +=3D s->drain_zc_ok; + dst->drain_zc_fallback +=3D s->drain_zc_fallback; + dst->finish_produced +=3D s->finish_produced; + dst->rx_peek_total +=3D s->rx_peek_total; + dst->rx_submit_fail +=3D s->rx_submit_fail; + } +} + +static int knod_ipsec_stats_show(struct seq_file *s, void *v) +{ + struct knod_ipsec_priv *priv =3D s->private; + struct knod_ipsec_stats tot; + int i; + + if (!priv) + return -ENODEV; + + knod_ipsec_stats_sum(priv, &tot); + + seq_printf(s, "stats_enabled : %d\n", + static_branch_unlikely(&ipsec_stats_enabled_key) ? 1 : 0); + seq_puts(s, "=3D=3D RX =3D=3D\n"); + seq_printf(s, "rx_packets : %llu\n", tot.rx_packets); + seq_printf(s, "rx_bytes : %llu\n", tot.rx_bytes); + seq_printf(s, "rx_dispatches : %llu\n", tot.rx_dispatches); + seq_printf(s, "rx_batch_total : %llu\n", tot.rx_batch_total); + seq_printf(s, "rx_batch_max : %llu\n", tot.rx_batch_max); + if (tot.rx_dispatches) + seq_printf(s, "rx_batch_avg : %llu\n", + tot.rx_batch_total / tot.rx_dispatches); + seq_printf(s, "rx_sdma_copies : %llu\n", tot.rx_sdma_copies_total); + seq_printf(s, "rx_sdma_cp_max : %llu\n", tot.rx_sdma_copies_max); + seq_printf(s, "rx_sdma_bytes : %llu\n", tot.rx_sdma_bytes_total); + if (tot.rx_dispatches) { + seq_printf(s, "rx_sdma_cp_avg : %llu\n", + tot.rx_sdma_copies_total / tot.rx_dispatches); + seq_printf(s, "rx_sdma_by_avg : %llu\n", + tot.rx_sdma_bytes_total / tot.rx_dispatches); + } + if (tot.rx_sdma_copies_total) + seq_printf(s, "rx_sdma_by_per : %llu\n", + tot.rx_sdma_bytes_total / tot.rx_sdma_copies_total); + seq_printf(s, "rx_drop_icv : %llu\n", tot.rx_drop_icv); + seq_printf(s, "rx_drop_replay : %llu\n", tot.rx_drop_replay); + seq_printf(s, "rx_drop_no_sa : %llu\n", tot.rx_drop_no_sa); + seq_printf(s, "rx_drop_malform: %llu\n", tot.rx_drop_malformed); + seq_printf(s, "rx_drop_descful: %llu\n", tot.rx_drop_desc_full); + seq_printf(s, "rx_drop_sdmaful: %llu\n", tot.rx_drop_sdma_full); + seq_puts(s, "-- RX timing (ns, summed across dispatches) --\n"); + seq_printf(s, "rx_build_ns : %llu\n", tot.rx_build_ns); + seq_printf(s, "rx_gpu_ns : %llu\n", tot.rx_gpu_ns); + seq_printf(s, "rx_sdma_ns : %llu\n", tot.rx_sdma_ns); + seq_printf(s, "rx_finalise_ns : %llu\n", tot.rx_finalise_ns); + seq_printf(s, "rx_total_ns : %llu\n", tot.rx_total_ns); + seq_printf(s, "rx_idle_ns : %llu\n", tot.rx_idle_ns); + if (tot.rx_dispatches) { + seq_printf(s, "rx_build_avg_ns : %llu\n", + tot.rx_build_ns / tot.rx_dispatches); + seq_printf(s, "rx_gpu_avg_ns : %llu\n", + tot.rx_gpu_ns / tot.rx_dispatches); + seq_printf(s, "rx_sdma_avg_ns : %llu\n", + tot.rx_sdma_ns / tot.rx_dispatches); + seq_printf(s, "rx_finalise_avg_ns: %llu\n", + tot.rx_finalise_ns / tot.rx_dispatches); + seq_printf(s, "rx_total_avg_ns : %llu\n", + tot.rx_total_ns / tot.rx_dispatches); + } + seq_puts(s, "=3D=3D Control =3D=3D\n"); + seq_printf(s, "sa_add : %llu\n", tot.sa_add); + seq_printf(s, "sa_del : %llu\n", tot.sa_del); + seq_printf(s, "sa_rekey : %llu\n", tot.sa_rekey); + seq_puts(s, "=3D=3D Debug =3D=3D\n"); + seq_printf(s, "drain_calls : %llu\n", tot.drain_calls); + seq_printf(s, "drain_found : %llu\n", tot.drain_found); + seq_printf(s, "drain_delivered: %llu\n", tot.drain_delivered); + seq_puts(s, "-- drain_rx timing (ns, summed) --\n"); + seq_printf(s, "drain_alloc_ns : %llu\n", tot.drain_alloc_ns); + seq_printf(s, "drain_copy_ns : %llu\n", tot.drain_copy_ns); + seq_printf(s, "drain_proto_ns : %llu\n", tot.drain_proto_ns); + seq_printf(s, "drain_gro_ns : %llu\n", tot.drain_gro_ns); + seq_printf(s, "drain_total_ns : %llu\n", tot.drain_total_ns); + if (tot.drain_delivered) { + seq_printf(s, "drain_alloc_per: %llu\n", + tot.drain_alloc_ns / tot.drain_delivered); + seq_printf(s, "drain_copy_per : %llu\n", + tot.drain_copy_ns / tot.drain_delivered); + seq_printf(s, "drain_proto_per: %llu\n", + tot.drain_proto_ns / tot.drain_delivered); + seq_printf(s, "drain_gro_per : %llu\n", + tot.drain_gro_ns / tot.drain_delivered); + seq_printf(s, "drain_total_per: %llu\n", + tot.drain_total_ns / tot.drain_delivered); + } + if (tot.drain_calls) + seq_printf(s, "drain_pkts_call: %llu\n", + tot.drain_delivered / tot.drain_calls); + seq_printf(s, "finish_produced: %llu\n", tot.finish_produced); + seq_printf(s, "rx_peek_total : %llu\n", tot.rx_peek_total); + seq_printf(s, "rx_submit_fail : %llu\n", tot.rx_submit_fail); + seq_printf(s, "drain_zc_ok : %llu\n", tot.drain_zc_ok); + seq_printf(s, "drain_zc_fallback: %llu\n", tot.drain_zc_fallback); + + /* Per-queue SPSC ring state - shows where bds are stuck */ + if (priv && priv->knodev) { + int nr_q =3D priv->knodev->netdev ? + priv->knodev->netdev->real_num_rx_queues : 0; + + if (nr_q > KNOD_SPSC_MAX) + nr_q =3D KNOD_SPSC_MAX; + seq_puts(s, "=3D=3D SPSC per-queue =3D=3D\n"); + for (i =3D 0; i < nr_q; i++) { + struct spsc_ring *r =3D &priv->knodev->wpriv[i].spsc_bds; + + if (!r->slots || r->mask =3D=3D 0) + continue; + /* Only show queues with non-zero activity */ + if (r->head =3D=3D 0 && r->acquired =3D=3D 0 && r->tail =3D=3D 0) + continue; + seq_printf(s, " q%02d: head=3D%u acq=3D%u tail=3D%u (unpeek=3D%u infli= ght=3D%u)\n", + i, r->head, r->acquired, r->tail, + r->head - r->acquired, + r->acquired - r->tail); + } + } + + /* Dispatcher */ + if (priv) { + int d, n_running =3D 0; + + for (d =3D 0; d < priv->nr_dispatchers; d++) + if (priv->disp[d].kthread) + n_running++; + seq_printf(s, "=3D=3D Dispatchers =3D=3D\n running=3D%d/%d\n", + n_running, priv->nr_dispatchers); + for (d =3D 0; d < priv->nr_dispatchers; d++) { + struct knod_ipsec_dispatcher *disp =3D &priv->disp[d]; + + seq_printf(s, + " disp[%d]: kaql=3D%d work=3D[%d..%d) rxq=3D[%d..%d)\n", + d, disp->kaql_idx, + disp->work_first, + disp->work_first + disp->work_count, + disp->rxq_first, + disp->rxq_first + disp->rxq_count); + } + } + + return 0; +} + +static int knod_ipsec_stats_open(struct inode *inode, struct file *file) +{ + return single_open(file, knod_ipsec_stats_show, inode->i_private); +} + +static const struct file_operations knod_ipsec_stats_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_stats_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t knod_ipsec_stats_enable_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *ppos) +{ + char buf[4] =3D {}; + int val; + + if (len =3D=3D 0 || len > sizeof(buf) - 1) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + if (kstrtoint(strim(buf), 0, &val)) + return -EINVAL; + + if (val) + static_branch_enable(&ipsec_stats_enabled_key); + else + static_branch_disable(&ipsec_stats_enabled_key); + return len; +} + +static int knod_ipsec_stats_enable_show(struct seq_file *s, void *v) +{ + seq_printf(s, "%d\n", + static_branch_unlikely(&ipsec_stats_enabled_key) ? 1 : 0); + return 0; +} + +static int knod_ipsec_stats_enable_open(struct inode *inode, struct file *= file) +{ + return single_open(file, knod_ipsec_stats_enable_show, NULL); +} + +static const struct file_operations knod_ipsec_stats_enable_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_stats_enable_open, + .read =3D seq_read, + .write =3D knod_ipsec_stats_enable_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t knod_ipsec_poll_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *ppos) +{ + char buf[4] =3D {}; + int val; + + if (len =3D=3D 0 || len > sizeof(buf) - 1) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + if (kstrtoint(strim(buf), 0, &val)) + return -EINVAL; + + WRITE_ONCE(knod_ipsec_poll_mode, !!val); + return len; +} + +static int knod_ipsec_poll_show(struct seq_file *s, void *v) +{ + seq_printf(s, "%d\n", READ_ONCE(knod_ipsec_poll_mode) ? 1 : 0); + return 0; +} + +static int knod_ipsec_poll_open(struct inode *inode, struct file *file) +{ + return single_open(file, knod_ipsec_poll_show, NULL); +} + +static const struct file_operations knod_ipsec_poll_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_poll_open, + .read =3D seq_read, + .write =3D knod_ipsec_poll_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t knod_ipsec_pkt_batch_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *ppos) +{ + struct knod_ipsec_priv *priv =3D file_inode(file)->i_private; + char buf[8] =3D {}; + u32 val; + + if (!priv) + return -ENODEV; + if (len =3D=3D 0 || len > sizeof(buf) - 1) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + if (kstrtou32(strim(buf), 0, &val)) + return -EINVAL; + if (val < 1 || val > KNOD_IPSEC_PKT_BATCH) + return -ERANGE; + + WRITE_ONCE(priv->pkt_batch, val); + return len; +} + +static int knod_ipsec_pkt_batch_show(struct seq_file *s, void *v) +{ + struct knod_ipsec_priv *priv =3D s->private; + + if (!priv) + return -ENODEV; + seq_printf(s, "%u\n", READ_ONCE(priv->pkt_batch)); + return 0; +} + +static int knod_ipsec_pkt_batch_open(struct inode *inode, struct file *fil= e) +{ + return single_open(file, knod_ipsec_pkt_batch_show, inode->i_private); +} + +static const struct file_operations knod_ipsec_pkt_batch_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_pkt_batch_open, + .read =3D seq_read, + .write =3D knod_ipsec_pkt_batch_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t knod_ipsec_stats_reset_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *ppos) +{ + struct knod_ipsec_priv *priv =3D file_inode(file)->i_private; + int cpu; + + if (!priv) + return -ENODEV; + + for_each_possible_cpu(cpu) { + struct knod_ipsec_stats *s =3D per_cpu_ptr(priv->stats, cpu); + u64 sa_add =3D s->sa_add, sa_del =3D s->sa_del; + u64 sa_rekey =3D s->sa_rekey; + + memset(s, 0, sizeof(*s)); + s->sa_add =3D sa_add; + s->sa_del =3D sa_del; + s->sa_rekey =3D sa_rekey; + } + return len; +} + +static const struct file_operations knod_ipsec_stats_reset_fops =3D { + .owner =3D THIS_MODULE, + .open =3D simple_open, + .write =3D knod_ipsec_stats_reset_write, +}; + +static int knod_ipsec_sa_table_show(struct seq_file *s, void *v) +{ + struct knod_ipsec_priv *priv =3D s->private; + int i, used =3D 0; + + if (!priv) + return -ENODEV; + + mutex_lock(&priv->slot_lock); + for (i =3D 0; i < KNOD_IPSEC_NR_SA; i++) { + struct knod_ipsec_sa_slot *slot =3D &priv->slots[i]; + + if (!slot->active) + continue; + seq_printf(s, "slot[%3d] spi=3D0x%08x version=3D%u\n", + i, slot->spi, slot->version); + used++; + } + mutex_unlock(&priv->slot_lock); + seq_printf(s, "used: %d / %d\n", used, KNOD_IPSEC_NR_SA); + return 0; +} + +static int knod_ipsec_sa_table_open(struct inode *inode, struct file *file) +{ + return single_open(file, knod_ipsec_sa_table_show, inode->i_private); +} + +static const struct file_operations knod_ipsec_sa_table_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_sa_table_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +/* + * Disassembly goes through the shared amdgcn_disasm_raw() (knod_amdgpu_in= sn.h): + * it classifies each raw instruction into a struct amdgcn_insn and prints= it + * via the complete-opnames disassembler, so every feature shares one deco= der + * instead of the old per-feature hand-rolled hex re-parsers. + */ +static int knod_ipsec_insn_show(struct seq_file *s, void *v) +{ + struct knod_ipsec_priv *priv =3D s->private; + struct kernel_descriptor *kd; + u32 *code; + int ndw, off; + + if (!priv || !priv->knod || !priv->knod->kernels[0]) + return -ENODEV; + + kd =3D priv->knod->kernels[0]->kaddr; + code =3D priv->knod->kernels[0]->kaddr + + kd->kernel_code_entry_byte_offset; + ndw =3D max_t(int, 1, (int)priv->shader_size / 4); + + seq_printf(s, "=3D=3D=3D IPsec fused RX shader (GFX%d, %zu bytes, %d dwor= ds) =3D=3D=3D\n", + priv->isa_version, priv->shader_size, ndw); + seq_printf(s, "kernel_code_gaddr: 0x%llx\n\n", + priv->knod->kernels[0]->gaddr + + kd->kernel_code_entry_byte_offset); + + off =3D 0; + while (off < ndw) { + int adv; + + seq_printf(s, "%04x: ", off * 4); + adv =3D amdgcn_disasm_raw(priv->isa_version, &code[off], + ndw - off, s); + if (adv <=3D 0) + break; + off +=3D adv; + } + + if (ndw =3D=3D 1 && code[0] =3D=3D 0xBF810000) + seq_puts(s, "\n(empty shader: single s_endpgm)\n"); + + return 0; +} + +static int knod_ipsec_insn_open(struct inode *inode, struct file *file) +{ + return single_open(file, knod_ipsec_insn_show, inode->i_private); +} + +static const struct file_operations knod_ipsec_insn_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_insn_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t knod_ipsec_selftest_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *ppos) +{ + char buf[4] =3D {}; + int val; + + if (len =3D=3D 0 || len > sizeof(buf) - 1) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + if (kstrtoint(strim(buf), 0, &val)) + return -EINVAL; + if (val) + knod_ipsec_run_kat(); + return len; +} + +static int knod_ipsec_selftest_show(struct seq_file *s, void *v) +{ + seq_puts(s, "write 1 to trigger in-kernel KAT\n"); + seq_puts(s, " cpu layer : H =3D AES_K(0^128) for aes128/192/256, "); + seq_printf(s, "%d random keys each\n", KNOD_IPSEC_KAT_RAND_ROUNDS); + seq_printf(s, " gpu layer : fused-shader dispatch at batch sizes 1/8/3= 2/%d\n", + KNOD_IPSEC_PKT_BATCH); + seq_puts(s, " crypto layer: AES-128-GCM full decrypt+ICV verify via GPU = shader\n"); + seq_puts(s, " tx-crypto : AES-128-GCM full encrypt+ICV generate via GP= U shader\n"); + if (knod_ipsec_last_kat_result < 0) { + seq_puts(s, "last run: never\n"); + } else if (knod_ipsec_last_kat_result =3D=3D 0) { + seq_puts(s, "last run: PASS\n"); + } else { + seq_printf(s, "last run: FAIL (%d checks failed)\n", + knod_ipsec_last_kat_result); + } + if (knod_ipsec_last_kat_detail[0]) + seq_printf(s, "detail: %s\n", knod_ipsec_last_kat_detail); + return 0; +} + +static int knod_ipsec_selftest_open(struct inode *inode, struct file *file) +{ + return single_open(file, knod_ipsec_selftest_show, NULL); +} + +static const struct file_operations knod_ipsec_selftest_fops =3D { + .owner =3D THIS_MODULE, + .open =3D knod_ipsec_selftest_open, + .read =3D seq_read, + .write =3D knod_ipsec_selftest_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +/* + * debugfs lives under the shared knod ctx dir so IPsec files coexist + * with MACsec (aesgcm_selftest) and WG (wg subdir) under one tree: + * /sys/kernel/debug/dri//knod/ipsec/{stats,selftest,insn,...} + * + * The knod ctx is owned by knod_init/knod_exit, so debugfs lifecycle is + * tied to those callbacks rather than module init/exit. + */ +static void knod_ipsec_debugfs_init(struct knod_ipsec_priv *priv) +{ + struct dentry *parent; + + if (!priv || !priv->knod || !priv->knod->debug_dir) + return; + + parent =3D debugfs_create_dir("ipsec", priv->knod->debug_dir); + if (IS_ERR_OR_NULL(parent)) + return; + priv->debug_dir =3D parent; + + /* Populate this module's opcode-name tables for the shared + * disassembler (amdgcn_disasm_raw, used by the "insn" file). Each + * feature module carries its own copy of the opnames_gfx9/10 tables + * (knod_amdgpu_insn.h), so the tables must be initialised here rather + * than relying on the BPF module having done it. + */ + + debugfs_create_file("stats", 0444, parent, + priv, &knod_ipsec_stats_fops); + debugfs_create_file("stats_enable", 0644, parent, + priv, &knod_ipsec_stats_enable_fops); + debugfs_create_file("stats_reset", 0200, parent, + priv, &knod_ipsec_stats_reset_fops); + debugfs_create_file("sa_table", 0444, parent, + priv, &knod_ipsec_sa_table_fops); + debugfs_create_file("insn", 0444, parent, + priv, &knod_ipsec_insn_fops); + debugfs_create_file("selftest", 0644, parent, + priv, &knod_ipsec_selftest_fops); + debugfs_create_file("poll", 0644, parent, + priv, &knod_ipsec_poll_fops); + debugfs_create_file("pkt_batch", 0644, parent, + priv, &knod_ipsec_pkt_batch_fops); +} + +static void knod_ipsec_debugfs_exit(struct knod_ipsec_priv *priv) +{ + if (!priv) + return; + debugfs_remove_recursive(priv->debug_dir); + priv->debug_dir =3D NULL; +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * Policy offload - accept PACKET-mode policies so xfrm_state_find() + * will match our PACKET-mode SAs. No GPU-side action needed; we just + * return 0 to let the kernel record the offload type on the policy. + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ +static int knod_ipsec_xdo_policy_add(struct knod_dev *knodev, + struct xfrm_policy *xp, + struct netlink_ext_ack *extack) +{ + return 0; /* accept unconditionally */ +} + +static void knod_ipsec_xdo_policy_delete(struct knod_dev *knodev, + struct xfrm_policy *xp) +{ + /* nothing to clean up on the GPU side */ +} + +static void knod_ipsec_xdo_policy_free(struct knod_dev *knodev, + struct xfrm_policy *xp) +{ + /* nothing to free */ +} + +/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + * knod_accel_ipsec_ops registration + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + */ + +static struct knod_accel_ipsec_ops knod_ipsec_ops =3D { + /* feature select: alloc/free the IPsec GPU resources */ + .activate =3D knod_ipsec_nod_init, + .deactivate =3D knod_ipsec_nod_exit, + .busy =3D knod_ipsec_nod_busy, + /* interface up/down (or feature switch): dispatchers + GPU drain */ + .start =3D knod_ipsec_nod_start, + .stop =3D knod_ipsec_nod_stop, + .xdo_dev_state_add =3D knod_ipsec_xdo_state_add, + .xdo_dev_state_delete =3D knod_ipsec_xdo_state_delete, + .xdo_dev_state_free =3D knod_ipsec_xdo_state_free, + .xdo_dev_offload_ok =3D knod_ipsec_xdo_offload_ok, + .xdo_dev_state_advance_esn =3D knod_ipsec_xdo_state_advance_esn, + .xdo_dev_state_update_stats =3D knod_ipsec_xdo_state_update_stats, + .xdo_dev_policy_add =3D knod_ipsec_xdo_policy_add, + .xdo_dev_policy_delete =3D knod_ipsec_xdo_policy_delete, + .xdo_dev_policy_free =3D knod_ipsec_xdo_policy_free, +}; + +static int __init knod_ipsec_init(void) +{ + pr_debug("knod_ipsec: module load\n"); + + /* Publish our dispatcher-count requirement to the shared knod + * core before any NOD attach happens so knod_attach() creates a + * context with enough kaql/sdma pairs. knod_request_queue_cnt + * is a high-water mark so raising it here is idempotent and + * won't stomp on a larger value from another accel. + */ + knod_request_queue_cnt(clamp(nr_dispatch, 1, + KNOD_IPSEC_MAX_DISPATCHERS)); + + /* knod_accel_ipsec_register() already attaches and inits the ipsec + * ops on every registered accel via knod_ipsec_attach(), so there is no + * separate per-accel init loop here: a second init() would only be + * rejected with -EBUSY by the ipsec_priv guard and log a misleading + * "init failed (-16)". + */ + knod_dev_lock(); + knod_accel_ipsec_register(&knod_ipsec_ops); + knod_dev_unlock(); + return 0; +} + +static void __exit knod_ipsec_exit(void) +{ + struct knod_accel *accel; + + rtnl_lock(); + knod_dev_lock(); + for_each_accel(accel) { + if (!strncmp(accel->name, "amdgpu-", 7) && accel->knodev) { + if (accel->accel_ops->ipsec_ops) + accel->accel_ops->ipsec_ops->exit( + accel->knodev); + } + } + knod_accel_ipsec_unregister(); + knod_dev_unlock(); + rtnl_unlock(); + + pr_debug("knod_ipsec: module unload\n"); +} + +module_init(knod_ipsec_init); +module_exit(knod_ipsec_exit); + +MODULE_DESCRIPTION("AMDGPU IPsec (xfrm) full-packet offload via KNOD"); +MODULE_AUTHOR("Taehee Yoo "); +MODULE_LICENSE("GPL"); diff --git a/drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h b/drivers/gpu/drm= /amd/amdkfd/knod/knod_ipsec.h new file mode 100644 index 000000000000..3754a86d6c63 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h @@ -0,0 +1,596 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2021 Taehee Yoo + * Copyright (c) 2021 Hoyeon Lee + */ + +#ifndef KNOD_IPSEC_H_ +#define KNOD_IPSEC_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include /* KNOD_SPSC_MAX */ + +struct knod; +struct knod_mem; +struct dentry; +struct xfrm_state; +struct sk_buff; +struct task_struct; +struct net_device; +struct napi_struct; +struct knod_ipsec_priv; + +#define KNOD_IPSEC_NR_SA 256 +/* The shader-dispatch KAT installs one fake SA per fused_sub entry, so + * its maximum batch size is bounded by NR_SA. Keep separate from + * KNOD_IPSEC_PKT_BATCH so production can run larger batches without + * growing the SA table BO. + */ +#define KNOD_IPSEC_KAT_MAX_BATCH 64 +/* 128-bit window, 4 u32 words */ +#define KNOD_IPSEC_REPLAY_WORDS 4 +#define KNOD_IPSEC_REPLAY_BITS (KNOD_IPSEC_REPLAY_WORDS * 32) +#define KNOD_IPSEC_REPLAY_BYTES (KNOD_IPSEC_REPLAY_WORDS * 4) + +/* + * CPU-side per-queue sliding anti-replay window (knod_ipsec_sa_window). + * Independent from the GPU-side REPLAY_BITS above - the shader's + * replay region stays 128 bits for GPU-visible state, but the CPU path + * needs a much larger window because with multi-queue RSS the packets + * of a single SA may land on several RX queues due to NIC hash + * collisions. Each per-queue window then sees only a sparse subset of + * the SA's monotonic seq stream, and the seq gap between consecutive + * packets arriving at one queue can exceed 128 easily. 2048 bits lets + * the window absorb worst-case gaps (e.g. 8 queues x 256 batch). + * + * Sized as u64 words for easier bit-shift logic in the check function. + */ +#define KNOD_IPSEC_CPU_REPLAY_WORDS 32 /* 32 * 64 =3D 2048 bits */ +#define KNOD_IPSEC_CPU_REPLAY_BITS \ + (KNOD_IPSEC_CPU_REPLAY_WORDS * 64) + +/* + * Per-dispatch packet batch. The RX fused shader launches grid_y=3Dnr_pac= kets + * workgroups and each workgroup indexes its own sub[wg_y] entry, so batch + * scales linearly with GPU occupancy up to the LDS-bound concurrent + * workgroup limit (~1024 on 64-CU gfx9 with 4 KB LDS per WG). Bumping this + * grows: + * - struct knod_ipsec_fused_param (sub[] inline in kernarg) + * - work_decrypt_pool VRAM (BATCH * DECRYPT_PKT_SIZE) + * - struct knod_ipsec_work (rx_bds[], rx_pkt_queue[], rx_pendi= ng[]) + */ +#define KNOD_IPSEC_PKT_BATCH 512 + +/* + * Kernarg layout consumed by the fused RX shader. + * + * The shader dereferences sa_table_addr/t_tables_addr from the top-level + * struct and then iterates `sub[]` for per-packet work. Decrypted inner + * packets are written to `out_addr` (VRAM); a CPU finish worker later + * SDMA-copies them into a per-queue framework delivery-pool page and + * publishes a `knod_pass_desc` onto the framework pass_pending ring. The + * bd ring is not used for verdict delivery. + */ +struct knod_ipsec_fused_sub { + __le64 pkt_addr; /* raw packet VRAM addr (ETH start) */ + __le64 out_addr; /* decrypted inner packet dest */ + __le64 bd_addr; /* SPSC bd for direct verdict write */ + __le32 pkt_len; + __le32 result_seq; /* shader writes bswapped ESP seq here */ +}; + +struct knod_ipsec_fused_param { + __le64 sa_table_addr; /* 0 */ + __le64 t_tables_addr; /* 8 */ + __le64 sdma_ring_addr; /* 16: SDMA ring buffer gaddr */ + __le32 nr_sa; /* 24 */ + __le32 family_filter; /* 28 */ + __le64 sdma_ctl_addr; /* 32: GPU VA of sdma_ctl region */ + struct knod_ipsec_fused_sub sub[KNOD_IPSEC_PKT_BATCH]; /* 40 */ +}; + +/* + * GPU-visible SA table entry. Layout is shared with the GPU shader and + * must remain stable / packed. + */ +struct knod_ipsec_sa_entry { + /* network byte order in wire, LE in table */ + __le32 spi; + __le32 dir; /* 0=3DIN, 1=3DOUT */ + __le32 family; /* AF_INET=3D2, AF_INET6=3D10 */ + __le32 flags; /* bit0: ESN, bit1: CRYPT_ONLY */ + __le64 key_gpu_addr; /* AES key buffer (VRAM) */ + __le64 htable_gpu_addr; /* GHASH H-power table (VRAM) */ + __le64 t_tables_gpu_addr; /* shared T-tables (VRAM) */ + u8 salt[4]; + __le32 key_len; /* 16/24/32 */ + __le32 nr_rounds; /* AES rounds: 10/12/14 */ + /* XFRM_MODE_TRANSPORT=3D0, XFRM_MODE_TUNNEL=3D1 */ + __le32 mode; + __le64 replay_bitmap_addr; /* GPU-visible replay bitmap */ + __le32 replay_window; /* 64/128/256 */ + __le32 seq_hi; /* ESN high-32 */ + __le64 seq_last; /* last accepted sequence number */ + __le32 active; + __le32 version; /* rekey protection */ + __le64 stats_addr; /* per-SA counters (optional) */ + __le64 _pad1; +}; + +#define KNOD_IPSEC_SA_ENTRY_SIZE sizeof(struct knod_ipsec_sa_entry) +#define KNOD_IPSEC_SA_TABLE_SIZE \ + (KNOD_IPSEC_NR_SA * KNOD_IPSEC_SA_ENTRY_SIZE) + +/* + * Replay bitmap + per-SA stats regions are appended to the SA entry table + * inside the same backing BO. Keeping everything in one BO avoids the + * multi-BO GPU VA mapping bug (see gtt_multi_bo_bug.md). + */ +#define KNOD_IPSEC_REPLAY_REGION_SIZE \ + (KNOD_IPSEC_NR_SA * KNOD_IPSEC_REPLAY_BYTES) +#define KNOD_IPSEC_REPLAY_REGION_OFF KNOD_IPSEC_SA_TABLE_SIZE + +/* + * Per-SA GPU-visible stats. The shader atomically increments these via + * global_atomic_add_x2 on every successful decrypt. CPU reads them back + * in xdo_state_update_stats to feed x->curlft. + */ +struct knod_ipsec_sa_gpu_stats { + __le64 rx_packets; + __le64 rx_bytes; +}; + +#define KNOD_IPSEC_SA_STATS_SIZE sizeof(struct knod_ipsec_sa_gpu_stats) +#define KNOD_IPSEC_STATS_REGION_SIZE \ + (KNOD_IPSEC_NR_SA * KNOD_IPSEC_SA_STATS_SIZE) +#define KNOD_IPSEC_STATS_REGION_OFF \ + (KNOD_IPSEC_REPLAY_REGION_OFF + KNOD_IPSEC_REPLAY_REGION_SIZE) + +#define KNOD_IPSEC_SA_BO_SIZE \ + (KNOD_IPSEC_SA_TABLE_SIZE + KNOD_IPSEC_REPLAY_REGION_SIZE + \ + KNOD_IPSEC_STATS_REGION_SIZE) + +/* + * RFC 4303 anti-replay sliding window - CPU-side, per-SA, per-RXQ. + * + * RSS hashes ESP flows on (saddr, daddr, proto, SPI) so that every packet + * of a given SA lands on the same NIC RX queue. That means a single writer + * (the NIC dd NAPI for that queue) owns the window and no locking is + * required on the fast path. Replicated per queue because different SAs + * may still be hashed to different queues, and we do not want false + * sharing between queues on a shared cacheline. + */ +struct knod_ipsec_sa_window { + u64 top_seq; /* highest accepted seq */ + /* N*64-bit sliding window */ + u64 bitmap[KNOD_IPSEC_CPU_REPLAY_WORDS]; +}; + +/* CPU-side slot metadata */ +struct knod_ipsec_sa_slot { + struct xfrm_state *x; /* back pointer (CPU only) */ + struct knod_mem *key_mem; + struct knod_mem *htable_mem; + struct knod_mem *replay_mem; + u32 spi; /* host order */ + u32 slot_idx; + u32 version; + bool active; + /* Per-RXQ sliding window state. Owned by the NIC dd NAPI for that + * queue - do not touch from control plane while SA is active. + */ + struct knod_ipsec_sa_window win[KNOD_SPSC_MAX]; +}; + +/* Percpu stats for observability */ +struct knod_ipsec_stats { + /* RX */ + u64 rx_packets; + u64 rx_bytes; + u64 rx_dispatches; + u64 rx_batch_total; + u64 rx_batch_max; + /* SDMA copy observability: per dispatch we emit 1 copy per raw-bypass + * or tunnel packet, 2 copies per transport packet (outer L3 + inner + * payload). rx_sdma_copies_total accumulates every call; max tracks + * the peak single-dispatch count; bytes tracks total DMA volume. + */ + u64 rx_sdma_copies_total; + u64 rx_sdma_copies_max; + u64 rx_sdma_bytes_total; + u64 rx_drop_icv; + u64 rx_drop_replay; + u64 rx_drop_no_sa; + u64 rx_drop_malformed; + u64 rx_drop_desc_full; + u64 rx_drop_sdma_full; + /* Per-phase RX dispatch timings, accumulated per percpu counter. + * Hot path guarded by ipsec_stats_enabled_key static branch so + * they cost zero cycles when disabled. + * + * rx_build_ns : try_rx pre-scan + per-queue drain loop + * (CPU staging work into kernarg sub[]). + * rx_gpu_ns : spin on GPU completion signal. + * rx_sdma_ns : spin on SDMA fence after finish scheduled + * per-packet SDMA copies. + * rx_finalise_ns : finish_rx_deliver CPU work excluding SDMA + * fence wait (verdict loop + desc publish + + * per-queue NAPI schedule). + * rx_total_ns : end-to-end try_rx call time (build + dispatch + * wait + finalise + napi kicks). + * rx_idle_ns : time the dispatcher spent in usleep_range + * waiting for work when both try_tx and + * try_rx returned false. + */ + u64 rx_build_ns; + u64 rx_gpu_ns; + u64 rx_sdma_ns; + u64 rx_finalise_ns; + u64 rx_total_ns; + u64 rx_idle_ns; + /* Control plane */ + u64 sa_add; + u64 sa_del; + u64 sa_rekey; + /* Debug: drain_rx pipeline visibility */ + u64 drain_calls; + u64 drain_found; + u64 drain_delivered; + u64 finish_produced; + u64 rx_peek_total; + u64 rx_submit_fail; + /* drain_rx per-phase timing (ns, summed). Each drain_rx call + * processes `drain_found` descriptors; these buckets split the + * CPU work per-phase so we can see which step dominates: + * drain_alloc_ns : knod_pass_build_skb cost + * drain_copy_ns : legacy copy cost (0 - delivery is zero-copy) + * drain_proto_ns : L3 header patch + secpath setup + * drain_gro_ns : netif_receive_skb_list (stack entry) + * drain_total_ns : end-to-end drain_rx call time + * Guarded by ipsec_stats_enabled_key so zero cost when off. + */ + u64 drain_alloc_ns; + u64 drain_copy_ns; + u64 drain_proto_ns; + u64 drain_gro_ns; + u64 drain_total_ns; + u64 drain_zc_ok; + u64 drain_zc_fallback; +}; + +/* + * Work-slot pool owned by the dispatcher kthread. Depth-2 pipelining: + * while one slot is executing on the GPU, the dispatcher can build and + * submit the next batch into the other slot and finalise the one that + * just completed. kaql[0] is still a single AQL queue; multiple + * in-flight dispatches are queued in-order and complete in-order. + * + * A slot's lifecycle: + * EMPTY -> try_tx/try_rx fills kernarg and submits -> INFLIGHT + * INFLIGHT -> GPU running; dispatcher polls completion signal + * INFLIGHT -> signal fires -> start SDMA copies + fence (no spin) + * -> SDMA_PENDING + * SDMA_PENDING -> dispatcher polls SDMA fence (non-blocking) + * SDMA_PENDING -> fence done -> desc publish, napi kicks, bd PASS, + * stats -> EMPTY + * + * The SDMA_PENDING state decouples the SDMA fence wait from the + * dispatcher loop so the CPU never busy-spins on the fence. While + * one slot sits in SDMA_PENDING, the dispatcher can build and + * submit the next batch into another EMPTY slot - true 3-way + * parallelism of GPU execution, SDMA transfer, and CPU build. + */ +#define KNOD_IPSEC_NR_WORK 4 + +/* + * Front offset reserved in each delivery page (=3D IPv4 transport L3 size= ). + * The shader decrypts into a per-work VRAM output buffer; finish_rx_deliv= er + * then SDMA-lays the packet into the delivery page by mode: + * + * transport: outer L3 header (20B IPv4 / 40B IPv6) to page+0, decrypted + * L4 payload to page+l3_len; inner_off =3D 0. + * tunnel: decrypted inner L3 packet to page+GTT_OUT_L3_OFF, leaving + * 20B of headroom at the front; inner_off =3D 20. + * + * (The name is a holdover from a removed mode where the shader wrote + * straight into a GTT slot at this offset.) + */ +#define KNOD_IPSEC_GTT_OUT_L3_OFF 20 + +/* + * GPU-initiated SDMA control block, placed at a fixed offset after + * knod_ipsec_fused_param in the same kernarg BO. The shader's Phase 12 + * uses atomic counters here to coordinate multi-workgroup SDMA ring + * writes without CPU involvement. + */ +#define KNOD_IPSEC_SDMA_CTL_OFF \ + ALIGN(sizeof(struct knod_ipsec_fused_param), 64) + +struct knod_ipsec_sdma_ctl { + /* 0: atomic - SDMA-needing WGs increment */ + __le32 claim_counter; + __le32 done_counter; /* 4: atomic - ALL WGs increment */ + /* 8: current wptr byte offset (CPU snapshot) */ + __le64 wptr_val; + __le64 fence_addr; /* 16: SDMA fence write target GPU VA */ + /* 24: value SDMA writes on completion */ + __le32 fence_val; + /* 28: wptr_val / 4 (dword index into ring) */ + __le32 wptr_base_dw; + __le32 ring_mask; /* 32: (ring_size_bytes/4) - 1 */ + __le32 nr_total_wg; /* 36: grid_size_y =3D nr_packets */ + __le32 copy_hdr; /* 40: SDMA COPY_LINEAR header dword */ + __le32 fence_hdr; /* 44: SDMA FENCE header dword */ + __le32 gpu_sdma_ready; /* 48: last WG sets 1 -> CPU polls */ + /* 52: total SDMA COPY packets emitted */ + __le32 final_sdma_count; + /* 56: GPU VA of HW wptr (queue->gaddr+8) */ + __le64 wptr_gpu_addr; +}; + +enum knod_ipsec_work_state { + KNOD_WORK_EMPTY =3D 0, + KNOD_WORK_INFLIGHT, + KNOD_WORK_SDMA_PENDING, +}; + +/* + * Per-work RX decrypt output buffer size. AES-CTR decrypt writes plaintext + * here instead of overwriting the ciphertext in-place (which would corrupt + * the data before GHASH reads it). One slot per packet in the batch. + * + * Must hold the largest expected plaintext (ctext_len, before ESP trailer + * strip). Sized for MTU 9000 ESP jumbo frames (up to ~9200B total), + * rounded up to 16 KB. At PKT_BATCH=3D64 the total pool is 1 MB of VRAM. + * Smaller values silently corrupt adjacent slots and eventually fault + * past the end of the pool BO. + */ +#define KNOD_IPSEC_DECRYPT_PKT_SIZE 16384 +#define KNOD_IPSEC_DECRYPT_WORK_SIZE \ + ((size_t)KNOD_IPSEC_PKT_BATCH * KNOD_IPSEC_DECRYPT_PKT_SIZE) + +/* + * Per-work kernarg and decrypt buffers are SLICES of a single large + * BO owned by `struct knod_ipsec_works`, not individual BOs. Allocating + * many small BOs and mapping them all to the KFD process GPU VA hits a + * long-standing AMDKFD issue where only the first BO is reliably mapped + * (see memory/gtt_multi_bo_bug.md) - subsequent BOs fault on GPU access. + * One large pool BO, sliced at fixed offsets, sidesteps this entirely. + */ +struct knod_ipsec_slice { + void *kaddr; + u64 gaddr; +}; + +/* + * Per-packet finish state built by knod_ipsec_finish_rx_deliver() while it + * schedules SDMA copies, then read back to publish desc_ring entries. + * + * Kept as an array inside struct knod_ipsec_work so the dispatcher does + * not have to stack-allocate BATCH * sizeof(...) on every finalise - at + * large PKT_BATCH values (512+) stack allocation would overflow the + * 16 KB kernel stack. + */ +struct knod_ipsec_rx_pending { + netmem_ref netmem; /* delivery page from framework pass_pool */ + u32 inner_len; /* bytes copied into the delivery page */ + u32 sa_slot; /* SA table index, or KNOD_IPSEC_NR_SA (raw) */ + u16 rxq_idx; /* which priv->rxq[] this packet belongs to */ + u8 mode; /* XFRM_MODE_TRANSPORT / TUNNEL */ + u8 next_hdr; /* ESP trailer next_hdr */ + u8 family; /* AF_INET / AF_INET6 */ + u8 inner_off; /* byte offset within the delivery page */ + u8 _pad[2]; +}; + +struct knod_ipsec_work { + /* Views into the shared pools in priv. No per-work BO. */ + struct knod_ipsec_slice param; /* kernarg slice (VRAM pool) */ + /* RX decrypt output (VRAM pool) */ + u64 rx_out_gaddr; + /* Pipelined slot state + deferred finalise tracking. The dispatcher + * sets state to INFLIGHT on submit, polls completion, then flips to + * EMPTY after finalise. Timestamps / per-queue napi info captured + * at build time get consumed when finalise runs later. + */ + enum knod_ipsec_work_state state; + u64 t_build_start; + u64 t_build_end; + u64 t_submit; + u64 t_finalise_start; + /* Per-queue tracker for deferred napi_schedule. try_rx records + * which NIC RX queues had packets drained into this slot so the + * finalise path (running later, possibly one iteration later) can + * wake those NAPIs after finalise completes. + */ + u16 per_q_touched[KNOD_SPSC_MAX]; + int per_q_n; + /* Per-dispatch state (owned by dispatcher) */ + s64 sigval; + u64 dispatch_ts; + /* SDMA deferred-fence state. When the work transitions from + * INFLIGHT -> SDMA_PENDING, finish_rx_deliver queues copies + + * fence but does NOT spin. The dispatcher checks sdma_fence_ptr + * on the next iteration and transitions to EMPTY once the fence + * fires. sdma_fence_target is the expected fence value; the + * pointer is the host-visible signal->value location. + */ + u32 sdma_fence_target; + s64 *sdma_fence_ptr; + u64 sdma_submit_ns; + /* Carry the SDMA fence wait time out of finish_rx_deliver so + * the caller (try_rx) can subtract it from the finalise phase + * and report it as rx_sdma_ns. Zero when stats are disabled. + */ + u64 sdma_wait_ns; + /* Per-dispatch SDMA copy accounting (populated by finish_rx_deliver). + * rx_sdma_copies =3D number of knod_sdma_copy() calls issued this + * batch (1 per raw/tunnel packet, 2 per transport packet). bytes =3D + * total bytes DMA'd. Zero when stats are disabled. + */ + u32 rx_sdma_copies; + u32 rx_sdma_bytes; + /* Deferred SDMA completion state. finish_rx_deliver stores + * n_sdma_pending + pkt_idx_of[] so the dispatcher's + * SDMA_PENDING -> EMPTY transition can publish descs and mark + * bds without re-scanning the verdict loop. + */ + int n_sdma_pending; + u16 sdma_pkt_idx_of[KNOD_IPSEC_PKT_BATCH]; + int nr_packets; + /* RX: napi to kick after GPU writes verdicts (single-queue fallback + * for KAT; production multi-queue dispatch schedules per-queue napis + * at dispatcher level and leaves rx_napi =3D=3D NULL). + */ + struct napi_struct *rx_napi; + int rx_queue_idx; + struct spsc_bd *rx_bds[KNOD_IPSEC_PKT_BATCH]; + /* Per-packet queue index for multi-queue batched RX dispatches. + * finish_rx_deliver uses this to route each decrypted packet to + * the correct priv->rxq[] delivery pool + desc_ring. + */ + u8 rx_pkt_queue[KNOD_IPSEC_PKT_BATCH]; + /* Heap-backed finish-state scratchpad used by finish_rx_deliver. + * Sized to KNOD_IPSEC_PKT_BATCH so we never overflow the dispatcher + * kernel stack when BATCH grows. + */ + struct knod_ipsec_rx_pending rx_pending[KNOD_IPSEC_PKT_BATCH]; +}; + +/* + * Hard upper bound on parallel dispatchers. Each dispatcher owns one + * kaql[i] / sdma[i] pair, one private slice of work_pool and the + * backing pool BOs, and one contiguous range of RX queues. The actual + * count is knod_ipsec_priv::nr_dispatchers, set at + * start time from knod->queue_cnt. + * + * Bumped beyond 1 for real parallelism: one GPU AQL queue is a + * hardware FIFO, so throughput is single-kaql drain rate bound. Two + * kaqls let the GPU scheduler run two dispatches on disjoint CUs in + * parallel, subject to CU / memory bandwidth contention. + */ +#define KNOD_IPSEC_MAX_DISPATCHERS 4 + +/* + * Per-dispatcher runtime state. Each dispatcher kthread owns exactly + * one of these and never shares hot-path state with any other + * dispatcher - cursors, in-flight work slots, pool BOs, fence counters + * and the kaql/sdma index are all private. + * + * Cross-dispatcher sharing lives in knod_ipsec_priv: the SA table + + * slot array + spi_to_slot xarray (read-mostly), the per-CPU stats, + * and the delivery pool / desc_ring for RX delivery (but each rxq slot is + * only drained by the owning dispatcher, so no locking needed inside + * a queue). + */ +struct knod_ipsec_dispatcher { + struct knod_ipsec_priv *priv; + struct task_struct *kthread; + + /* kaql[kaql_idx] + sdma[kaql_idx] owned exclusively by this disp. */ + int kaql_idx; + + /* Slice into priv->work_pool[]. Dispatcher N uses slots + * [work_first, work_first+work_count). work_count is usually + * KNOD_IPSEC_NR_WORK but can be smaller if the final dispatcher + * got a partial slice. + */ + int work_first; + int work_count; + + /* Private backing BOs for this dispatcher's work slice. Each + * slot within [work_first, work_first+work_count) gets its own + * sub-range of these BOs, so no cross-dispatcher aliasing. + */ + struct knod_mem *param_pool; + struct knod_mem *decrypt_pool; + + /* RX queue range this dispatcher drains. [rxq_first, rxq_first+ + * rxq_count) indexes into priv->rxq[] and knodev->wpriv[].spsc_bds. + */ + int rxq_first; + int rxq_count; + + /* Hot-path cursors, previously locals in the dispatcher loop. */ + int rx_rr; + int build_cursor; + + /* Forward-looking AQL completion-signal counter. See old priv-> + * dispatch_sigval_next comment; now per-dispatcher because each + * disp has its own kaql signal. + */ + s64 dispatch_sigval_next; +}; + +struct knod_ipsec_priv { + struct knod *knod; + struct knod_dev *knodev; + + /* Shared GCM tables (VRAM, one per priv, owned by knod ctx) */ + struct knod_mem *t_tables; + + /* GPU-visible SA table */ + struct knod_mem *sa_table; + + /* CPU-side slot metadata */ + struct knod_ipsec_sa_slot slots[KNOD_IPSEC_NR_SA]; + struct xarray spi_to_slot; + struct mutex slot_lock; + + /* + * Work pool is sized NR_WORK * MAX_DISPATCHERS so each + * dispatcher gets its own NR_WORK-sized slice via work_first. + * With nr_dispatchers=3D1 only the first slice is populated, so + * runtime cost is identical to the old single-slot layout when + * a single dispatcher is in use. + */ + struct knod_ipsec_work work_pool[KNOD_IPSEC_NR_WORK * + KNOD_IPSEC_MAX_DISPATCHERS]; + + /* Number of NIC RX queues bound to the NOD (<=3D KNOD_SPSC_MAX); the + * finish worker bounds-checks the shader's queue index against it. + */ + int nr_rxq; + + /* Dispatcher state. nr_dispatchers <=3D KNOD_IPSEC_MAX_DISPATCHERS, + * set at start time from the knod queue_cnt the accel was + * created with. Each disp[i] owns kaql[i] / sdma[i]. + */ + struct knod_ipsec_dispatcher disp[KNOD_IPSEC_MAX_DISPATCHERS]; + int nr_dispatchers; + bool running; + u32 pkt_batch; + + /* Shaders */ + size_t shader_size; + int isa_version; + + /* Observability */ + struct dentry *debug_dir; + struct knod_ipsec_stats __percpu *stats; + + /* Persistent KAT scratch BO. Allocated once at priv init, reused on + * every KAT invocation. Re-allocating per-KAT re-triggers the GTT/VRAM + * multi-BO mapping symptom (only the first BO is reliably mapped on + * the KFD process VM), which is the same class of bug documented for + * the BPF delivery path. One long-lived BO sidesteps it entirely. + */ + struct knod_mem *kat_scratch; +}; + +/* Public entry points for NIC consumers. */ +struct spsc_bd; +int knod_ipsec_rx_submit(struct knod_ipsec_fused_sub *sub, int nr, + struct napi_struct *napi, int queue_idx); +int knod_ipsec_rx_submit_bds(struct knod_ipsec_fused_sub *sub, + struct spsc_bd **bds, int nr, + struct napi_struct *napi, int queue_idx); + +#endif /* KNOD_IPSEC_H_ */ diff --git a/tools/testing/selftests/drivers/net/knod/config b/tools/testin= g/selftests/drivers/net/knod/config index a60ece9da112..837187205f89 100644 --- a/tools/testing/selftests/drivers/net/knod/config +++ b/tools/testing/selftests/drivers/net/knod/config @@ -4,4 +4,5 @@ CONFIG_XDP_SOCKETS=3Dy CONFIG_KNOD=3Dm CONFIG_HSA_AMD=3Dy CONFIG_HSA_AMD_KNOD_BPF=3Dm +CONFIG_HSA_AMD_KNOD_IPSEC=3Dm CONFIG_DEBUG_FS=3Dy --=20 2.43.0