From nobody Sun May 24 19:33:57 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D60A338838E for ; Sat, 23 May 2026 12:15:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538546; cv=none; b=i7r95aE9jT8bPlrsCqTuFSRZzGrczJFKAdaEFC+cgU0LlGk/VNfJVRRlGdtReca73D7MCIDFCI9dR4GCZ5V7Ok5ckqHLOTz0jAA/mWI+/1PvTkC5wRWaY+Nus/hg3orA3ORgUf/2d3xLSJkW55h056nfKk3mrw+6ZcLQ7ErAf/8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538546; c=relaxed/simple; bh=8l4tyC4y5/Ak0oYZbkuWnFYCquWoCgn8Deh8zHDa4TM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n0VdrLEiNPHvs9YTbk30sv5i003/7xDONZarCaTqFOpWHYGf8EcwNdmG77XRX6K2293nrPi8RdNdIVi3Y8hTNy7yLTxwMKWLpIfodOxQedgXylnyUyFy9OSLxX/LCHHWJ1LuxFa/T3BDMOByx6wHxmAigrdHxDr8So0hqxy/Yk8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NqFuQuuy; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NqFuQuuy" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2b9fcf7c91bso92549065ad.0 for ; Sat, 23 May 2026 05:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779538544; x=1780143344; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ptBan/oDFwIr7VXZlN8TdNfSiKDV9ouul3U9bPm2km8=; b=NqFuQuuyrFcUeHomeDO7Hu7KciKYD0AtdvZe5Fq1EApZxDErm6dOjtTJpdQaALo+qn INy/62H2MX0W9Vz2YkYQVAmNjYSZBT3JtCEDGnT1UWoKuwzPoNl3LPjDJ0LQQXwpN0z0 m8CXesgDEjaCGeQIjPoOHNedW/w92IAsMHVfjcoSVdwRPb6te8P16e3FxKqtz/crX8zR w2mxQXLCCSmpgwqM8Ub3vUNz0rgYSqzmPj7i+kR79vn6LmNjqIm+RT5MeKW5Uj7oFDeU F0MRQSZebF4NqTm2IVL8F7Ii6a9VVE71qneF+H2nq1e+S/JbmuCkWsqZRhy3+TS0RTUq Wvdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779538544; x=1780143344; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ptBan/oDFwIr7VXZlN8TdNfSiKDV9ouul3U9bPm2km8=; b=lQMCIRi69rMHtdFZkXu0PSA73HooNEKgEB9P+6qDNEJWuE6YyroCBlbHgb+/Ny+iDi wAeN+c8TaM/WA24zLMRx3Vfb5lr9aM6pTif3mjo4ESfB6ENrc1HDUjpma83mYiXfthMA iuMSGmQRrlzRGfovF6MOFIzqM7jy0B+aoSn3U+nVGk38cwa2KPhrwRnvMS8F5wbsVQBM 6DFLHoCpyh0ouQ3MZ0iLh4q4ara68T+8/aq23RbQBELkKAo+afsXY3JCdasNXogYLBdk lgQ39HIX73bEYO/B1U/PqQJWdzjUGKbmOuttxh6Oq0+ja6d+Oek/fZxljZ//xRlA+tji k2yw== X-Gm-Message-State: AOJu0YxYNDX489gOoxDIvdcFIRejgmqDHzmaaeM1CFhfNx9BUghgbjge 97fL7fq1wXpmFhktpTPSsy6GcogDxLEiX5ekPXInUXPJG829Lj9P5Fjl X-Gm-Gg: Acq92OFzGRP8kUMNIEpZV4zv3+sFfFoPJxTmHNNunhI1xkw+F5n7DCVn6gO5i7dOMvx ar16Vz0IyEJWuB9LA+CIbkI2QvHNeycZi7XAfrJjlZB1qZF3I0vQHLizpiFb6AXlBVo3kPALYYX iU3H7LivmRQtSf5Ldl6gAdHKrz8IILgaXAY3SPAuoAqw/A8uUQiTJkjbEy14BBNnJl9hdX5J6Qb +JZ+dTEPYw9T9nHWrb7gn8GajaMqY+CqNHC4uOwYDV6N2uEJuqJrmrL9e+RFBI41g/S2T8L6vOt 6vKZ+LvEXlFVYYq+gYXk8SfWi4Bl50z0nu/YCE2qD/SmNJkCLpQlDLsBhz6pK12+AYcMOlDrnlU zWR4mSJRSE8YUlNwfHvXwY8ohvPm+z9GeDqujeLxwPzUFw8MbZmDLQ3CiE3vEhEFg2WCQpjb4lR hq6MLlkb2Cg5mJpuiT2npQyfsgt0U83wWYPv4= X-Received: by 2002:a17:903:3d10:b0:2bc:6784:5260 with SMTP id d9443c01a7336-2beb06eb0e4mr90794065ad.37.1779538543856; Sat, 23 May 2026 05:15:43 -0700 (PDT) Received: from mincom1 ([125.149.177.227]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2beb56ce4easm41746305ad.30.2026.05.23.05.15.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 May 2026 05:15:43 -0700 (PDT) From: Jihong Min To: Christian Marangi , Antoine Tenart , Herbert Xu , "David S . Miller" , Lorenzo Bianconi , Andrew Lunn , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, netdev@vger.kernel.org, Jihong Min Subject: [PATCH 1/3] xfrm: extend ESP offload infrastructure for packet engines Date: Sat, 23 May 2026 21:15:20 +0900 Message-ID: <20260523121522.3023992-2-hurryman2212@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com> References: <20260523121522.3023992-1-hurryman2212@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some ESP offload engines operate on whole ESP packets rather than the generic software trailer layout. They can generate outbound ESP padding, next-header and ICV bytes in hardware, and inbound decapsulation can return an already-trimmed packet with the recovered next-header value. Add a netdev offload callback for drivers to opt into hardware-generated ESP TX trailers, carry the reserved ESP TX tail length in xfrm_offload, and let ESP input skip software trailer removal when hardware has already done it. This keeps the default ESP offload behavior unchanged for existing devices while providing the infrastructure needed by packet-mode ESP engines. Assisted-by: Codex:gpt-5.5 Signed-off-by: Jihong Min --- include/linux/netdevice.h | 3 +++ include/net/xfrm.h | 8 +++++++- net/ipv4/esp4.c | 6 +++++- net/ipv4/esp4_offload.c | 29 ++++++++++++++++++++++++++++- net/ipv6/esp6.c | 6 +++++- net/ipv6/esp6_offload.c | 29 ++++++++++++++++++++++++++++- 6 files changed, 76 insertions(+), 5 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0e1e581efc5a..b6ff04c3df78 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1043,6 +1043,9 @@ struct xfrmdev_ops { struct xfrm_state *x); bool (*xdo_dev_offload_ok) (struct sk_buff *skb, struct xfrm_state *x); + /* Return true when the device generates the ESP trailer/ICV itself. */ + bool (*xdo_dev_esp_tx_hw_trailer)(struct sk_buff *skb, + struct xfrm_state *x); void (*xdo_dev_state_advance_esn) (struct xfrm_state *x); void (*xdo_dev_state_update_stats) (struct xfrm_state *x); int (*xdo_dev_policy_add) (struct xfrm_policy *x, struct netlink_ext_ack = *extack); diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 10d3edde6b2f..160069901e0a 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -1141,7 +1141,7 @@ struct xfrm_offload { #define CRYPTO_FALLBACK 8 #define XFRM_GSO_SEGMENT 16 #define XFRM_GRO 32 -/* 64 is free */ +#define XFRM_ESP_NO_TRAILER 64 #define XFRM_DEV_RESUME 128 #define XFRM_XMIT 256 =20 @@ -1158,6 +1158,12 @@ struct xfrm_offload { /* Used to keep whole l2 header for transport mode GRO */ __u16 orig_mac_len; =20 + /* + * ESP packet engines can reserve tailroom in the generic ESP path and + * generate padding, next-header and ICV bytes during device TX. + */ + __u16 esp_tx_tailen; + __u8 proto; __u8 inner_ipproto; }; diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c index 6a5febbdbee4..f21c8f2e60f7 100644 --- a/net/ipv4/esp4.c +++ b/net/ipv4/esp4.c @@ -720,7 +720,11 @@ int esp_input_done2(struct sk_buff *skb, int err) if (unlikely(err)) goto out; =20 - err =3D esp_remove_trailer(skb); + /* Hardware ESP decapsulation can already remove pad/trailer/ICV. */ + if (xo && (xo->flags & XFRM_ESP_NO_TRAILER)) + err =3D xo->proto; + else + err =3D esp_remove_trailer(skb); if (unlikely(err < 0)) goto out; =20 diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c index abd77162f5e7..f00fff98b69f 100644 --- a/net/ipv4/esp4_offload.c +++ b/net/ipv4/esp4_offload.c @@ -270,8 +270,10 @@ static int esp_xmit(struct xfrm_state *x, struct sk_bu= ff *skb, netdev_features_ struct xfrm_offload *xo; struct ip_esp_hdr *esph; struct crypto_aead *aead; + struct sk_buff *trailer; struct esp_info esp; bool hw_offload =3D true; + bool hw_trailer =3D false; __u32 seq; int encap_type =3D 0; =20 @@ -281,6 +283,7 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buf= f *skb, netdev_features_ =20 if (!xo) return -EINVAL; + xo->esp_tx_tailen =3D 0; =20 if ((!(features & NETIF_F_HW_ESP) && !(skb->dev->gso_partial_features & NETIF_F_HW_ESP)) || @@ -303,13 +306,37 @@ static int esp_xmit(struct xfrm_state *x, struct sk_b= uff *skb, netdev_features_ esp.clen =3D ALIGN(skb->len + 2 + esp.tfclen, blksize); esp.plen =3D esp.clen - skb->len - esp.tfclen; esp.tailen =3D esp.tfclen + esp.plen + alen; + if (esp.tailen > U16_MAX) + return -EINVAL; =20 esp.esph =3D ip_esp_hdr(skb); =20 if (x->encap) encap_type =3D x->encap->encap_type; =20 - if (!hw_offload || !skb_is_gso(skb) || (hw_offload && encap_type =3D=3D U= DP_ENCAP_ESPINUDP)) { + if (hw_offload && !skb_is_gso(skb) && !encap_type && x->xso.dev && + x->xso.dev->xfrmdev_ops && + x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer) + hw_trailer =3D + x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer(skb, x); + + if (hw_trailer) { + int esph_offset; + + /* + * The device packet engine will write ESP padding, next-header + * and ICV bytes. Keep skb->len unchanged here, but make sure the + * later DMA writer owns enough linear tailroom. + */ + esph_offset =3D (unsigned char *)esp.esph - skb_transport_header(skb); + esp.nfrags =3D skb_cow_data(skb, esp.tailen, &trailer); + if (esp.nfrags < 0) + return esp.nfrags; + esp.esph =3D (struct ip_esp_hdr *)(skb_transport_header(skb) + + esph_offset); + xo->esp_tx_tailen =3D esp.tailen; + } else if (!hw_offload || !skb_is_gso(skb) || + (hw_offload && encap_type =3D=3D UDP_ENCAP_ESPINUDP)) { esp.nfrags =3D esp_output_head(x, skb, &esp); if (esp.nfrags < 0) return esp.nfrags; diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c index 9c06c5a1419d..730588f8eaba 100644 --- a/net/ipv6/esp6.c +++ b/net/ipv6/esp6.c @@ -751,7 +751,11 @@ int esp6_input_done2(struct sk_buff *skb, int err) if (unlikely(err)) goto out; =20 - err =3D esp_remove_trailer(skb); + /* Hardware ESP decapsulation can already remove pad/trailer/ICV. */ + if (xo && (xo->flags & XFRM_ESP_NO_TRAILER)) + err =3D xo->proto; + else + err =3D esp_remove_trailer(skb); if (unlikely(err < 0)) goto out; =20 diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c index 22895521a57d..d124493da40b 100644 --- a/net/ipv6/esp6_offload.c +++ b/net/ipv6/esp6_offload.c @@ -308,8 +308,10 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_b= uff *skb, netdev_features int blksize; struct xfrm_offload *xo; struct crypto_aead *aead; + struct sk_buff *trailer; struct esp_info esp; bool hw_offload =3D true; + bool hw_trailer =3D false; __u32 seq; =20 esp.inplace =3D true; @@ -318,6 +320,7 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_bu= ff *skb, netdev_features =20 if (!xo) return -EINVAL; + xo->esp_tx_tailen =3D 0; =20 if (!(features & NETIF_F_HW_ESP) || x->xso.dev !=3D skb->dev) { xo->flags |=3D CRYPTO_FALLBACK; @@ -338,8 +341,32 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_b= uff *skb, netdev_features esp.clen =3D ALIGN(skb->len + 2 + esp.tfclen, blksize); esp.plen =3D esp.clen - skb->len - esp.tfclen; esp.tailen =3D esp.tfclen + esp.plen + alen; + if (esp.tailen > U16_MAX) + return -EINVAL; =20 - if (!hw_offload || !skb_is_gso(skb)) { + if (hw_offload && !skb_is_gso(skb) && !x->encap && x->xso.dev && + x->xso.dev->xfrmdev_ops && + x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer) + hw_trailer =3D + x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer(skb, x); + + if (hw_trailer) { + int esph_offset; + + /* + * The device packet engine will write ESP padding, next-header + * and ICV bytes. Keep skb->len unchanged here, but make sure the + * later DMA writer owns enough linear tailroom. + */ + esp.esph =3D ip_esp_hdr(skb); + esph_offset =3D (unsigned char *)esp.esph - skb_transport_header(skb); + esp.nfrags =3D skb_cow_data(skb, esp.tailen, &trailer); + if (esp.nfrags < 0) + return esp.nfrags; + esp.esph =3D (struct ip_esp_hdr *)(skb_transport_header(skb) + + esph_offset); + xo->esp_tx_tailen =3D esp.tailen; + } else if (!hw_offload || !skb_is_gso(skb)) { esp.nfrags =3D esp6_output_head(x, skb, &esp); if (esp.nfrags < 0) return esp.nfrags; --=20 2.53.0 From nobody Sun May 24 19:33:57 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65AE63955DE for ; Sat, 23 May 2026 12:15:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538552; cv=none; b=YONs+/Rv3YZE1/+1lKzCERLEcNMsbdKN0S0Pzcw05CLgcFbSjxUa29YeTpJjrZ4PChgCjO4df4TL8oSz6H3wjESs/x9Nc9XkREYg7vHm7CTjClgdwcVASjpwvHfpPIymqcVCWuUwUVszmQxatOsTlf9iBiNCxGWdH7KlmIiGsB4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538552; c=relaxed/simple; bh=KHyLsuehdlpg0hwY3KJNtON2x37x6MuLItwo1XMPiH0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MvNhTeii2chuWnGtxtz8AwAyqIh5YCZhMdmYarYbeuwEn17YvRyR+4CbiPP3khtxxNU8BcqptSalMCmjSP8nrjCewUgymx2DWgwcqkCxNjEZ4xPtjcUjy3mSB4UfpkLtbiT2zafYcAFgOWujUBCLmlVLSdu+BqOll7nOVr8lqOU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dCp8bATS; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dCp8bATS" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2ba1e9d3687so54630585ad.3 for ; Sat, 23 May 2026 05:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779538547; x=1780143347; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b2zrtx57P1TJdR+y3qfIhSkztYF/cKJ64UoY0xbwGSU=; b=dCp8bATSAZSXp5JSjCus752mIR6VPVYhGh8YzrVQhbMbCuhvZAD8Gvt+OgZ986S+Vi PGVhOfHa+1i36SLpcvthhjwAS4GxjVBM81N8BEZ90wE0XNFKygkNTXQhu6P5HPEeEjHT NjzQTg+wR8KMw9lkwiVzzCcm6c2HAbTUG2FhddmXyIjpnZpipbj6xQXTG1bZzXbhKeSr Hs7cZ4rUZFNAax+ok5hjRmLcLD+6tuoioiKKo1Da+fqKW91edfsjGZ75fnDprTtaTdwa /ttZQ8hLPmu6TD600JHOM9En07ryBHp4kNUBWlUHDhArpetdKT9UxhLqIJzigTzpiz1v /02w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779538547; x=1780143347; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=b2zrtx57P1TJdR+y3qfIhSkztYF/cKJ64UoY0xbwGSU=; b=JmXTUdz0N9aG7LIBqIy8IJcgPePPSAII4gIhYf6qZw0DLyYzurx2EFLFDWzF/RfJWf Fi+Jalra9mm16DeVXaBKsEv4e/IMrASZw7OwPVfGxuioQr294UcVbsKjuRgQQrIG/s77 Tm0ZY3IYbUumEmZ7aoXe7MJ0De3Wtl/oM/j9oVRZEmg2EL21E9LOZYtBvuGA2dcLKMFE q0780TZ522jGUX4x5/P+GJ21oKldPJX6Tu2NIUxvIdcJkWk7cA4MBt3oIpqSjDWigN0A P0S54S8u4ut6JGRti9i0xNtDa7RgYGqgjJzSWntcZ4/6BzIP2gOZQjve6wkDvwHyw70X 9d4g== X-Gm-Message-State: AOJu0Yxywu+P9yXEVEh6YOZjsZSF5+eiJZ4/1PkZ1FKlmhOv2uwvVsFj QA/TgfGu/Vniqag1iUii6bCbCLHkGTaRri1bqSDGfeYmNjjoT6aQUc/X X-Gm-Gg: Acq92OEXBbaVEqEcbhNiJ8evv5QrOFPHvHoTY+XBJPaN+3XjCbSoXG/hxcF4myaSds8 LNNSMB3GkvJx6lWWUXAzhaU6tBssaBdtGEhORpKugVGee7WwRQNxhN+jfNQ4o/y+iTY3NskdxzK mLX6qvpVShnRtwRlfvKZglFgnIsztrLdR2dAmkB7eKfsCq7CUVuBNRYF00AjAa8+V1ZYjSi0lwq YmhQ8JDHbLL22G+BnSYgkr0ZfhPqNGB6UL+LynRLv63vvRFcEsqyJi3NyNQPzJHSa7IUJMhLztJ OZe295+tH3RBxpX+GyGxFC60ap/FpsMlqXv7p48IPxFd3gbbzTvjhv9XZDuU7Cqt+XQU+S5wepU AFK30TTg4UJpcg1H+r1ESq7hS9qkwG1+mQG4NgzngDhbzhFSJGvHi739lJUbxGWVopVuPVGEsdT 6ozjU3XWLIQIU9lu7DsW4b9CfP X-Received: by 2002:a17:902:f650:b0:2bd:8db9:cc0c with SMTP id d9443c01a7336-2beb038e50cmr78918005ad.6.1779538547270; Sat, 23 May 2026 05:15:47 -0700 (PDT) Received: from mincom1 ([125.149.177.227]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2beb56ce4easm41746305ad.30.2026.05.23.05.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 May 2026 05:15:46 -0700 (PDT) From: Jihong Min To: Christian Marangi , Antoine Tenart , Herbert Xu , "David S . Miller" , Lorenzo Bianconi , Andrew Lunn , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, netdev@vger.kernel.org, Jihong Min Subject: [PATCH 2/3] crypto: inside-secure: add EIP93 ESP packet backend Date: Sat, 23 May 2026 21:15:21 +0900 Message-ID: <20260523121522.3023992-3-hurryman2212@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com> References: <20260523121522.3023992-1-hurryman2212@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Expose an EIP93 packet-mode IPsec backend for netdev drivers that need ESP encapsulation and decapsulation offload without advertising EIP93 itself as a netdev. Add provider selection, capability reporting, SA lifecycle management, IPsec request completion, and provider fault notification around the existing EIP93 descriptor path. Assisted-by: Codex:gpt-5.5 Signed-off-by: Jihong Min --- MAINTAINERS | 1 + drivers/crypto/inside-secure/eip93/Kconfig | 10 + drivers/crypto/inside-secure/eip93/Makefile | 1 + .../crypto/inside-secure/eip93/eip93-ipsec.c | 1413 +++++++++++++++++ .../crypto/inside-secure/eip93/eip93-main.c | 69 +- .../crypto/inside-secure/eip93/eip93-main.h | 38 +- include/crypto/eip93-ipsec.h | 132 ++ 7 files changed, 1643 insertions(+), 21 deletions(-) create mode 100644 drivers/crypto/inside-secure/eip93/eip93-ipsec.c create mode 100644 include/crypto/eip93-ipsec.h diff --git a/MAINTAINERS b/MAINTAINERS index f1e5e4258e7b..08cfede333e8 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12743,6 +12743,7 @@ L: linux-crypto@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/crypto/inside-secure,safexcel-eip93.y= aml F: drivers/crypto/inside-secure/eip93/ +F: include/crypto/eip93-ipsec.h =20 INTEGRITY MEASUREMENT ARCHITECTURE (IMA) M: Mimi Zohar diff --git a/drivers/crypto/inside-secure/eip93/Kconfig b/drivers/crypto/in= side-secure/eip93/Kconfig index 29523f6927dd..1a33ab6f04da 100644 --- a/drivers/crypto/inside-secure/eip93/Kconfig +++ b/drivers/crypto/inside-secure/eip93/Kconfig @@ -18,3 +18,13 @@ config CRYPTO_DEV_EIP93 CTR crypto. Also provide DES and 3DES ECB and CBC. =20 Also provide AEAD authenc(hmac(x), cipher(y)) for supported algo. + +config CRYPTO_DEV_EIP93_IPSEC + bool + depends on CRYPTO_DEV_EIP93 + depends on XFRM_OFFLOAD + default y + help + Select this if a netdev driver should be allowed to use EIP93 for + ESP packet encapsulation and decapsulation rather than only the + crypto transform. diff --git a/drivers/crypto/inside-secure/eip93/Makefile b/drivers/crypto/i= nside-secure/eip93/Makefile index a3d3d3677cdc..a5bb98370ff0 100644 --- a/drivers/crypto/inside-secure/eip93/Makefile +++ b/drivers/crypto/inside-secure/eip93/Makefile @@ -3,3 +3,4 @@ obj-$(CONFIG_CRYPTO_DEV_EIP93) +=3D crypto-hw-eip93.o crypto-hw-eip93-y +=3D eip93-main.o eip93-common.o crypto-hw-eip93-y +=3D eip93-cipher.o eip93-aead.o crypto-hw-eip93-y +=3D eip93-hash.o +crypto-hw-eip93-$(CONFIG_CRYPTO_DEV_EIP93_IPSEC) +=3D eip93-ipsec.o diff --git a/drivers/crypto/inside-secure/eip93/eip93-ipsec.c b/drivers/cry= pto/inside-secure/eip93/eip93-ipsec.c new file mode 100644 index 000000000000..7338f4c7e24a --- /dev/null +++ b/drivers/crypto/inside-secure/eip93/eip93-ipsec.c @@ -0,0 +1,1413 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2026 + * + * Jihong Min + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "eip93-main.h" +#include "eip93-regs.h" +#include "eip93-common.h" + +#define EIP93_IPSEC_PAD_ALIGN 2 +#define EIP93_IPSEC_IDR_MIN 0 +#define EIP93_IPSEC_IDR_MAX (EIP93_RING_NUM - 1) +#define EIP93_IPSEC_DIGEST_WORD_BITS (BITS_PER_BYTE * sizeof(u32)) +#define EIP93_IPSEC_DIGEST_WORDS(bits) ((bits) / EIP93_IPSEC_DIGEST_WORD_B= ITS) +#define EIP93_IPSEC_HMAC_STATE_SIZE SHA256_DIGEST_SIZE +#define EIP93_IPSEC_PRNG_BUF_SIZE 4080 +#define EIP93_IPSEC_PRNG_RESET_MODE 1 +#define EIP93_IPSEC_PRNG_POLL_US 10000 +#define EIP93_IPSEC_PRNG_POLL_STEP_US 10 + +struct eip93_ipsec { + struct eip93_device *eip93; + struct list_head node; + struct list_head sa_list; + struct work_struct fault_work; + spinlock_t lock; /* protects dead/refcount admission */ + refcount_t refcnt; + struct completion done; + enum eip93_ipsec_event fault_event; + u32 algo_flags; + bool dead; +}; + +struct eip93_ipsec_sa { + struct eip93_ipsec *ipsec; + struct sa_record *sa_record; + dma_addr_t sa_record_base; + struct list_head node; + struct list_head requests; + spinlock_t lock; /* protects dead/refcount admission */ + refcount_t refcnt; + struct completion done; + u32 flags; + u16 family; + u8 authsize; + u8 blocksize; + u8 ivsize; + u8 encap_type; + bool esn; + bool dead; + bool aborting; +}; + +struct eip93_ipsec_request { + struct eip93_ipsec_sa *sa; + struct sk_buff *skb; + struct list_head node; + refcount_t refcnt; + eip93_ipsec_complete_t complete; + void *data; + dma_addr_t dma; + unsigned int dma_len; + enum dma_data_direction dma_dir; + int idr; +}; + +static DEFINE_MUTEX(eip93_ipsec_devices_lock); +static LIST_HEAD(eip93_ipsec_devices); +static BLOCKING_NOTIFIER_HEAD(eip93_ipsec_notifier); + +static bool eip93_ipsec_get_ref(struct eip93_ipsec *ipsec) +{ + bool ret =3D false; + + spin_lock_bh(&ipsec->lock); + if (!ipsec->dead) + ret =3D refcount_inc_not_zero(&ipsec->refcnt); + spin_unlock_bh(&ipsec->lock); + + return ret; +} + +void eip93_ipsec_put(struct eip93_ipsec *ipsec) +{ + if (ipsec && refcount_dec_and_test(&ipsec->refcnt)) + complete(&ipsec->done); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_put); + +static bool eip93_ipsec_same_subsystem(struct device *consumer, + struct eip93_ipsec *ipsec) +{ + struct device_node *consumer_parent; + struct device_node *eip93_parent; + struct device_node *consumer_np; + struct device_node *eip93_np; + bool match; + + consumer_np =3D dev_of_node(consumer); + eip93_np =3D dev_of_node(ipsec->eip93->dev); + if (!consumer_np || !eip93_np) + return false; + + consumer_parent =3D of_get_parent(consumer_np); + eip93_parent =3D of_get_parent(eip93_np); + match =3D consumer_parent && consumer_parent =3D=3D eip93_parent; + of_node_put(consumer_parent); + of_node_put(eip93_parent); + + return match; +} + +static bool eip93_ipsec_hw_available(u32 flags) +{ + if (!(flags & EIP93_PE_OPTION_AES)) + return false; + + if (!(flags & (EIP93_PE_OPTION_AES_KEY128 | EIP93_PE_OPTION_AES_KEY192 | + EIP93_PE_OPTION_AES_KEY256))) + return false; + + return flags & (EIP93_PE_OPTION_SHA_1 | EIP93_PE_OPTION_SHA_256); +} + +static bool eip93_ipsec_mark_dead(struct eip93_ipsec *ipsec) +{ + bool marked =3D false; + + spin_lock_bh(&ipsec->lock); + if (!ipsec->dead) { + ipsec->dead =3D true; + marked =3D true; + } + spin_unlock_bh(&ipsec->lock); + + return marked; +} + +static bool eip93_ipsec_mark_dead_async(struct eip93_ipsec *ipsec, + enum eip93_ipsec_event event) +{ + bool marked =3D false; + + spin_lock_bh(&ipsec->lock); + if (!ipsec->dead && refcount_inc_not_zero(&ipsec->refcnt)) { + ipsec->dead =3D true; + ipsec->fault_event =3D event; + marked =3D true; + } + spin_unlock_bh(&ipsec->lock); + + if (marked) + schedule_work(&ipsec->fault_work); + + return marked; +} + +static bool eip93_ipsec_live_hw_available(struct eip93_ipsec *ipsec) +{ + u32 flags =3D readl(ipsec->eip93->base + EIP93_REG_PE_OPTION_1); + + spin_lock_bh(&ipsec->lock); + ipsec->algo_flags =3D flags; + spin_unlock_bh(&ipsec->lock); + + return eip93_ipsec_hw_available(flags); +} + +struct eip93_ipsec *eip93_ipsec_get(struct device *consumer) +{ + struct eip93_ipsec *ipsec; + int err =3D -ENODEV; + + if (!consumer) + return ERR_PTR(-EINVAL); + + mutex_lock(&eip93_ipsec_devices_lock); + list_for_each_entry(ipsec, &eip93_ipsec_devices, node) { + if (!eip93_ipsec_same_subsystem(consumer, ipsec)) + continue; + + if (!eip93_ipsec_live_hw_available(ipsec)) { + enum eip93_ipsec_event event; + + event =3D EIP93_IPSEC_EVENT_CAPABILITY_LOSS; + eip93_ipsec_mark_dead_async(ipsec, event); + err =3D -EOPNOTSUPP; + continue; + } + + if (!eip93_ipsec_get_ref(ipsec)) { + err =3D -ENODEV; + continue; + } + + mutex_unlock(&eip93_ipsec_devices_lock); + return ipsec; + } + mutex_unlock(&eip93_ipsec_devices_lock); + + return ERR_PTR(err); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_get); + +bool eip93_ipsec_available(struct eip93_ipsec *ipsec) +{ + bool available; + + if (!ipsec) + return false; + + spin_lock_bh(&ipsec->lock); + available =3D !ipsec->dead; + spin_unlock_bh(&ipsec->lock); + if (!available) + return false; + + available =3D eip93_ipsec_live_hw_available(ipsec); + if (!available) + eip93_ipsec_mark_dead_async(ipsec, + EIP93_IPSEC_EVENT_CAPABILITY_LOSS); + + return available; +} +EXPORT_SYMBOL_GPL(eip93_ipsec_available); + +u32 eip93_ipsec_features(struct eip93_ipsec *ipsec) +{ + if (!eip93_ipsec_available(ipsec)) + return 0; + + return EIP93_IPSEC_FEATURE_ESP; +} +EXPORT_SYMBOL_GPL(eip93_ipsec_features); + +int eip93_ipsec_register_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&eip93_ipsec_notifier, nb); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_register_notifier); + +void eip93_ipsec_unregister_notifier(struct notifier_block *nb) +{ + blocking_notifier_chain_unregister(&eip93_ipsec_notifier, nb); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_unregister_notifier); + +static bool eip93_ipsec_sa_get(struct eip93_ipsec_sa *sa) +{ + bool ret =3D false; + + spin_lock_bh(&sa->ipsec->lock); + spin_lock(&sa->lock); + if (!sa->ipsec->dead && !sa->dead) + ret =3D refcount_inc_not_zero(&sa->refcnt); + spin_unlock(&sa->lock); + spin_unlock_bh(&sa->ipsec->lock); + + return ret; +} + +static void eip93_ipsec_sa_put(struct eip93_ipsec_sa *sa) +{ + if (refcount_dec_and_test(&sa->refcnt)) + complete(&sa->done); +} + +static bool eip93_ipsec_request_get(struct eip93_ipsec_request *req) +{ + return refcount_inc_not_zero(&req->refcnt); +} + +static void eip93_ipsec_request_put(struct eip93_ipsec_request *req) +{ + if (refcount_dec_and_test(&req->refcnt)) + kfree(req); +} + +static int eip93_ipsec_parse_flags(struct xfrm_state *x, u32 *flags) +{ + switch (x->props.ealgo) { + case SADB_X_EALG_AESCBC: + *flags |=3D EIP93_ALG_AES | EIP93_MODE_CBC; + break; + default: + return -EOPNOTSUPP; + } + + switch (x->props.aalgo) { + case SADB_AALG_SHA1HMAC: + *flags |=3D EIP93_HASH_HMAC | EIP93_HASH_SHA1; + break; + case SADB_X_AALG_SHA2_256HMAC: + *flags |=3D EIP93_HASH_HMAC | EIP93_HASH_SHA256; + break; + default: + return -EOPNOTSUPP; + } + + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_IN) + *flags |=3D EIP93_DECRYPT; + else + *flags |=3D EIP93_ENCRYPT; + + return 0; +} + +static unsigned int eip93_ipsec_auth_digest_size(struct xfrm_state *x) +{ + switch (x->props.aalgo) { + case SADB_AALG_SHA1HMAC: + return SHA1_DIGEST_SIZE; + case SADB_X_AALG_SHA2_256HMAC: + return SHA256_DIGEST_SIZE; + default: + return 0; + } +} + +static int eip93_ipsec_hmac_setkey(u32 flags, const u8 *key, + unsigned int keylen, u8 *dest_ipad, + u8 *dest_opad) +{ + u8 *ipad, *opad; + struct crypto_shash *tfm; + const char *alg_name; + unsigned int blocksize; + unsigned int digestsize; + unsigned int statesize; + unsigned int alloc_size; + unsigned int i; + int err; + + switch (flags & EIP93_HASH_MASK) { + case EIP93_HASH_SHA1: + alg_name =3D "sha1"; + digestsize =3D SHA1_DIGEST_SIZE; + break; + case EIP93_HASH_SHA256: + alg_name =3D "sha256"; + digestsize =3D SHA256_DIGEST_SIZE; + break; + default: + return -EINVAL; + } + + tfm =3D crypto_alloc_shash(alg_name, 0, CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(tfm)) + return PTR_ERR(tfm); + + blocksize =3D crypto_shash_blocksize(tfm); + statesize =3D crypto_shash_statesize(tfm); + if (statesize < EIP93_IPSEC_HMAC_STATE_SIZE) { + err =3D -EINVAL; + goto free_tfm; + } + + alloc_size =3D 2 * (blocksize + statesize); + ipad =3D kzalloc(alloc_size, GFP_KERNEL); + if (!ipad) { + err =3D -ENOMEM; + goto free_tfm; + } + opad =3D ipad + blocksize + statesize; + + if (keylen > blocksize) { + SHASH_DESC_ON_STACK(desc, tfm); + + desc->tfm =3D tfm; + err =3D crypto_shash_digest(desc, key, keylen, ipad); + shash_desc_zero(desc); + if (err) + goto free_pad; + + keylen =3D digestsize; + } else { + memcpy(ipad, key, keylen); + } + + memcpy(opad, ipad, blocksize); + for (i =3D 0; i < blocksize; i++) { + ipad[i] ^=3D HMAC_IPAD_VALUE; + opad[i] ^=3D HMAC_OPAD_VALUE; + } + + { + SHASH_DESC_ON_STACK(desc, tfm); + + desc->tfm =3D tfm; + err =3D crypto_shash_init(desc) ?: + crypto_shash_update(desc, ipad, blocksize) ?: + crypto_shash_export(desc, ipad) ?: + crypto_shash_init(desc) ?: + crypto_shash_update(desc, opad, blocksize) ?: + crypto_shash_export(desc, opad); + shash_desc_zero(desc); + } + if (err) + goto free_pad; + + /* + * EIP93 ESP protocol mode consumes the raw exported HMAC ipad/opad + * state. The crypto API AEAD helper byteswaps this state for its basic + * authenc path, but packet ESP mode matches mtk-eip93 with native + * exported bytes in the SA record. + */ + memcpy(dest_ipad, ipad, EIP93_IPSEC_HMAC_STATE_SIZE); + memcpy(dest_opad, opad, EIP93_IPSEC_HMAC_STATE_SIZE); + +free_pad: + kfree_sensitive(ipad); +free_tfm: + crypto_free_shash(tfm); + return err; +} + +static int eip93_ipsec_validate_algo(struct xfrm_state *x, + struct netlink_ext_ack *extack) +{ + unsigned int authsize; + unsigned int keylen; + + if (x->aead) { + NL_SET_ERR_MSG_MOD(extack, "AEAD SAs are unsupported"); + return -EOPNOTSUPP; + } + + if (!x->ealg || !x->aalg) { + NL_SET_ERR_MSG_MOD(extack, "encryption/auth required"); + return -EOPNOTSUPP; + } + + if (x->props.ealgo !=3D SADB_X_EALG_AESCBC) { + NL_SET_ERR_MSG_MOD(extack, "only AES-CBC is supported"); + return -EOPNOTSUPP; + } + + keylen =3D x->ealg->alg_key_len / BITS_PER_BYTE; + if (keylen !=3D AES_KEYSIZE_128 && keylen !=3D AES_KEYSIZE_192 && + keylen !=3D AES_KEYSIZE_256) { + NL_SET_ERR_MSG_MOD(extack, "unsupported AES-CBC key length"); + return -EOPNOTSUPP; + } + + authsize =3D eip93_ipsec_auth_digest_size(x); + if (!authsize) { + NL_SET_ERR_MSG_MOD(extack, "only SHA1/SHA256 HMAC"); + return -EOPNOTSUPP; + } + + if (x->aalg->alg_trunc_len % EIP93_IPSEC_DIGEST_WORD_BITS || + x->aalg->alg_trunc_len < EIP93_IPSEC_DIGEST_WORD_BITS || + x->aalg->alg_trunc_len > authsize * BITS_PER_BYTE) { + NL_SET_ERR_MSG_MOD(extack, "bad auth truncation length"); + return -EOPNOTSUPP; + } + + return 0; +} + +static int eip93_ipsec_validate_state(struct xfrm_state *x, + struct netlink_ext_ack *extack) +{ + switch (x->xso.dir) { + case XFRM_DEV_OFFLOAD_OUT: + case XFRM_DEV_OFFLOAD_IN: + break; + default: + NL_SET_ERR_MSG_MOD(extack, "only in/out SAs are supported"); + return -EOPNOTSUPP; + } + + if (x->xso.type !=3D XFRM_DEV_OFFLOAD_CRYPTO) { + NL_SET_ERR_MSG_MOD(extack, "only crypto offload is supported"); + return -EOPNOTSUPP; + } + + if (x->id.proto !=3D IPPROTO_ESP) { + NL_SET_ERR_MSG_MOD(extack, + "EIP93 packet backend supports ESP only"); + return -EOPNOTSUPP; + } + + switch (x->props.family) { + case AF_INET: + break; +#if IS_ENABLED(CONFIG_IPV6) + case AF_INET6: + break; +#endif + default: + NL_SET_ERR_MSG_MOD(extack, "only IPv4/IPv6 is supported"); + return -EOPNOTSUPP; + } + + if (x->outer_mode.family !=3D x->props.family) { + NL_SET_ERR_MSG_MOD(extack, "only same-family ESP is supported"); + return -EOPNOTSUPP; + } + + switch (x->props.mode) { + case XFRM_MODE_TUNNEL: + case XFRM_MODE_TRANSPORT: + break; + default: + NL_SET_ERR_MSG_MOD(extack, "only tunnel/transport"); + return -EOPNOTSUPP; + } + + if (x->outer_mode.encap !=3D x->props.mode) { + NL_SET_ERR_MSG_MOD(extack, + "outer ESP mode does not match state mode"); + return -EOPNOTSUPP; + } + + if (x->encap) { + NL_SET_ERR_MSG_MOD(extack, + "NAT-T is unsupported by EIP93 packet ESP"); + return -EOPNOTSUPP; + } + + if (x->tfcpad) { + NL_SET_ERR_MSG_MOD(extack, "TFC padding is unsupported"); + return -EOPNOTSUPP; + } + + return eip93_ipsec_validate_algo(x, extack); +} + +static int eip93_ipsec_validate_hw(struct xfrm_state *x, u32 flags, + struct netlink_ext_ack *extack) +{ + unsigned int keylen =3D x->ealg->alg_key_len / BITS_PER_BYTE; + u32 required; + + if (!(flags & EIP93_PE_OPTION_AES)) { + NL_SET_ERR_MSG_MOD(extack, "EIP93 AES engine is unavailable"); + return -EOPNOTSUPP; + } + + switch (keylen) { + case AES_KEYSIZE_128: + required =3D EIP93_PE_OPTION_AES_KEY128; + break; + case AES_KEYSIZE_192: + required =3D EIP93_PE_OPTION_AES_KEY192; + break; + case AES_KEYSIZE_256: + required =3D EIP93_PE_OPTION_AES_KEY256; + break; + default: + return -EOPNOTSUPP; + } + + if (!(flags & required)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported AES key length"); + return -EOPNOTSUPP; + } + + switch (x->props.aalgo) { + case SADB_AALG_SHA1HMAC: + required =3D EIP93_PE_OPTION_SHA_1; + break; + case SADB_X_AALG_SHA2_256HMAC: + required =3D EIP93_PE_OPTION_SHA_256; + break; + default: + return -EOPNOTSUPP; + } + + if (!(flags & required)) { + NL_SET_ERR_MSG_MOD(extack, + "EIP93 does not support this HMAC hash"); + return -EOPNOTSUPP; + } + + return 0; +} + +static void eip93_ipsec_init_sa_record(struct eip93_ipsec_sa *sa, + struct xfrm_state *x) +{ + struct sa_record *record =3D sa->sa_record; + unsigned int auth_words; + unsigned int enckeylen; + + enckeylen =3D x->ealg->alg_key_len / BITS_PER_BYTE; + auth_words =3D EIP93_IPSEC_DIGEST_WORDS(x->aalg->alg_trunc_len); + + eip93_set_sa_record(record, enckeylen, sa->flags); + + record->sa_cmd0_word &=3D + ~(EIP93_SA_CMD_OPGROUP | EIP93_SA_CMD_OPCODE | + EIP93_SA_CMD_DIGEST_LENGTH | EIP93_SA_CMD_PAD_TYPE | + EIP93_SA_CMD_IV_SOURCE | EIP93_SA_CMD_SAVE_IV); + record->sa_cmd0_word |=3D + EIP93_SA_CMD_OP_PROTOCOL | EIP93_SA_CMD_HDR_PROC | + EIP93_SA_CMD_PAD_IPSEC | EIP93_SA_CMD_SCPAD | + FIELD_PREP(EIP93_SA_CMD_OPCODE, + EIP93_SA_CMD_OPCODE_PROTOCOL_OUT_ESP) | + FIELD_PREP(EIP93_SA_CMD_DIGEST_LENGTH, auth_words); + + /* + * ESP packet mode authenticates from the ESP header when the hash + * crypt offset is zero. This is intentionally different from the AEAD + * authenc path, whose AAD starts after the ESP header. + */ + record->sa_cmd1_word &=3D + ~(EIP93_SA_CMD_HASH_CRYPT_OFFSET | EIP93_SA_CMD_BYTE_OFFSET | + EIP93_SA_CMD_COPY_PAD | EIP93_SA_CMD_COPY_HEADER | + EIP93_SA_CMD_COPY_DIGEST | EIP93_SA_CMD_COPY_PAYLOAD | + EIP93_SA_CMD_EN_SEQNUM_CHK); + record->sa_cmd1_word |=3D EIP93_SA_CMD_HMAC | EIP93_SA_CMD_EN_SEQNUM_CHK; + + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_IN) { + record->sa_cmd0_word |=3D EIP93_SA_CMD_DIRECTION_IN | + EIP93_SA_CMD_IV_FROM_INPUT; + /* + * Inbound ESP decapsulation keeps the outer header for XFRM and + * lets hardware remove ESP pad/trailer/ICV bytes. + */ + record->sa_cmd1_word |=3D EIP93_SA_CMD_COPY_HEADER; + } else { + record->sa_cmd0_word |=3D EIP93_SA_CMD_IV_FROM_PRNG; + record->sa_cmd1_word |=3D EIP93_SA_CMD_COPY_DIGEST; + } + + record->sa_spi =3D ntohl(x->id.spi); + if (sa->esn && x->replay_esn) { + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_IN) + record->sa_seqnum[1] =3D x->replay_esn->seq_hi; + else + record->sa_seqnum[1] =3D x->replay_esn->oseq_hi; + } else { + record->sa_seqnum[1] =3D 0; + } + record->sa_seqnum[0] =3D 0; + record->sa_seqmum_mask[0] =3D 0xffffffff; + record->sa_seqmum_mask[1] =3D sa->esn ? 0xffffffff : 0; +} + +static int eip93_ipsec_poll_result(struct eip93_device *eip93, + struct eip93_descriptor **rdesc) +{ + struct eip93_descriptor *desc; + unsigned int i; + u32 pe_ctrl_stat; + u32 pe_length; + + for (i =3D 0; i < EIP93_IPSEC_PRNG_POLL_US; + i +=3D EIP93_IPSEC_PRNG_POLL_STEP_US) { + if (readl(eip93->base + EIP93_REG_PE_RD_COUNT) & + EIP93_PE_RD_COUNT) + break; + udelay(EIP93_IPSEC_PRNG_POLL_STEP_US); + } + if (i >=3D EIP93_IPSEC_PRNG_POLL_US) + return -ETIMEDOUT; + + scoped_guard(spinlock_irqsave, &eip93->ring->read_lock) + desc =3D eip93_get_descriptor(eip93); + if (IS_ERR(desc)) + return PTR_ERR(desc); + *rdesc =3D desc; + + for (i =3D 0; i < EIP93_IPSEC_PRNG_POLL_US; + i +=3D EIP93_IPSEC_PRNG_POLL_STEP_US) { + pe_ctrl_stat =3D READ_ONCE((*rdesc)->pe_ctrl_stat_word); + pe_length =3D READ_ONCE((*rdesc)->pe_length_word); + if (FIELD_GET(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN, + pe_ctrl_stat) =3D=3D EIP93_PE_CTRL_PE_READY && + FIELD_GET(EIP93_PE_LENGTH_HOST_PE_READY, pe_length) =3D=3D + EIP93_PE_LENGTH_PE_READY) + break; + udelay(EIP93_IPSEC_PRNG_POLL_STEP_US); + } + + writel(1, eip93->base + EIP93_REG_PE_RD_COUNT); + writel(EIP93_INT_RDR_THRESH, eip93->base + EIP93_REG_INT_CLR); + + if (i >=3D EIP93_IPSEC_PRNG_POLL_US) + return -ETIMEDOUT; + + return 0; +} + +static int eip93_ipsec_init_prng(struct eip93_device *eip93) +{ + static const u32 prng_dt[4] =3D {}; + static const u32 prng_key[4] =3D { + 0xe0fc631d, 0xcbb9fb9a, 0x869285cb, 0xcbb9fb9a + }; + static const u32 prng_seed[4] =3D { + 0x758bac03, 0xf20ab39e, 0xa569f104, 0x95dfaea6 + }; + struct eip93_descriptor cdesc =3D {}; + struct eip93_descriptor *rdesc; + struct sa_record *record; + dma_addr_t record_dma; + dma_addr_t buf_dma; + void *buf; + int err; + + record =3D dma_alloc_coherent(eip93->dev, sizeof(*record), &record_dma, + GFP_KERNEL); + if (!record) + return -ENOMEM; + + buf =3D dma_alloc_coherent(eip93->dev, EIP93_IPSEC_PRNG_BUF_SIZE, + &buf_dma, GFP_KERNEL); + if (!buf) { + err =3D -ENOMEM; + goto free_record; + } + + memset(record, 0, sizeof(*record)); + record->sa_cmd0_word =3D + EIP93_SA_CMD_OP_BASIC | + FIELD_PREP(EIP93_SA_CMD_OPCODE, + EIP93_SA_CMD_OPCODE_BASIC_OUT_PRNG) | + EIP93_SA_CMD_CIPHER_AES | EIP93_SA_CMD_HASH_SHA1; + record->sa_cmd1_word =3D EIP93_SA_CMD_AES_KEY_128BIT; + memcpy(record->sa_key, prng_key, sizeof(prng_key)); + memcpy(record->sa_i_digest, prng_seed, sizeof(prng_seed)); + memcpy(record->sa_o_digest, prng_dt, sizeof(prng_dt)); + + cdesc.pe_ctrl_stat_word =3D + FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN, + EIP93_PE_CTRL_HOST_READY) | + FIELD_PREP(EIP93_PE_CTRL_PE_PRNG_MODE, + EIP93_IPSEC_PRNG_RESET_MODE); + cdesc.dst_addr =3D (u32 __force)buf_dma; + cdesc.sa_addr =3D record_dma; + cdesc.user_id =3D FIELD_PREP(EIP93_PE_USER_ID_DESC_FLAGS, + EIP93_DESC_PRNG | EIP93_DESC_LAST); + cdesc.pe_length_word =3D + FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY, + EIP93_PE_LENGTH_HOST_READY) | + FIELD_PREP(EIP93_PE_LENGTH_LENGTH, + EIP93_IPSEC_PRNG_BUF_SIZE); + + /* + * Outbound ESP SAs use IV_FROM_PRNG. Initialize the EIP93 PRNG before + * exposing the packet backend, otherwise the first ESP encrypt can + * fail or emit unusable IV material. + */ + scoped_guard(spinlock_irqsave, &eip93->ring->write_lock) + err =3D eip93_put_descriptor(eip93, &cdesc); + if (err) + goto free_buf; + + writel(1, eip93->base + EIP93_REG_PE_CD_COUNT); + err =3D eip93_ipsec_poll_result(eip93, &rdesc); + if (err) + goto free_buf; + + err =3D rdesc->pe_ctrl_stat_word & + (EIP93_PE_CTRL_PE_EXT_ERR_CODE | EIP93_PE_CTRL_PE_EXT_ERR | + EIP93_PE_CTRL_PE_SEQNUM_ERR | EIP93_PE_CTRL_PE_PAD_ERR | + EIP93_PE_CTRL_PE_AUTH_ERR); + err =3D eip93_parse_ctrl_stat_err(eip93, err); + if (err) + dev_err(eip93->dev, "IPsec PRNG init failed: %d\n", err); + +free_buf: + dma_free_coherent(eip93->dev, EIP93_IPSEC_PRNG_BUF_SIZE, buf, buf_dma); +free_record: + dma_free_coherent(eip93->dev, sizeof(*record), record, record_dma); + + return err; +} + +static int eip93_ipsec_submit(struct eip93_ipsec_request *req, + struct eip93_descriptor *cdesc) +{ + struct eip93_device *eip93 =3D req->sa->ipsec->eip93; + struct eip93_ipsec_sa *sa =3D req->sa; + struct eip93_ipsec *ipsec =3D sa->ipsec; + int err; + + spin_lock_bh(&ipsec->lock); + if (ipsec->dead) { + err =3D -EOPNOTSUPP; + goto unlock_ipsec; + } + + scoped_guard(spinlock_bh, &eip93->ring->idr_lock) req->idr =3D + idr_alloc(&eip93->ring->crypto_async_idr, req, + EIP93_IPSEC_IDR_MIN, EIP93_IPSEC_IDR_MAX, GFP_ATOMIC); + if (req->idr < 0) { + err =3D req->idr =3D=3D -ENOSPC ? -EBUSY : req->idr; + goto unlock_ipsec; + } + + spin_lock(&sa->lock); + if (sa->dead) { + spin_unlock(&sa->lock); + err =3D -EOPNOTSUPP; + goto remove_idr; + } + list_add_tail(&req->node, &sa->requests); + spin_unlock(&sa->lock); + + cdesc->user_id =3D + FIELD_PREP(EIP93_PE_USER_ID_CRYPTO_IDR, (u16)req->idr) | + FIELD_PREP(EIP93_PE_USER_ID_DESC_FLAGS, + EIP93_DESC_IPSEC | EIP93_DESC_LAST); + + scoped_guard(spinlock_irqsave, &eip93->ring->write_lock) + err =3D eip93_put_descriptor(eip93, cdesc); + if (err) + goto unlink_request; + + writel(1, eip93->base + EIP93_REG_PE_CD_COUNT); + spin_unlock_bh(&ipsec->lock); + + return -EINPROGRESS; + +unlink_request: + spin_lock(&sa->lock); + list_del_init(&req->node); + spin_unlock(&sa->lock); +remove_idr: + scoped_guard(spinlock_bh, &eip93->ring->idr_lock) + idr_remove(&eip93->ring->crypto_async_idr, req->idr); + err =3D err =3D=3D -ENOENT ? -EBUSY : err; +unlock_ipsec: + spin_unlock_bh(&ipsec->lock); + return err; +} + +static void eip93_ipsec_unlink_request(struct eip93_ipsec_request *req) +{ + struct eip93_ipsec_sa *sa =3D req->sa; + + spin_lock_bh(&sa->lock); + if (!list_empty(&req->node)) + list_del_init(&req->node); + spin_unlock_bh(&sa->lock); +} + +static void eip93_ipsec_complete_request(struct eip93_ipsec_request *req, + int err, + struct eip93_ipsec_result result) +{ + struct eip93_ipsec_sa *sa =3D req->sa; + eip93_ipsec_complete_t complete =3D req->complete; + void *data =3D req->data; + + dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len, + req->dma_dir); + eip93_ipsec_unlink_request(req); + eip93_ipsec_sa_put(sa); + complete(data, err, result); + eip93_ipsec_request_put(req); +} + +static void eip93_ipsec_abort_sa(struct eip93_ipsec_sa *sa, int err) +{ + struct eip93_device *eip93 =3D sa->ipsec->eip93; + struct eip93_ipsec_request *req; + bool claimed; + + while (true) { + spin_lock_bh(&sa->lock); + if (list_empty(&sa->requests)) { + spin_unlock_bh(&sa->lock); + return; + } + + req =3D list_first_entry(&sa->requests, + struct eip93_ipsec_request, node); + if (!eip93_ipsec_request_get(req)) { + list_del_init(&req->node); + spin_unlock_bh(&sa->lock); + continue; + } + list_del_init(&req->node); + spin_unlock_bh(&sa->lock); + + claimed =3D false; + scoped_guard(spinlock_bh, &eip93->ring->idr_lock) { + if (idr_find(&eip93->ring->crypto_async_idr, + req->idr) =3D=3D req) { + idr_remove(&eip93->ring->crypto_async_idr, + req->idr); + claimed =3D true; + } + } + + if (claimed) { + struct eip93_ipsec_result result =3D {}; + + eip93_ipsec_complete_request(req, err, result); + } + eip93_ipsec_request_put(req); + } +} + +static void eip93_ipsec_abort_requests(struct eip93_ipsec *ipsec, int err) +{ + struct eip93_ipsec_sa *sa; + + while (true) { + bool found =3D false; + + spin_lock_bh(&ipsec->lock); + list_for_each_entry(sa, &ipsec->sa_list, node) { + spin_lock(&sa->lock); + if (sa->aborting) { + spin_unlock(&sa->lock); + continue; + } + + sa->aborting =3D true; + found =3D refcount_inc_not_zero(&sa->refcnt); + spin_unlock(&sa->lock); + if (found) + break; + } + spin_unlock_bh(&ipsec->lock); + if (!found) + return; + + eip93_ipsec_abort_sa(sa, err); + eip93_ipsec_sa_put(sa); + } +} + +static void eip93_ipsec_fault_work(struct work_struct *work) +{ + struct eip93_ipsec *ipsec =3D + container_of(work, struct eip93_ipsec, fault_work); + enum eip93_ipsec_event event; + + spin_lock_bh(&ipsec->lock); + event =3D ipsec->fault_event; + spin_unlock_bh(&ipsec->lock); + + eip93_ipsec_abort_requests(ipsec, -EIO); + blocking_notifier_call_chain(&eip93_ipsec_notifier, event, ipsec); + eip93_ipsec_put(ipsec); +} + +void eip93_ipsec_handle_result(struct eip93_ipsec_request *req, int err, + u32 pe_ctrl_stat, u32 pe_length) +{ + struct eip93_ipsec_result result =3D {}; + + if (!req) + return; + + if (err =3D=3D -EIO || err =3D=3D -EACCES) + eip93_ipsec_mark_dead_async(req->sa->ipsec, + EIP93_IPSEC_EVENT_DMA_ERROR); + + if (!err) { + result.packet_len =3D FIELD_GET(EIP93_PE_LENGTH_LENGTH, pe_length); + result.nexthdr =3D FIELD_GET(EIP93_PE_CTRL_PE_PAD_VALUE, + pe_ctrl_stat); + } + + eip93_ipsec_complete_request(req, err, result); +} + +void eip93_ipsec_report_irq(struct eip93_device *eip93, u32 irq_status) +{ + struct eip93_ipsec *ipsec =3D eip93->ipsec; + + if (!ipsec) + return; + + if (irq_status & EIP93_INT_HALT) { + eip93_ipsec_mark_dead_async(ipsec, EIP93_IPSEC_EVENT_RESET); + return; + } + + if (irq_status & (EIP93_INT_INTERFACE_ERR | EIP93_INT_RPOC_ERR | + EIP93_INT_PE_RING_ERR)) + eip93_ipsec_mark_dead_async(ipsec, EIP93_IPSEC_EVENT_DMA_ERROR); +} + +int eip93_ipsec_register(struct eip93_device *eip93) +{ + struct eip93_ipsec *ipsec; + int err; + + ipsec =3D kzalloc(sizeof(*ipsec), GFP_KERNEL); + if (!ipsec) + return -ENOMEM; + + err =3D eip93_ipsec_init_prng(eip93); + if (err) { + kfree(ipsec); + return err; + } + + ipsec->eip93 =3D eip93; + ipsec->algo_flags =3D readl(eip93->base + EIP93_REG_PE_OPTION_1); + ipsec->fault_event =3D EIP93_IPSEC_EVENT_REMOVE; + INIT_WORK(&ipsec->fault_work, eip93_ipsec_fault_work); + spin_lock_init(&ipsec->lock); + refcount_set(&ipsec->refcnt, 1); + init_completion(&ipsec->done); + INIT_LIST_HEAD(&ipsec->node); + INIT_LIST_HEAD(&ipsec->sa_list); + + mutex_lock(&eip93_ipsec_devices_lock); + eip93->ipsec =3D ipsec; + list_add_tail(&ipsec->node, &eip93_ipsec_devices); + mutex_unlock(&eip93_ipsec_devices_lock); + + return 0; +} + +void eip93_ipsec_unregister(struct eip93_device *eip93) +{ + struct eip93_ipsec *ipsec =3D eip93->ipsec; + bool notify_remove; + + if (!ipsec) + return; + + mutex_lock(&eip93_ipsec_devices_lock); + notify_remove =3D eip93_ipsec_mark_dead(ipsec); + list_del_init(&ipsec->node); + eip93->ipsec =3D NULL; + mutex_unlock(&eip93_ipsec_devices_lock); + + eip93_ipsec_abort_requests(ipsec, -ENODEV); + if (notify_remove) + blocking_notifier_call_chain(&eip93_ipsec_notifier, + EIP93_IPSEC_EVENT_REMOVE, ipsec); + + eip93_ipsec_put(ipsec); + wait_for_completion(&ipsec->done); + cancel_work_sync(&ipsec->fault_work); + kfree(ipsec); +} + +int eip93_ipsec_state_add(struct eip93_ipsec *ipsec, struct xfrm_state *x, + struct netlink_ext_ack *extack, + struct eip93_ipsec_sa **sa) +{ + struct eip93_device *eip93; + struct eip93_ipsec_sa *new_sa; + unsigned int authkeylen; + unsigned int enckeylen; + int err; + + if (!ipsec || !eip93_ipsec_get_ref(ipsec)) { + NL_SET_ERR_MSG_MOD(extack, + "EIP93 packet backend is unavailable"); + return -EOPNOTSUPP; + } + + err =3D eip93_ipsec_validate_state(x, extack); + if (err) + goto put_ipsec; + + err =3D eip93_ipsec_validate_hw(x, ipsec->algo_flags, extack); + if (err) + goto put_ipsec; + + eip93 =3D ipsec->eip93; + new_sa =3D kzalloc(sizeof(*new_sa), GFP_KERNEL); + if (!new_sa) { + err =3D -ENOMEM; + goto put_ipsec; + } + + new_sa->ipsec =3D ipsec; + new_sa->family =3D x->props.family; + new_sa->ivsize =3D AES_BLOCK_SIZE; + new_sa->authsize =3D x->aalg->alg_trunc_len / BITS_PER_BYTE; + new_sa->blocksize =3D AES_BLOCK_SIZE; + new_sa->encap_type =3D x->encap ? x->encap->encap_type : 0; + new_sa->esn =3D x->props.flags & XFRM_STATE_ESN; + INIT_LIST_HEAD(&new_sa->node); + INIT_LIST_HEAD(&new_sa->requests); + spin_lock_init(&new_sa->lock); + refcount_set(&new_sa->refcnt, 1); + init_completion(&new_sa->done); + + err =3D eip93_ipsec_parse_flags(x, &new_sa->flags); + if (err) + goto free_sa; + + new_sa->sa_record =3D kzalloc(sizeof(*new_sa->sa_record), GFP_KERNEL); + if (!new_sa->sa_record) { + err =3D -ENOMEM; + goto free_sa; + } + + eip93_ipsec_init_sa_record(new_sa, x); + + enckeylen =3D x->ealg->alg_key_len / BITS_PER_BYTE; + memcpy(new_sa->sa_record->sa_key, x->ealg->alg_key, enckeylen); + + authkeylen =3D x->aalg->alg_key_len / BITS_PER_BYTE; + err =3D eip93_ipsec_hmac_setkey(new_sa->flags, x->aalg->alg_key, + authkeylen, + new_sa->sa_record->sa_i_digest, + new_sa->sa_record->sa_o_digest); + if (err) + goto free_record; + + new_sa->sa_record_base =3D dma_map_single(eip93->dev, new_sa->sa_record, + sizeof(*new_sa->sa_record), + DMA_TO_DEVICE); + if (dma_mapping_error(eip93->dev, new_sa->sa_record_base)) { + err =3D -ENOMEM; + goto free_record; + } + + spin_lock_bh(&ipsec->lock); + if (ipsec->dead) { + spin_unlock_bh(&ipsec->lock); + err =3D -EOPNOTSUPP; + goto unmap_record; + } + list_add_tail(&new_sa->node, &ipsec->sa_list); + spin_unlock_bh(&ipsec->lock); + + *sa =3D new_sa; + + return 0; + +unmap_record: + dma_unmap_single(eip93->dev, new_sa->sa_record_base, + sizeof(*new_sa->sa_record), DMA_TO_DEVICE); +free_record: + kfree_sensitive(new_sa->sa_record); +free_sa: + kfree(new_sa); +put_ipsec: + eip93_ipsec_put(ipsec); + return err; +} +EXPORT_SYMBOL_GPL(eip93_ipsec_state_add); + +void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa) +{ + if (!sa) + return; + + spin_lock_bh(&sa->ipsec->lock); + spin_lock(&sa->lock); + sa->dead =3D true; + list_del_init(&sa->node); + spin_unlock(&sa->lock); + spin_unlock_bh(&sa->ipsec->lock); + + eip93_ipsec_sa_put(sa); + wait_for_completion(&sa->done); + + dma_unmap_single(sa->ipsec->eip93->dev, sa->sa_record_base, + sizeof(*sa->sa_record), DMA_TO_DEVICE); + kfree_sensitive(sa->sa_record); + eip93_ipsec_put(sa->ipsec); + kfree(sa); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_state_delete); + +void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa, + struct xfrm_state *x) +{ + u32 seq_hi =3D 0; + + if (!sa || !x || !sa->esn || !x->replay_esn) + return; + + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_IN) + seq_hi =3D x->replay_esn->seq_hi; + else if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_OUT) + seq_hi =3D x->replay_esn->oseq_hi; + + spin_lock_bh(&sa->lock); + if (!sa->dead) { + sa->sa_record->sa_seqnum[1] =3D seq_hi; + dma_sync_single_for_device(sa->ipsec->eip93->dev, + sa->sa_record_base, + sizeof(*sa->sa_record), + DMA_TO_DEVICE); + } + spin_unlock_bh(&sa->lock); +} +EXPORT_SYMBOL_GPL(eip93_ipsec_state_advance_esn); + +int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa, struct sk_buff *skb, + unsigned int esp_offset, eip93_ipsec_complete_t complete, + void *data) +{ + struct eip93_descriptor cdesc =3D {}; + struct eip93_ipsec_request *req; + struct xfrm_offload *xo; + unsigned int payload_len; + unsigned int crypt_len; + unsigned int dma_len; + unsigned int tailen; + int err; + + if (!sa || !complete || !eip93_ipsec_sa_get(sa)) + return -EOPNOTSUPP; + + if (skb_is_nonlinear(skb)) { + err =3D -EINVAL; + goto put_sa; + } + + if (skb->len <=3D esp_offset + sizeof(struct ip_esp_hdr) + sa->ivsize) { + err =3D -EINVAL; + goto put_sa; + } + + xo =3D xfrm_offload(skb); + if (!xo) { + err =3D -EINVAL; + goto put_sa; + } + + tailen =3D xo->esp_tx_tailen; + if (tailen) { + payload_len =3D skb->len - esp_offset - sizeof(struct ip_esp_hdr) - + sa->ivsize; + dma_len =3D skb->len + tailen; + if (tailen > skb_tailroom(skb) || dma_len < skb->len) { + err =3D -ENOMEM; + goto put_sa; + } + } else { + u8 *trail; + u8 padlen; + + if (skb->len <=3D esp_offset + sizeof(struct ip_esp_hdr) + + sa->ivsize + sa->authsize) { + err =3D -EINVAL; + goto put_sa; + } + + crypt_len =3D skb->len - esp_offset - sizeof(struct ip_esp_hdr) - + sa->ivsize - sa->authsize; + if (crypt_len < 2) { + err =3D -EINVAL; + goto put_sa; + } + + trail =3D skb_tail_pointer(skb) - sa->authsize - 2; + padlen =3D trail[0]; + if (crypt_len < padlen + 2) { + err =3D -EINVAL; + goto put_sa; + } + + payload_len =3D crypt_len - padlen - 2; + dma_len =3D skb->len; + } + if (payload_len > FIELD_MAX(EIP93_PE_LENGTH_LENGTH)) { + err =3D -EINVAL; + goto put_sa; + } + + req =3D kmalloc(sizeof(*req), GFP_ATOMIC); + if (!req) { + err =3D -ENOMEM; + goto put_sa; + } + + req->sa =3D sa; + req->skb =3D skb; + INIT_LIST_HEAD(&req->node); + refcount_set(&req->refcnt, 1); + req->complete =3D complete; + req->data =3D data; + req->dma_len =3D dma_len; + req->dma_dir =3D DMA_BIDIRECTIONAL; + req->dma =3D dma_map_single(sa->ipsec->eip93->dev, skb->data, + req->dma_len, req->dma_dir); + if (dma_mapping_error(sa->ipsec->eip93->dev, req->dma)) { + err =3D -ENOMEM; + goto free_req; + } + + cdesc.pe_ctrl_stat_word =3D + FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN, + EIP93_PE_CTRL_HOST_READY) | + FIELD_PREP(EIP93_PE_CTRL_PE_PAD_CTRL_STAT, + EIP93_IPSEC_PAD_ALIGN) | + FIELD_PREP(EIP93_PE_CTRL_PE_PAD_VALUE, xo->proto) | + EIP93_PE_CTRL_PE_HASH_FINAL; + cdesc.src_addr =3D (u32 __force)req->dma + esp_offset + + sizeof(struct ip_esp_hdr) + sa->ivsize; + cdesc.dst_addr =3D (u32 __force)req->dma + esp_offset; + cdesc.sa_addr =3D sa->sa_record_base; + /* + * EIP93 ESP protocol-out mode wants the plaintext payload length. It + * generates ESP padding, next-header and ICV itself when tailroom was + * reserved instead of filled by the generic ESP path. + */ + cdesc.pe_length_word =3D FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY, + EIP93_PE_LENGTH_HOST_READY) | + FIELD_PREP(EIP93_PE_LENGTH_LENGTH, payload_len); + + err =3D eip93_ipsec_submit(req, &cdesc); + if (err =3D=3D -EINPROGRESS) + return err; + + dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len, + req->dma_dir); +free_req: + eip93_ipsec_request_put(req); +put_sa: + eip93_ipsec_sa_put(sa); + return err; +} +EXPORT_SYMBOL_GPL(eip93_ipsec_xmit); + +int eip93_ipsec_receive(struct eip93_ipsec_sa *sa, struct sk_buff *skb, + unsigned int packet_len, + eip93_ipsec_complete_t complete, void *data) +{ + struct eip93_descriptor cdesc =3D {}; + struct eip93_ipsec_request *req; + int err; + + if (!sa || !complete || !eip93_ipsec_sa_get(sa)) + return -EOPNOTSUPP; + + if (skb_is_nonlinear(skb)) { + err =3D -EINVAL; + goto put_sa; + } + + req =3D kmalloc(sizeof(*req), GFP_ATOMIC); + if (!req) { + err =3D -ENOMEM; + goto put_sa; + } + + req->sa =3D sa; + req->skb =3D skb; + INIT_LIST_HEAD(&req->node); + refcount_set(&req->refcnt, 1); + req->complete =3D complete; + req->data =3D data; + if (!packet_len || packet_len > skb->len || + packet_len > FIELD_MAX(EIP93_PE_LENGTH_LENGTH)) { + err =3D -EINVAL; + goto free_req; + } + + req->dma_len =3D packet_len; + req->dma_dir =3D DMA_BIDIRECTIONAL; + req->dma =3D dma_map_single(sa->ipsec->eip93->dev, skb->data, + req->dma_len, req->dma_dir); + if (dma_mapping_error(sa->ipsec->eip93->dev, req->dma)) { + err =3D -ENOMEM; + goto free_req; + } + + cdesc.pe_ctrl_stat_word =3D + FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN, + EIP93_PE_CTRL_HOST_READY) | + FIELD_PREP(EIP93_PE_CTRL_PE_PAD_CTRL_STAT, + EIP93_IPSEC_PAD_ALIGN) | + EIP93_PE_CTRL_PE_HASH_FINAL; + cdesc.src_addr =3D (u32 __force)req->dma; + cdesc.dst_addr =3D (u32 __force)req->dma; + cdesc.sa_addr =3D sa->sa_record_base; + cdesc.pe_length_word =3D FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY, + EIP93_PE_LENGTH_HOST_READY) | + FIELD_PREP(EIP93_PE_LENGTH_LENGTH, req->dma_len); + + err =3D eip93_ipsec_submit(req, &cdesc); + if (err =3D=3D -EINPROGRESS) + return err; + + dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len, + req->dma_dir); +free_req: + eip93_ipsec_request_put(req); +put_sa: + eip93_ipsec_sa_put(sa); + return err; +} +EXPORT_SYMBOL_GPL(eip93_ipsec_receive); diff --git a/drivers/crypto/inside-secure/eip93/eip93-main.c b/drivers/cryp= to/inside-secure/eip93/eip93-main.c index 7dccfdeb7b11..1505e33d62bf 100644 --- a/drivers/crypto/inside-secure/eip93/eip93-main.c +++ b/drivers/crypto/inside-secure/eip93/eip93-main.c @@ -185,7 +185,9 @@ static int eip93_register_algs(struct eip93_device *eip= 93, u32 supported_algo_fl =20 static void eip93_handle_result_descriptor(struct eip93_device *eip93) { - struct crypto_async_request *async; + struct crypto_async_request *async =3D NULL; + struct eip93_ipsec_request *ipsec =3D NULL; + void *request; struct eip93_descriptor *rdesc; u16 desc_flags, crypto_idr; bool last_entry; @@ -224,11 +226,11 @@ static void eip93_handle_result_descriptor(struct eip= 93_device *eip93) FIELD_GET(EIP93_PE_LENGTH_HOST_PE_READY, pe_length) !=3D EIP93_PE_LENGTH_PE_READY); =20 - err =3D rdesc->pe_ctrl_stat_word & (EIP93_PE_CTRL_PE_EXT_ERR_CODE | - EIP93_PE_CTRL_PE_EXT_ERR | - EIP93_PE_CTRL_PE_SEQNUM_ERR | - EIP93_PE_CTRL_PE_PAD_ERR | - EIP93_PE_CTRL_PE_AUTH_ERR); + err =3D pe_ctrl_stat & (EIP93_PE_CTRL_PE_EXT_ERR_CODE | + EIP93_PE_CTRL_PE_EXT_ERR | + EIP93_PE_CTRL_PE_SEQNUM_ERR | + EIP93_PE_CTRL_PE_PAD_ERR | + EIP93_PE_CTRL_PE_AUTH_ERR); =20 desc_flags =3D FIELD_GET(EIP93_PE_USER_ID_DESC_FLAGS, rdesc->user_id); crypto_idr =3D FIELD_GET(EIP93_PE_USER_ID_CRYPTO_IDR, rdesc->user_id); @@ -248,23 +250,37 @@ static void eip93_handle_result_descriptor(struct eip= 93_device *eip93) if (!last_entry) goto get_more; =20 - /* Get crypto async ref only for last descriptor */ + /* Get request ref only for last descriptor */ scoped_guard(spinlock_bh, &eip93->ring->idr_lock) { - async =3D idr_find(&eip93->ring->crypto_async_idr, crypto_idr); + request =3D idr_find(&eip93->ring->crypto_async_idr, crypto_idr); idr_remove(&eip93->ring->crypto_async_idr, crypto_idr); } + if (!request) { + dev_warn_ratelimited(eip93->dev, "missing request id %u\n", + crypto_idr); + goto get_more; + } =20 /* Parse error in ctrl stat word */ err =3D eip93_parse_ctrl_stat_err(eip93, err); =20 + if (desc_flags & EIP93_DESC_IPSEC) { + ipsec =3D request; + eip93_ipsec_handle_result(ipsec, err, pe_ctrl_stat, pe_length); + goto get_more; + } + + async =3D request; + if (desc_flags & EIP93_DESC_SKCIPHER) eip93_skcipher_handle_result(async, err); - - if (desc_flags & EIP93_DESC_AEAD) + else if (desc_flags & EIP93_DESC_AEAD) eip93_aead_handle_result(async, err); - - if (desc_flags & EIP93_DESC_HASH) + else if (desc_flags & EIP93_DESC_HASH) eip93_hash_handle_result(async, err); + else + dev_warn_ratelimited(eip93->dev, "unknown descriptor flags %#x\n", + desc_flags); =20 goto get_more; } @@ -279,21 +295,26 @@ static void eip93_done_task(unsigned long data) static irqreturn_t eip93_irq_handler(int irq, void *data) { struct eip93_device *eip93 =3D data; + bool handled =3D false; u32 irq_status; =20 irq_status =3D readl(eip93->base + EIP93_REG_INT_MASK_STAT); if (FIELD_GET(EIP93_INT_RDR_THRESH, irq_status)) { eip93_irq_disable(eip93, EIP93_INT_RDR_THRESH); tasklet_schedule(&eip93->ring->done_task); - return IRQ_HANDLED; + irq_status &=3D ~EIP93_INT_RDR_THRESH; + handled =3D true; } =20 - /* Ignore errors in AUTO mode, handled by the RDR */ + if (!irq_status) + return handled ? IRQ_HANDLED : IRQ_NONE; + + eip93_ipsec_report_irq(eip93, irq_status); + eip93_irq_clear(eip93, irq_status); - if (irq_status) - eip93_irq_disable(eip93, irq_status); + eip93_irq_disable(eip93, irq_status); =20 - return IRQ_NONE; + return IRQ_HANDLED; } =20 static void eip93_initialize(struct eip93_device *eip93, u32 supported_alg= o_flags) @@ -455,15 +476,24 @@ static int eip93_crypto_probe(struct platform_device = *pdev) =20 eip93_initialize(eip93, algo_flags); =20 - /* Init finished, enable RDR interrupt */ - eip93_irq_enable(eip93, EIP93_INT_RDR_THRESH); + ret =3D eip93_ipsec_register(eip93); + if (ret) { + eip93_cleanup(eip93); + return ret; + } =20 ret =3D eip93_register_algs(eip93, algo_flags); if (ret) { + eip93_ipsec_unregister(eip93); eip93_cleanup(eip93); return ret; } =20 + /* Init finished, enable RDR and fatal error interrupts */ + eip93_irq_enable(eip93, EIP93_INT_RDR_THRESH | EIP93_INT_INTERFACE_ERR | + EIP93_INT_RPOC_ERR | EIP93_INT_PE_RING_ERR | + EIP93_INT_HALT); + ver =3D readl(eip93->base + EIP93_REG_PE_REVISION); /* EIP_EIP_NO:MAJOR_HW_REV:MINOR_HW_REV:HW_PATCH,PE(ALGO_FLAGS) */ dev_info(eip93->dev, "EIP%lu:%lx:%lx:%lx,PE(0x%x:0x%x)\n", @@ -484,6 +514,7 @@ static void eip93_crypto_remove(struct platform_device = *pdev) =20 algo_flags =3D readl(eip93->base + EIP93_REG_PE_OPTION_1); =20 + eip93_ipsec_unregister(eip93); eip93_unregister_algs(algo_flags, ARRAY_SIZE(eip93_algs)); eip93_cleanup(eip93); } diff --git a/drivers/crypto/inside-secure/eip93/eip93-main.h b/drivers/cryp= to/inside-secure/eip93/eip93-main.h index 990c2401b7ce..ca1bda5b2ac0 100644 --- a/drivers/crypto/inside-secure/eip93/eip93-main.h +++ b/drivers/crypto/inside-secure/eip93/eip93-main.h @@ -13,6 +13,7 @@ #include #include #include +#include =20 #define EIP93_RING_BUSY_DELAY 500 =20 @@ -92,6 +93,8 @@ EIP93_HASH_SHA224 | \ EIP93_HASH_SHA256)) =20 +struct eip93_ipsec; + /** * struct eip93_device - crypto engine device structure */ @@ -101,6 +104,7 @@ struct eip93_device { struct clk *clk; int irq; struct eip93_ring *ring; + struct eip93_ipsec *ipsec; }; =20 struct eip93_desc_ring { @@ -124,8 +128,8 @@ struct eip93_ring { /* command/result rings */ struct eip93_desc_ring cdr; struct eip93_desc_ring rdr; - spinlock_t write_lock; - spinlock_t read_lock; + spinlock_t write_lock; /* command descriptor enqueue */ + spinlock_t read_lock; /* result descriptor dequeue */ /* aync idr */ spinlock_t idr_lock; struct idr crypto_async_idr; @@ -148,4 +152,34 @@ struct eip93_alg_template { } alg; }; =20 +struct eip93_ipsec_request; + +#if IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC) +int eip93_ipsec_register(struct eip93_device *eip93); +void eip93_ipsec_unregister(struct eip93_device *eip93); +void eip93_ipsec_handle_result(struct eip93_ipsec_request *req, int err, + u32 pe_ctrl_stat, u32 pe_length); +void eip93_ipsec_report_irq(struct eip93_device *eip93, u32 irq_status); +#else +static inline int eip93_ipsec_register(struct eip93_device *eip93) +{ + return 0; +} + +static inline void eip93_ipsec_unregister(struct eip93_device *eip93) +{ +} + +static inline void eip93_ipsec_handle_result(struct eip93_ipsec_request *r= eq, + int err, u32 pe_ctrl_stat, + u32 pe_length) +{ +} + +static inline void eip93_ipsec_report_irq(struct eip93_device *eip93, + u32 irq_status) +{ +} +#endif + #endif /* _EIP93_MAIN_H_ */ diff --git a/include/crypto/eip93-ipsec.h b/include/crypto/eip93-ipsec.h new file mode 100644 index 000000000000..bc0ba8f4f84e --- /dev/null +++ b/include/crypto/eip93-ipsec.h @@ -0,0 +1,132 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * EIP93 IPsec offload API + * + * Copyright (c) 2026 Jihong Min + */ +#ifndef _CRYPTO_EIP93_IPSEC_H +#define _CRYPTO_EIP93_IPSEC_H + +#include +#include +#include +#include +#include + +struct device; +struct netlink_ext_ack; +struct notifier_block; +struct sk_buff; +struct xfrm_state; + +struct eip93_ipsec; +struct eip93_ipsec_sa; + +struct eip93_ipsec_result { + unsigned int packet_len; + u8 nexthdr; +}; + +enum eip93_ipsec_feature { + EIP93_IPSEC_FEATURE_ESP =3D BIT(0), + EIP93_IPSEC_FEATURE_GSO_ESP =3D BIT(1), + EIP93_IPSEC_FEATURE_HW_ESP_TX_CSUM =3D BIT(2), +}; + +enum eip93_ipsec_event { + EIP93_IPSEC_EVENT_REMOVE, + EIP93_IPSEC_EVENT_RESET, + EIP93_IPSEC_EVENT_DMA_ERROR, + EIP93_IPSEC_EVENT_CAPABILITY_LOSS, +}; + +typedef void (*eip93_ipsec_complete_t)(void *data, int err, + struct eip93_ipsec_result result); + +#if IS_REACHABLE(CONFIG_CRYPTO_DEV_EIP93) && \ + IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC) +struct eip93_ipsec *eip93_ipsec_get(struct device *consumer); +void eip93_ipsec_put(struct eip93_ipsec *ipsec); +bool eip93_ipsec_available(struct eip93_ipsec *ipsec); +u32 eip93_ipsec_features(struct eip93_ipsec *ipsec); +int eip93_ipsec_register_notifier(struct notifier_block *nb); +void eip93_ipsec_unregister_notifier(struct notifier_block *nb); +int eip93_ipsec_state_add(struct eip93_ipsec *ipsec, struct xfrm_state *x, + struct netlink_ext_ack *extack, + struct eip93_ipsec_sa **sa); +void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa); +void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa, + struct xfrm_state *x); +int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa, struct sk_buff *skb, + unsigned int esp_offset, eip93_ipsec_complete_t complete, + void *data); +int eip93_ipsec_receive(struct eip93_ipsec_sa *sa, struct sk_buff *skb, + unsigned int packet_len, + eip93_ipsec_complete_t complete, void *data); +#else +static inline struct eip93_ipsec *eip93_ipsec_get(struct device *consumer) +{ + return ERR_PTR(-EOPNOTSUPP); +} + +static inline void eip93_ipsec_put(struct eip93_ipsec *ipsec) +{ +} + +static inline bool eip93_ipsec_available(struct eip93_ipsec *ipsec) +{ + return false; +} + +static inline u32 eip93_ipsec_features(struct eip93_ipsec *ipsec) +{ + return 0; +} + +static inline int eip93_ipsec_register_notifier(struct notifier_block *nb) +{ + return 0; +} + +static inline void eip93_ipsec_unregister_notifier(struct notifier_block *= nb) +{ +} + +static inline int eip93_ipsec_state_add(struct eip93_ipsec *ipsec, + struct xfrm_state *x, + struct netlink_ext_ack *extack, + struct eip93_ipsec_sa **sa) +{ + if (sa) + *sa =3D NULL; + + return -EOPNOTSUPP; +} + +static inline void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa) +{ +} + +static inline void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa, + struct xfrm_state *x) +{ +} + +static inline int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa, + struct sk_buff *skb, unsigned int esp_offset, + eip93_ipsec_complete_t complete, void *data) +{ + return -EOPNOTSUPP; +} + +static inline int eip93_ipsec_receive(struct eip93_ipsec_sa *sa, + struct sk_buff *skb, + unsigned int packet_len, + eip93_ipsec_complete_t complete, + void *data) +{ + return -EOPNOTSUPP; +} +#endif + +#endif /* _CRYPTO_EIP93_IPSEC_H */ --=20 2.53.0 From nobody Sun May 24 19:33:57 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A639C33E35B for ; Sat, 23 May 2026 12:15:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538554; cv=none; b=HcUFOjrHsjdQkT9eB6tcmLbz4X5g9pgnTj0jZM9pyMIqpyG/BbFqg8wsSjPVkhn+uLdsuNvBCe4V8IfSUDu3YX1Q1VO0GcqqJBbM8Nb1Nr8c7s+eurLCP5EOh9ELoy2aeFiOm5DLyaUprONsx6fpSI86LG/NAPrOoYNjA9DS2Ek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779538554; c=relaxed/simple; bh=fjwfnG5yWXy4P8hjyB4XJxxhQUW0RkOHtDFLSSnU9Rc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Kw7VMQiZC4Shpjr16JhTjb8PcgrvzBXa1w1GGNdcvjh0gvF0mWGySCeVMb6vgNWficLwVe7Y9//oKYiR8qy5TdEOwQSrTzkcHKrKk7lPp9ixxtV0oRSnN9OP/CkxhKrvuJRBBssbL/sGVMOrX84HM6hfpMqRuIatCEXjSkLKn88= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fexTbfTT; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fexTbfTT" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-36936dcf19dso3997085a91.0 for ; Sat, 23 May 2026 05:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779538551; x=1780143351; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5sAsnB5BoZCJUM/j96raiEkdLzi0ahm+edM5QqiMf8Q=; b=fexTbfTTIcfJl6hpmBqnwyMy9SBHob7BZS93YNP7mwUQ0nYzFNPMBzpXnGcqxtrqiG irHNu4HhxRMk74Uh0ratiEiUp3Osi7puov3n8q16cqF1a6DXvz4xbZcZ6Lq5xhw4ac6Z 53iIpM3VV3xzkrMDIQE6ZoUmr3ufVDEXrPoUzzdBfAHjStlU/fTEv4D+u6xBM/HmhZmU Yogfy+evK7ZAmLwvKiIY14nqVfVf0sbCooeOw/OFx5MjVHShOHvo9xM5tR4AEuta3rMP sgE/1WwfO1vAzjI6OLvVAyMTzgja1pUTEq6MsWeP8jVuVujGD2AiEzV3CLnwksDr/FBp Sjpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779538551; x=1780143351; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5sAsnB5BoZCJUM/j96raiEkdLzi0ahm+edM5QqiMf8Q=; b=LPsSAiXHwbADrpERaZVaahObV9baubgYG05TrIwR882+0SFLWI7sc/xY5z3wrlTHsA jL8Uk592V9EpqTuxhjr/9cD+rvQ5RFSOBZXq/2ZPlT5LlSf8xMHMvg5KCmZbJTSuPh7w UAyMuM+oenBUwEzOD1dCJYenoMr7hxmGErs9FN1JruDYVjE3eBh2QhsD6FqjEP4Qa3B9 FtS+IpbAlC+j4URWPoqt1de7+G5Xw9ZI9obMAqlndAiwNS+pOOrqUeLImaqBcaeDaIeJ 38UVactPOMWoTVenvS++OBwmtFDV0wr+EvH5oFlsQxdC5JOOAhfVvRmXktlyHIH5zr94 PpfQ== X-Gm-Message-State: AOJu0YwmiuqWJL+GJGxmz/iNcR9vkMr/PZnw9v1bw6ap+k6+N10WyJkC F6OYB68yy876J3IdCrnZktzQVK3+aSCZFNPR9xRzKa0RGHu0bzebd/h/ X-Gm-Gg: Acq92OFYxIvh0u3s5x1HmajFCLEsYy6mw+mcs0OFCNC6VA1Drv93mogs8tiIkHd/pFN elQi0gOq05Vu9ZFGGHvldx48ElscLcNTquKokeiPipvMPiFhXYLP9uz+4Sp/2yvO8Q7K0JhuWWF ZYkFqJTdy3xlbgbWV/uL44wT3EfOqG1acg8E296OFzKL2M6XJhSqiJtchZxLfeK8Kh9dWtQUjsZ LvXaZDBif8KYvWEhJaxqMuCFHZIr7UOVXzgND6zlh8TsgIiD5NXrbMgR1Jwbm3lXdgFYkShtekx q3tHxmECZ8TRgQ/cJiBzkpH2wZGSLrxfyzYYoNWn78pX9/Y/leDzCEJTA0lmPvMrs+VJq2XVgOA 0uTp5qBhPpaTgHsQFeVYArc5SLGZz7LTJX/5WPtSAZ466gnLd6rgqa54Yui3qJwcZBn7vQKaQGN jOp7D5h6o2nkc9MCGbf0k85LCmAMOJUhqwdVQ= X-Received: by 2002:a17:902:ffcf:b0:2b0:663f:6b53 with SMTP id d9443c01a7336-2beb0385f3amr81201125ad.13.1779538550701; Sat, 23 May 2026 05:15:50 -0700 (PDT) Received: from mincom1 ([125.149.177.227]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2beb56ce4easm41746305ad.30.2026.05.23.05.15.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 May 2026 05:15:50 -0700 (PDT) From: Jihong Min To: Christian Marangi , Antoine Tenart , Herbert Xu , "David S . Miller" , Lorenzo Bianconi , Andrew Lunn , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, netdev@vger.kernel.org, Jihong Min Subject: [PATCH 3/3] net: airoha: add EIP93-backed ESP XFRM offload Date: Sat, 23 May 2026 21:15:22 +0900 Message-ID: <20260523121522.3023992-4-hurryman2212@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com> References: <20260523121522.3023992-1-hurryman2212@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Wire Airoha GDM netdevs and DSA user ports to the EIP93 ESP packet backend through xfrmdev_ops. Gate netdev feature advertisement on backend capability, add TX and RX submit paths, preserve opt-out builds, and handle SA lifetime across feature changes, DSA detach, and EIP93 provider loss. Assisted-by: Codex:gpt-5.5 Signed-off-by: Jihong Min --- drivers/net/ethernet/airoha/Kconfig | 11 + drivers/net/ethernet/airoha/Makefile | 1 + drivers/net/ethernet/airoha/airoha_eth.c | 51 +- drivers/net/ethernet/airoha/airoha_eth.h | 69 + drivers/net/ethernet/airoha/airoha_xfrm.c | 1474 +++++++++++++++++++++ 5 files changed, 1605 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/airoha/airoha_xfrm.c diff --git a/drivers/net/ethernet/airoha/Kconfig b/drivers/net/ethernet/air= oha/Kconfig index ad3ce501e7a5..302534c89fdd 100644 --- a/drivers/net/ethernet/airoha/Kconfig +++ b/drivers/net/ethernet/airoha/Kconfig @@ -31,4 +31,15 @@ config NET_AIROHA_FLOW_STATS help Enable Aiorha flowtable statistic counters. =20 +config NET_AIROHA_XFRM + bool "Airoha ESP XFRM offload support" + depends on NET_AIROHA + default y + help + Enable ESP XFRM offload support for Airoha Ethernet netdevs. + + If unsure, say Y. Say N to opt out of advertising ESP hardware + offload from the Airoha Ethernet driver even when the EIP93 IPsec + packet backend and XFRM offload support are available. + endif #NET_VENDOR_AIROHA diff --git a/drivers/net/ethernet/airoha/Makefile b/drivers/net/ethernet/ai= roha/Makefile index 94468053e34b..15386665bb27 100644 --- a/drivers/net/ethernet/airoha/Makefile +++ b/drivers/net/ethernet/airoha/Makefile @@ -5,5 +5,6 @@ =20 obj-$(CONFIG_NET_AIROHA) +=3D airoha-eth.o airoha-eth-y :=3D airoha_eth.o airoha_ppe.o +airoha-eth-$(CONFIG_NET_AIROHA_XFRM) +=3D airoha_xfrm.o airoha-eth-$(CONFIG_DEBUG_FS) +=3D airoha_ppe_debugfs.o obj-$(CONFIG_NET_AIROHA_NPU) +=3D airoha_npu.o diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/etherne= t/airoha/airoha_eth.c index cecd66251dba..877002c03738 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.c +++ b/drivers/net/ethernet/airoha/airoha_eth.c @@ -684,6 +684,14 @@ static int airoha_qdma_rx_process(struct airoha_queue = *q, int budget) false); =20 done++; +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) + if (airoha_xfrm_in_active(port) && + airoha_xfrm_rx_skb(port, q->skb)) { + q->skb =3D NULL; + continue; + } +#endif + napi_gro_receive(&q->napi, q->skb); q->skb =3D NULL; continue; @@ -2010,6 +2018,19 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *s= kb, void *data; u16 index; u8 fport; +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) + int err; + + if (airoha_xfrm_out_active(port)) { + err =3D airoha_xfrm_encrypt_skb(port, skb); + if (err =3D=3D -EINPROGRESS) + return NETDEV_TX_OK; + if (err =3D=3D -EBUSY) + return NETDEV_TX_BUSY; + if (err) + goto error; + } +#endif =20 qid =3D airoha_qdma_get_txq(qdma, skb_get_queue_mapping(skb)); tag =3D airoha_get_dsa_tag(skb, dev); @@ -2895,6 +2916,8 @@ static const struct net_device_ops airoha_netdev_ops = =3D { .ndo_stop =3D airoha_dev_stop, .ndo_change_mtu =3D airoha_dev_change_mtu, .ndo_select_queue =3D airoha_dev_select_queue, + .ndo_fix_features =3D airoha_xfrm_fix_features, + .ndo_set_features =3D airoha_xfrm_set_features, .ndo_start_xmit =3D airoha_dev_xmit, .ndo_get_stats64 =3D airoha_dev_get_stats64, .ndo_set_mac_address =3D airoha_dev_set_macaddr, @@ -3025,6 +3048,7 @@ static int airoha_alloc_gdm_port(struct airoha_eth *e= th, /* XXX: Read nbq from DTS */ port->nbq =3D id =3D=3D AIROHA_GDM3_IDX && airoha_is_7581(eth) ? 4 : 0; eth->ports[p] =3D port; + airoha_xfrm_build_netdev(dev); =20 return airoha_metadata_dst_alloc(port); } @@ -3155,6 +3179,7 @@ static int airoha_probe(struct platform_device *pdev) =20 if (port->dev->reg_state =3D=3D NETREG_REGISTERED) unregister_netdev(port->dev); + airoha_xfrm_teardown_netdev(port->dev); airoha_metadata_dst_free(port); } airoha_hw_cleanup(eth); @@ -3180,6 +3205,7 @@ static void airoha_remove(struct platform_device *pde= v) continue; =20 unregister_netdev(port->dev); + airoha_xfrm_teardown_netdev(port->dev); airoha_metadata_dst_free(port); } airoha_hw_cleanup(eth); @@ -3328,7 +3354,30 @@ static struct platform_driver airoha_driver =3D { .of_match_table =3D of_airoha_match, }, }; -module_platform_driver(airoha_driver); + +static int __init airoha_init(void) +{ + int err; + + err =3D airoha_xfrm_register_notifier(); + if (err) + return err; + + err =3D platform_driver_register(&airoha_driver); + if (err) + airoha_xfrm_unregister_notifier(); + + return err; +} + +static void __exit airoha_exit(void) +{ + platform_driver_unregister(&airoha_driver); + airoha_xfrm_unregister_notifier(); +} + +module_init(airoha_init); +module_exit(airoha_exit); =20 MODULE_LICENSE("GPL"); MODULE_AUTHOR("Lorenzo Bianconi "); diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/etherne= t/airoha/airoha_eth.h index 4fad3acc3ccf..4fe04c763271 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.h +++ b/drivers/net/ethernet/airoha/airoha_eth.h @@ -11,6 +11,8 @@ #include #include #include +#include +#include #include #include #include @@ -533,6 +535,12 @@ struct airoha_qdma { struct airoha_queue q_rx[AIROHA_NUM_RX_RING]; }; =20 +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) +struct eip93_ipsec; +DECLARE_STATIC_KEY_FALSE(airoha_xfrm_in_state_key); +DECLARE_STATIC_KEY_FALSE(airoha_xfrm_out_state_key); +#endif + struct airoha_gdm_port { struct airoha_qdma *qdma; struct airoha_eth *eth; @@ -549,6 +557,13 @@ struct airoha_gdm_port { u64 fwd_tx_packets; =20 struct metadata_dst *dsa_meta[AIROHA_MAX_DSA_PORTS]; + +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) + struct eip93_ipsec *xfrm_ipsec; + atomic_t xfrm_state_count; + atomic_t xfrm_out_state_count; + atomic_t xfrm_in_state_count; +#endif }; =20 #define AIROHA_RXD4_PPE_CPU_REASON GENMASK(20, 16) @@ -683,4 +698,58 @@ static inline int airoha_ppe_debugfs_init(struct airoh= a_ppe *ppe) } #endif =20 +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) +static inline bool airoha_xfrm_in_active(struct airoha_gdm_port *port) +{ + return static_branch_unlikely(&airoha_xfrm_in_state_key) && + atomic_read(&port->xfrm_in_state_count); +} + +static inline bool airoha_xfrm_out_active(struct airoha_gdm_port *port) +{ + return static_branch_unlikely(&airoha_xfrm_out_state_key) && + atomic_read(&port->xfrm_out_state_count); +} + +void airoha_xfrm_build_netdev(struct net_device *dev); +void airoha_xfrm_teardown_netdev(struct net_device *dev); +netdev_features_t airoha_xfrm_fix_features(struct net_device *dev, + netdev_features_t features); +int airoha_xfrm_set_features(struct net_device *dev, + netdev_features_t features); +bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb); +int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *= skb); +int airoha_xfrm_register_notifier(void); +void airoha_xfrm_unregister_notifier(void); +#else +static inline void airoha_xfrm_build_netdev(struct net_device *dev) +{ +} + +static inline void airoha_xfrm_teardown_netdev(struct net_device *dev) +{ +} + +static inline netdev_features_t +airoha_xfrm_fix_features(struct net_device *dev, netdev_features_t feature= s) +{ + return features; +} + +static inline int airoha_xfrm_set_features(struct net_device *dev, + netdev_features_t features) +{ + return 0; +} + +static inline int airoha_xfrm_register_notifier(void) +{ + return 0; +} + +static inline void airoha_xfrm_unregister_notifier(void) +{ +} +#endif + #endif /* AIROHA_ETH_H */ diff --git a/drivers/net/ethernet/airoha/airoha_xfrm.c b/drivers/net/ethern= et/airoha/airoha_xfrm.c new file mode 100644 index 000000000000..58461954d098 --- /dev/null +++ b/drivers/net/ethernet/airoha/airoha_xfrm.c @@ -0,0 +1,1474 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2026 Jihong Min + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "airoha_eth.h" + +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) +DEFINE_STATIC_KEY_FALSE(airoha_xfrm_in_state_key); +DEFINE_STATIC_KEY_FALSE(airoha_xfrm_out_state_key); +#endif + +#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) && \ + IS_REACHABLE(CONFIG_CRYPTO_DEV_EIP93) && \ + IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC) && \ + IS_REACHABLE(CONFIG_INET_ESP) && \ + IS_REACHABLE(CONFIG_INET_ESP_OFFLOAD) && \ + IS_ENABLED(CONFIG_XFRM_OFFLOAD) +#define AIROHA_XFRM_FEATURES \ + (NETIF_F_HW_ESP | NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP) + +struct airoha_xfrm_state { + struct airoha_gdm_port *port; + struct eip93_ipsec_sa *sa; +}; + +static netdev_features_t airoha_xfrm_ipsec_features(struct eip93_ipsec *ip= sec) +{ + netdev_features_t features =3D 0; + u32 ipsec_features; + + ipsec_features =3D eip93_ipsec_features(ipsec); + if (ipsec_features & EIP93_IPSEC_FEATURE_ESP) + features |=3D NETIF_F_HW_ESP; + if (ipsec_features & EIP93_IPSEC_FEATURE_HW_ESP_TX_CSUM) + features |=3D NETIF_F_HW_ESP_TX_CSUM; + if (ipsec_features & EIP93_IPSEC_FEATURE_GSO_ESP) + features |=3D NETIF_F_GSO_ESP; + + return features; +} + +static int airoha_xfrm_request_module(struct net_device *dev, + const char *module_name) +{ + int err; + + err =3D request_module("%s", module_name); + if (err) { + netdev_err(dev, "failed requesting module %s: %d\n", + module_name, err); + return err < 0 ? err : -ENOENT; + } + + return 0; +} + +static int airoha_xfrm_request_modules(struct net_device *dev) +{ + int err; + + if (IS_MODULE(CONFIG_INET_ESP)) { + err =3D airoha_xfrm_request_module(dev, "esp4"); + if (err) + return err; + } + + if (IS_MODULE(CONFIG_INET_ESP_OFFLOAD)) { + err =3D airoha_xfrm_request_module(dev, "esp4_offload"); + if (err) + return err; + } + +#if IS_REACHABLE(CONFIG_INET6_ESP) + if (IS_MODULE(CONFIG_INET6_ESP)) { + err =3D airoha_xfrm_request_module(dev, "esp6"); + if (err) + return err; + } +#endif + +#if IS_REACHABLE(CONFIG_INET6_ESP_OFFLOAD) + if (IS_MODULE(CONFIG_INET6_ESP_OFFLOAD)) { + err =3D airoha_xfrm_request_module(dev, "esp6_offload"); + if (err) + return err; + } +#endif + + if (IS_MODULE(CONFIG_CRYPTO_DEV_EIP93)) { + err =3D airoha_xfrm_request_module(dev, "crypto-hw-eip93"); + if (err) + return err; + } + + return 0; +} + +static int airoha_xfrm_prepare_ipsec(struct net_device *dev) +{ + struct airoha_gdm_port *port =3D netdev_priv(dev); + struct eip93_ipsec *ipsec; + int err; + + if (port->xfrm_ipsec) + return eip93_ipsec_available(port->xfrm_ipsec) ? 0 : -ENODEV; + + err =3D airoha_xfrm_request_modules(dev); + if (err) + return err; + + ipsec =3D eip93_ipsec_get(port->eth->dev); + if (IS_ERR(ipsec)) { + netdev_dbg(dev, + "EIP93 ESP packet backend is unavailable: %ld\n", + PTR_ERR(ipsec)); + return PTR_ERR(ipsec); + } + + port->xfrm_ipsec =3D ipsec; + netdev_info(dev, "ESP HW offload available via EIP93 packet backend\n"); + + return 0; +} + +static bool airoha_xfrm_state_supported(struct xfrm_state *x, + struct netlink_ext_ack *extack) +{ + if (x->xso.type !=3D XFRM_DEV_OFFLOAD_CRYPTO) { + NL_SET_ERR_MSG_MOD(extack, + "only XFRM crypto offload is supported"); + return false; + } + + switch (x->xso.dir) { + case XFRM_DEV_OFFLOAD_OUT: + case XFRM_DEV_OFFLOAD_IN: + break; + default: + NL_SET_ERR_MSG_MOD(extack, "only in/out SAs are supported"); + return false; + } + + switch (x->props.family) { + case AF_INET: + break; +#if IS_REACHABLE(CONFIG_INET6_ESP) && IS_REACHABLE(CONFIG_INET6_ESP_OFFLOA= D) + case AF_INET6: + break; +#endif + default: + NL_SET_ERR_MSG_MOD(extack, + "only IPv4/IPv6 ESP offload is supported"); + return false; + } + + if (x->outer_mode.family !=3D x->props.family) { + NL_SET_ERR_MSG_MOD(extack, + "only same-family ESP offload is supported"); + return false; + } + + if (x->id.proto !=3D IPPROTO_ESP) { + NL_SET_ERR_MSG_MOD(extack, "only ESP offload is supported"); + return false; + } + + switch (x->props.mode) { + case XFRM_MODE_TUNNEL: + case XFRM_MODE_TRANSPORT: + break; + default: + NL_SET_ERR_MSG_MOD(extack, + "only tunnel/transport modes are supported"); + return false; + } + + if (x->outer_mode.encap !=3D x->props.mode) { + NL_SET_ERR_MSG_MOD(extack, + "outer ESP mode does not match state mode"); + return false; + } + + if (x->encap) { + NL_SET_ERR_MSG_MOD(extack, + "NAT-T is unsupported by EIP93 packet ESP"); + return false; + } + + if (x->tfcpad) { + NL_SET_ERR_MSG_MOD(extack, "TFC padding is not supported"); + return false; + } + + if (x->aead) { + NL_SET_ERR_MSG_MOD(extack, "AEAD SAs are unsupported"); + return false; + } + + if (!x->ealg || !x->aalg) { + NL_SET_ERR_MSG_MOD(extack, + "encryption/authentication required"); + return false; + } + + return true; +} + +static const struct xfrmdev_ops airoha_xfrmdev_ops; + +#if IS_ENABLED(CONFIG_NET_DSA) +static struct airoha_gdm_port *airoha_xfrm_dsa_dev_port(struct net_device = *dev) +{ + struct net_device *conduit; + struct dsa_port *dp; + + if (!dsa_user_dev_check(dev)) + return NULL; + + dp =3D dsa_port_from_netdev(dev); + if (IS_ERR(dp)) + return NULL; + + conduit =3D dsa_port_to_conduit(dp); + if (!conduit || conduit->xfrmdev_ops !=3D &airoha_xfrmdev_ops) + return NULL; + + return netdev_priv(conduit); +} + +static struct net_device *airoha_xfrm_dsa_rx_dev(struct airoha_gdm_port *p= ort, + struct sk_buff *skb) +{ + struct metadata_dst *md_dst =3D skb_metadata_dst(skb); + struct dsa_port *cpu_dp =3D port->dev->dsa_ptr; + struct dsa_port *dp; + u32 source_port; + + if (!md_dst || md_dst->type !=3D METADATA_HW_PORT_MUX) + return port->dev; + + if (!cpu_dp || !cpu_dp->dst) + return NULL; + + source_port =3D md_dst->u.port_info.port_id; + list_for_each_entry(dp, &cpu_dp->dst->ports, list) { + if (dp->type !=3D DSA_PORT_TYPE_USER || + dp->index !=3D source_port || dp->cpu_dp !=3D cpu_dp || + dsa_port_to_conduit(dp) !=3D port->dev || !dp->user) + continue; + + return dp->user; + } + + return NULL; +} + +static bool airoha_xfrm_dsa_user_matches_port(struct net_device *user, + struct net_device *conduit) +{ + struct dsa_port *dp; + + if (!dsa_user_dev_check(user)) + return false; + + dp =3D dsa_port_from_netdev(user); + if (IS_ERR(dp)) + return false; + + return dsa_port_to_conduit(dp) =3D=3D conduit; +} +#else +static struct airoha_gdm_port *airoha_xfrm_dsa_dev_port(struct net_device = *dev) +{ + return NULL; +} + +static struct net_device *airoha_xfrm_dsa_rx_dev(struct airoha_gdm_port *p= ort, + struct sk_buff *skb) +{ + return port->dev; +} +#endif + +static struct airoha_gdm_port *airoha_xfrm_dev_port(struct net_device *dev) +{ + struct airoha_gdm_port *port; + + if (dev->xfrmdev_ops !=3D &airoha_xfrmdev_ops) + return NULL; + + port =3D airoha_xfrm_dsa_dev_port(dev); + if (port) + return port; + + return netdev_priv(dev); +} + +static netdev_features_t airoha_xfrm_dev_features(struct net_device *dev) +{ + struct airoha_gdm_port *port =3D airoha_xfrm_dev_port(dev); + + if (!port || !port->xfrm_ipsec) + return 0; + + return airoha_xfrm_ipsec_features(port->xfrm_ipsec); +} + +static struct net_device *airoha_xfrm_rx_dev(struct airoha_gdm_port *port, + struct sk_buff *skb) +{ + if (!netdev_uses_dsa(port->dev)) + return port->dev; + + return airoha_xfrm_dsa_rx_dev(port, skb); +} + +static void airoha_xfrm_state_advance_esn(struct xfrm_state *x) +{ + struct airoha_xfrm_state *state; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (state) + eip93_ipsec_state_advance_esn(state->sa, x); +} + +static int airoha_xfrm_state_add(struct net_device *dev, struct xfrm_state= *x, + struct netlink_ext_ack *extack) +{ + struct airoha_gdm_port *port =3D airoha_xfrm_dev_port(dev); + struct airoha_xfrm_state *state; + int err; + + if (!port) { + NL_SET_ERR_MSG_MOD(extack, "device lacks Airoha ESP offload"); + return -EOPNOTSUPP; + } + + if (!(dev->features & NETIF_F_HW_ESP)) { + NL_SET_ERR_MSG_MOD(extack, + "ESP HW offload is disabled on device"); + return -EOPNOTSUPP; + } + + if (!port->xfrm_ipsec || !eip93_ipsec_available(port->xfrm_ipsec)) { + NL_SET_ERR_MSG_MOD(extack, + "EIP93 packet backend is unavailable"); + return -EOPNOTSUPP; + } + + if (!airoha_xfrm_state_supported(x, extack)) + return -EOPNOTSUPP; + + state =3D kzalloc(sizeof(*state), GFP_KERNEL); + if (!state) + return -ENOMEM; + + state->port =3D port; + err =3D eip93_ipsec_state_add(port->xfrm_ipsec, x, extack, &state->sa); + if (err) { + kfree(state); + return err; + } + + x->xso.offload_handle =3D (unsigned long)state; + atomic_inc(&port->xfrm_state_count); + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_OUT) { + atomic_inc(&port->xfrm_out_state_count); + static_branch_inc(&airoha_xfrm_out_state_key); + } else { + atomic_inc(&port->xfrm_in_state_count); + static_branch_inc(&airoha_xfrm_in_state_key); + } + + return 0; +} + +static void airoha_xfrm_state_delete(struct net_device *dev, + struct xfrm_state *x) +{ + struct airoha_xfrm_state *state; + struct airoha_gdm_port *port; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (!state) + return; + + port =3D state->port; + x->xso.offload_handle =3D 0; + atomic_dec(&port->xfrm_state_count); + if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_OUT) { + atomic_dec(&port->xfrm_out_state_count); + static_branch_dec(&airoha_xfrm_out_state_key); + } else if (x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_IN) { + atomic_dec(&port->xfrm_in_state_count); + static_branch_dec(&airoha_xfrm_in_state_key); + } + + eip93_ipsec_state_delete(state->sa); + kfree(state); +} + +static bool airoha_xfrm_offload_ok(struct sk_buff *skb, struct xfrm_state = *x) +{ + struct net_device *dev =3D skb->dev; + struct airoha_xfrm_state *state; + struct airoha_gdm_port *port; + + if (!dev) + return false; + + port =3D airoha_xfrm_dev_port(dev); + if (!port) + return false; + + if (unlikely(x->xso.dir !=3D XFRM_DEV_OFFLOAD_OUT || + x->xso.type !=3D XFRM_DEV_OFFLOAD_CRYPTO || + !(dev->features & NETIF_F_HW_ESP) || x->xso.dev !=3D dev)) + return false; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (!state || state->port !=3D port) + return false; + + if (unlikely(skb_is_gso(skb))) + return false; + + return true; +} + +/* + * EIP93 packet-out mode creates ESP padding, trailer and ICV. The generic= ESP + * xmit path should reserve tailroom only for plain, non-GSO ESP packets. + */ +static bool airoha_xfrm_esp_tx_hw_trailer(struct sk_buff *skb, + struct xfrm_state *x) +{ + return x->xso.dir =3D=3D XFRM_DEV_OFFLOAD_OUT && + x->xso.type =3D=3D XFRM_DEV_OFFLOAD_CRYPTO && !x->encap && + !skb_is_gso(skb); +} + +static const struct xfrmdev_ops airoha_xfrmdev_ops =3D { + .xdo_dev_state_add =3D airoha_xfrm_state_add, + .xdo_dev_state_delete =3D airoha_xfrm_state_delete, + .xdo_dev_state_free =3D airoha_xfrm_state_delete, + .xdo_dev_offload_ok =3D airoha_xfrm_offload_ok, + .xdo_dev_esp_tx_hw_trailer =3D airoha_xfrm_esp_tx_hw_trailer, + .xdo_dev_state_advance_esn =3D airoha_xfrm_state_advance_esn, +}; + +void airoha_xfrm_build_netdev(struct net_device *dev) +{ + struct airoha_gdm_port *port =3D netdev_priv(dev); + netdev_features_t features; + + atomic_set(&port->xfrm_state_count, 0); + atomic_set(&port->xfrm_out_state_count, 0); + atomic_set(&port->xfrm_in_state_count, 0); + if (airoha_xfrm_prepare_ipsec(dev)) + return; + + features =3D airoha_xfrm_ipsec_features(port->xfrm_ipsec); + if (!(features & NETIF_F_HW_ESP)) { + eip93_ipsec_put(port->xfrm_ipsec); + port->xfrm_ipsec =3D NULL; + return; + } + + dev->xfrmdev_ops =3D &airoha_xfrmdev_ops; + dev->hw_features |=3D features; + dev->hw_enc_features |=3D features; + dev->gso_partial_features |=3D features & NETIF_F_GSO_ESP; +} + +void airoha_xfrm_teardown_netdev(struct net_device *dev) +{ + struct airoha_gdm_port *port =3D netdev_priv(dev); + + if (port->xfrm_ipsec) { + eip93_ipsec_put(port->xfrm_ipsec); + port->xfrm_ipsec =3D NULL; + } +} + +/* Airoha TX checksum/GSO offloads run after EIP93 has encrypted the skb, = so + * they cannot operate on plaintext ESP payloads or build per-segment ESP = data. + */ +netdev_features_t airoha_xfrm_fix_features(struct net_device *dev, + netdev_features_t features) +{ + netdev_features_t supported =3D airoha_xfrm_dev_features(dev); + netdev_features_t unsupported =3D AIROHA_XFRM_FEATURES & ~supported; + + if (features & unsupported) + features &=3D ~unsupported; + + if (!(features & NETIF_F_HW_ESP)) + features &=3D ~(NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP); + + return features; +} + +int airoha_xfrm_set_features(struct net_device *dev, netdev_features_t fea= tures) +{ + netdev_features_t changed =3D (dev->features ^ features) & + AIROHA_XFRM_FEATURES; + netdev_features_t requested =3D features & AIROHA_XFRM_FEATURES; + struct airoha_gdm_port *port =3D netdev_priv(dev); + netdev_features_t supported; + int err; + + if (!changed) + return 0; + + if (requested & NETIF_F_HW_ESP) { + err =3D airoha_xfrm_prepare_ipsec(dev); + if (err) + return err; + } + + supported =3D airoha_xfrm_dev_features(dev); + if (requested & ~supported) + return -EOPNOTSUPP; + + if (atomic_read(&port->xfrm_state_count)) { + netdev_err(dev, "cannot change ESP features with active SAs\n"); + return -EBUSY; + } + + if (!(features & NETIF_F_HW_ESP)) + netdev_info(dev, "ESP HW offload disabled\n"); + + return 0; +} + +struct airoha_xfrm_rx_info { + unsigned short family; + int encap_type; + int esp_offset; + int packet_len; + __be32 spi; + __be32 seq; +}; + +struct airoha_xfrm_rx_ctx { + struct sk_buff *skb; + struct net_device *dev; +}; + +static bool airoha_xfrm_parse_rx_ipv4(struct sk_buff *skb, + struct airoha_xfrm_rx_info *info) +{ + struct ip_esp_hdr *esph; + struct iphdr *iph; + int packet_len; + int iphlen; + + if (!pskb_may_pull(skb, sizeof(*iph))) + return false; + + iph =3D ip_hdr(skb); + if (iph->version !=3D 4) + return false; + + iphlen =3D iph->ihl * 4; + if (iphlen < sizeof(*iph) || !pskb_may_pull(skb, iphlen)) + return false; + + if (ip_is_fragment(iph)) + return false; + + packet_len =3D ntohs(iph->tot_len); + if (packet_len < iphlen || packet_len > skb->len) + return false; + + switch (iph->protocol) { + case IPPROTO_ESP: + info->encap_type =3D 0; + info->esp_offset =3D iphlen; + info->packet_len =3D packet_len; + break; + case IPPROTO_UDP: { + struct udphdr *uh; + int udp_len; + __be32 marker; + + if (!pskb_may_pull(skb, iphlen + sizeof(*uh) + sizeof(*esph))) + return false; + + uh =3D (struct udphdr *)(skb->data + iphlen); + udp_len =3D ntohs(uh->len); + if (udp_len <=3D sizeof(*uh) + sizeof(*esph) || + iphlen + udp_len > packet_len) + return false; + + memcpy(&marker, skb->data + iphlen + sizeof(*uh), + sizeof(marker)); + if (!marker) + return false; + + info->encap_type =3D UDP_ENCAP_ESPINUDP; + info->esp_offset =3D iphlen + sizeof(*uh); + info->packet_len =3D iphlen + udp_len; + break; + } + default: + return false; + } + + if (info->esp_offset + sizeof(*esph) > info->packet_len || + !pskb_may_pull(skb, info->esp_offset + sizeof(*esph))) + return false; + + esph =3D (struct ip_esp_hdr *)(skb->data + info->esp_offset); + info->family =3D AF_INET; + info->spi =3D esph->spi; + info->seq =3D esph->seq_no; + + return !!info->spi; +} + +#if IS_ENABLED(CONFIG_IPV6) +static bool airoha_xfrm_parse_rx_ipv6(struct sk_buff *skb, + struct airoha_xfrm_rx_info *info) +{ + struct ip_esp_hdr *esph; + struct ipv6hdr *ip6h; + __be16 frag_off; + int packet_len; + int offset; + u8 nexthdr; + + if (!pskb_may_pull(skb, sizeof(*ip6h))) + return false; + + ip6h =3D ipv6_hdr(skb); + if (ip6h->version !=3D 6) + return false; + + if (!ip6h->payload_len) + return false; + + packet_len =3D sizeof(*ip6h) + ntohs(ip6h->payload_len); + if (packet_len < sizeof(*ip6h) || packet_len > skb->len) + return false; + + nexthdr =3D ip6h->nexthdr; + offset =3D ipv6_skip_exthdr(skb, sizeof(*ip6h), &nexthdr, &frag_off); + if (offset < 0 || frag_off) + return false; + + switch (nexthdr) { + case NEXTHDR_ESP: + info->encap_type =3D 0; + info->esp_offset =3D offset; + info->packet_len =3D packet_len; + break; + case NEXTHDR_UDP: { + struct udphdr *uh; + int udp_len; + __be32 marker; + + if (!pskb_may_pull(skb, offset + sizeof(*uh) + sizeof(*esph))) + return false; + + uh =3D (struct udphdr *)(skb->data + offset); + udp_len =3D ntohs(uh->len); + if (udp_len <=3D sizeof(*uh) + sizeof(*esph) || + offset + udp_len > packet_len) + return false; + + memcpy(&marker, skb->data + offset + sizeof(*uh), + sizeof(marker)); + if (!marker) + return false; + + info->encap_type =3D UDP_ENCAP_ESPINUDP; + info->esp_offset =3D offset + sizeof(*uh); + info->packet_len =3D offset + udp_len; + break; + } + default: + return false; + } + + if (info->esp_offset + sizeof(*esph) > info->packet_len || + !pskb_may_pull(skb, info->esp_offset + sizeof(*esph))) + return false; + + esph =3D (struct ip_esp_hdr *)(skb->data + info->esp_offset); + info->family =3D AF_INET6; + info->spi =3D esph->spi; + info->seq =3D esph->seq_no; + + return !!info->spi; +} +#else +static bool airoha_xfrm_parse_rx_ipv6(struct sk_buff *skb, + struct airoha_xfrm_rx_info *info) +{ + return false; +} +#endif + +static bool airoha_xfrm_parse_rx_skb(struct sk_buff *skb, + struct airoha_xfrm_rx_info *info) +{ + switch (skb->protocol) { + case htons(ETH_P_IP): + return airoha_xfrm_parse_rx_ipv4(skb, info); + case htons(ETH_P_IPV6): + return airoha_xfrm_parse_rx_ipv6(skb, info); + default: + return false; + } +} + +static struct xfrm_state * +airoha_xfrm_rx_state_lookup(struct airoha_gdm_port *port, struct sk_buff *= skb, + const struct airoha_xfrm_rx_info *info) +{ + struct airoha_xfrm_state *state; + xfrm_address_t daddr =3D {}; + struct net_device *dev; + struct xfrm_state *x; + + dev =3D airoha_xfrm_rx_dev(port, skb); + if (!dev) + return NULL; + + switch (info->family) { + case AF_INET: + daddr.a4 =3D ip_hdr(skb)->daddr; + break; + case AF_INET6: + daddr.in6 =3D ipv6_hdr(skb)->daddr; + break; + default: + return NULL; + } + + x =3D xfrm_input_state_lookup(dev_net(dev), skb->mark, &daddr, info->spi, + IPPROTO_ESP, info->family); + if (!x) + return NULL; + + if (x->dir && x->dir !=3D XFRM_SA_DIR_IN) + goto err_put; + + if (x->xso.dir !=3D XFRM_DEV_OFFLOAD_IN || + x->xso.type !=3D XFRM_DEV_OFFLOAD_CRYPTO || x->xso.dev !=3D dev || + !(dev->features & NETIF_F_HW_ESP) || !x->type_offload || + !x->type_offload->input_tail) + goto err_put; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (!state || state->port !=3D port) + goto err_put; + + if ((x->encap ? x->encap->encap_type : 0) !=3D info->encap_type) + goto err_put; + + return x; + +err_put: + xfrm_state_put(x); + return NULL; +} + +static u32 airoha_xfrm_rx_status(int err, struct xfrm_state *x) +{ + if (!err) + return CRYPTO_SUCCESS; + + if (err =3D=3D -EBADMSG) { + if (x->props.mode =3D=3D XFRM_MODE_TUNNEL) + return CRYPTO_TUNNEL_ESP_AUTH_FAILED; + + return CRYPTO_TRANSPORT_ESP_AUTH_FAILED; + } + + if (err =3D=3D -EINVAL) + return CRYPTO_INVALID_PACKET_SYNTAX; + + return CRYPTO_GENERIC_ERROR; +} + +static int airoha_xfrm_rx_apply_result(struct sk_buff *skb, + struct xfrm_state *x, + struct eip93_ipsec_result result) +{ + struct xfrm_offload *xo =3D xfrm_offload(skb); + + if (!x || !result.packet_len || result.packet_len > skb->len || !xo) + return -EINVAL; + + /* + * EIP93 inbound ESP mode removes the ESP pad/trailer/ICV and reports + * the decapsulated outer packet length plus the recovered next-header. + */ + xo->proto =3D result.nexthdr; + xo->flags |=3D XFRM_ESP_NO_TRAILER; + if (pskb_trim(skb, result.packet_len)) + return -EINVAL; + + if (x->props.family =3D=3D AF_INET) { + ip_hdr(skb)->tot_len =3D htons(skb->len); + ip_send_check(ip_hdr(skb)); + } else if (x->props.family =3D=3D AF_INET6) { + int len =3D skb->len - skb_network_offset(skb) - + sizeof(struct ipv6hdr); + + if (len < 0) + return -EINVAL; + + ipv6_hdr(skb)->payload_len =3D len > IPV6_MAXPLEN ? 0 : + htons(len); + } + + return 0; +} + +static void airoha_xfrm_rx_free_ctx(struct airoha_xfrm_rx_ctx *ctx) +{ + kfree(ctx); +} + +static void airoha_xfrm_rx_finish(void *data, int err, + struct eip93_ipsec_result result) +{ + struct airoha_xfrm_rx_ctx *ctx =3D data; + struct net_device *dev =3D ctx->dev; + struct sk_buff *skb =3D ctx->skb; + struct xfrm_offload *xo; + struct xfrm_state *x; + + x =3D xfrm_input_state(skb); + xo =3D xfrm_offload(skb); + if (!err) + err =3D airoha_xfrm_rx_apply_result(skb, x, result); + if (xo) { + xo->flags |=3D CRYPTO_DONE; + xo->status =3D airoha_xfrm_rx_status(err, x); + } + + airoha_xfrm_rx_free_ctx(ctx); + netif_receive_skb(skb); + dev_put(dev); +} + +static bool airoha_xfrm_tx_esp_offset(struct sk_buff *skb, struct xfrm_sta= te *x, + unsigned int *esp_offset) +{ + u8 *esph =3D (u8 *)ip_esp_hdr(skb); + + if (x->encap) + esph +=3D sizeof(struct udphdr); + + if (esph < skb->data || + esph + sizeof(struct ip_esp_hdr) > skb_tail_pointer(skb)) + return false; + + *esp_offset =3D esph - skb->data; + + return true; +} + +static void airoha_xfrm_tx_update_outer_len(struct sk_buff *skb) +{ + struct iphdr *iph =3D ip_hdr(skb); + + if (iph->version =3D=3D 4) { + iph->tot_len =3D htons(skb->len - skb_network_offset(skb)); + ip_send_check(iph); + } else if (iph->version =3D=3D 6) { + int len =3D skb->len - skb_network_offset(skb) - + sizeof(struct ipv6hdr); + + if (len < 0) + return; + + ipv6_hdr(skb)->payload_len =3D len > IPV6_MAXPLEN ? 0 : + htons(len); + } +} + +static void airoha_xfrm_tx_udp6_csum(struct sk_buff *skb, + struct xfrm_state *x) +{ +#if IS_ENABLED(CONFIG_IPV6) + struct udphdr *uh; + struct ipv6hdr *ip6h; + unsigned int offset; + __wsum csum; + int len; + + if (x->props.family !=3D AF_INET6 || !x->encap || + x->encap->encap_type !=3D UDP_ENCAP_ESPINUDP) + return; + + offset =3D skb_transport_offset(skb); + if (offset + sizeof(*uh) > skb->len) + return; + + uh =3D udp_hdr(skb); + ip6h =3D ipv6_hdr(skb); + len =3D ntohs(uh->len); + if (len < sizeof(*uh) || len > skb->len - offset) + return; + + uh->check =3D 0; + csum =3D skb_checksum(skb, offset, len, 0); + uh->check =3D csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr, len, + IPPROTO_UDP, csum); + if (!uh->check) + uh->check =3D CSUM_MANGLED_0; + #endif +} + +static int airoha_xfrm_tx_apply_result(struct sk_buff *skb, + struct xfrm_state *x, + struct eip93_ipsec_result result) +{ + unsigned int current_esp_len; + unsigned int esp_offset; + unsigned int new_len; + + if (!result.packet_len || + !airoha_xfrm_tx_esp_offset(skb, x, &esp_offset)) + return -EINVAL; + + current_esp_len =3D skb->len - esp_offset; + if (result.packet_len =3D=3D current_esp_len) + return 0; + + new_len =3D esp_offset + result.packet_len; + if (new_len < esp_offset) + return -EINVAL; + + /* + * EIP93 outbound ESP mode reports the generated ESP packet length. + * Reflect it in skb->len before the packet resumes into the Ethernet + * TX path, because generic ESP left hardware-generated trailer bytes + * outside skb->len. + */ + if (new_len > skb->len) { + unsigned int delta =3D new_len - skb->len; + + if (delta > skb_tailroom(skb)) + return -ENOMEM; + skb_put(skb, delta); + + return 0; + } + + return pskb_trim(skb, new_len); +} + +bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb) +{ + struct airoha_xfrm_rx_info info; + struct airoha_xfrm_state *state; + struct airoha_xfrm_rx_ctx *ctx; + struct sk_buff *trailer; + struct xfrm_offload *xo; + struct xfrm_state *x; + struct sec_path *sp; + int err; + u32 mark =3D skb->mark; + + if (!airoha_xfrm_parse_rx_skb(skb, &info)) + return false; + + x =3D airoha_xfrm_rx_state_lookup(port, skb, &info); + if (!x) + return false; + + sp =3D secpath_set(skb); + if (!sp) + goto err_put_state; + + if (sp->len =3D=3D XFRM_MAX_DEPTH) { + secpath_reset(skb); + goto err_put_state; + } + + skb->mark =3D xfrm_smark_get(mark, x); + sp->xvec[sp->len++] =3D x; + sp->olen++; + XFRM_SKB_CB(skb)->seq.input.low =3D info.seq; + XFRM_SKB_CB(skb)->seq.input.hi =3D htonl(xfrm_replay_seqhi(x, info.seq)); + XFRM_SPI_SKB_CB(skb)->family =3D info.family; + XFRM_SPI_SKB_CB(skb)->seq =3D info.seq; + if (info.family =3D=3D AF_INET) { + XFRM_SPI_SKB_CB(skb)->daddroff =3D offsetof(struct iphdr, daddr); + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 =3D NULL; + } else { + XFRM_SPI_SKB_CB(skb)->daddroff =3D + offsetof(struct ipv6hdr, daddr); + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6 =3D NULL; + } + + xo =3D xfrm_offload(skb); + if (!xo) + goto err_reset; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (!state || state->port !=3D port) + goto err_reset; + + if (skb_cloned(skb) || skb_is_nonlinear(skb)) { + err =3D skb_cow_data(skb, 0, &trailer); + if (err < 0) + goto err_reset; + + if (skb_is_nonlinear(skb)) { + err =3D skb_linearize(skb); + if (err) + goto err_reset; + } + } + + ctx =3D kmalloc(sizeof(*ctx), GFP_ATOMIC); + if (!ctx) + goto err_reset; + + if (!skb->dev) + goto err_free_ctx; + + ctx->skb =3D skb; + ctx->dev =3D skb->dev; + skb->ip_summed =3D CHECKSUM_NONE; + + dev_hold(ctx->dev); + err =3D eip93_ipsec_receive(state->sa, skb, info.packet_len, + airoha_xfrm_rx_finish, ctx); + if (err =3D=3D -EINPROGRESS) + return true; + + dev_put(ctx->dev); + airoha_xfrm_rx_free_ctx(ctx); + skb->mark =3D mark; + secpath_reset(skb); + + return false; + +err_free_ctx: + airoha_xfrm_rx_free_ctx(ctx); +err_reset: + skb->mark =3D mark; + secpath_reset(skb); + return false; + +err_put_state: + xfrm_state_put(x); + return false; +} + +static void airoha_xfrm_tx_done(void *data, int err, + struct eip93_ipsec_result result) +{ + struct sk_buff *skb =3D data; + struct xfrm_offload *xo =3D xfrm_offload(skb); + struct sec_path *sp =3D skb_sec_path(skb); + struct xfrm_state *x; + + if (!xo || !sp || !sp->len) { + kfree_skb(skb); + return; + } + + x =3D sp->xvec[sp->len - 1]; + if (!err) + err =3D airoha_xfrm_tx_apply_result(skb, x, result); + if (err) { + XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTSTATEPROTOERROR); + kfree_skb(skb); + return; + } + + airoha_xfrm_tx_update_outer_len(skb); + airoha_xfrm_tx_udp6_csum(skb, x); + xo->flags |=3D CRYPTO_DONE; + xo->status =3D CRYPTO_SUCCESS; + skb_push(skb, skb->data - skb_mac_header(skb)); + secpath_reset(skb); + xfrm_dev_resume(skb); +} + +int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *= skb) +{ + struct xfrm_offload *xo =3D xfrm_offload(skb); + struct airoha_xfrm_state *state; + struct net_device *dev; + struct xfrm_state *x; + struct sec_path *sp; + struct ip_esp_hdr *esph; + struct sk_buff *trailer; + unsigned int esp_offset; + unsigned int tailen; + int err; + + if (!xo || !(xo->flags & XFRM_XMIT) || (xo->flags & CRYPTO_DONE)) + return 0; + + sp =3D skb_sec_path(skb); + if (!sp || !sp->len) + return -EINVAL; + + x =3D sp->xvec[sp->len - 1]; + dev =3D x->xso.dev; + if (unlikely(x->xso.dir !=3D XFRM_DEV_OFFLOAD_OUT || + x->xso.type !=3D XFRM_DEV_OFFLOAD_CRYPTO || !dev || + !(dev->features & NETIF_F_HW_ESP))) + return -EOPNOTSUPP; + + state =3D (struct airoha_xfrm_state *)x->xso.offload_handle; + if (!state || state->port !=3D port) + return -EOPNOTSUPP; + + if (unlikely(skb_is_gso(skb))) + return -EOPNOTSUPP; + + if (unlikely(skb->ip_summed =3D=3D CHECKSUM_PARTIAL)) { + err =3D skb_checksum_help(skb); + if (err) + return err; + } + + tailen =3D xo->esp_tx_tailen; + if (skb_cloned(skb) || skb_is_nonlinear(skb)) { + err =3D skb_cow_data(skb, tailen, &trailer); + if (err < 0) + return err; + + if (skb_is_nonlinear(skb)) { + err =3D skb_linearize(skb); + if (err) + return err; + } + } + /* + * Generic ESP reserves this tailroom before the skb reaches us. Keep a + * small guard here because COW/linearization can replace the skb head. + */ + if (tailen && skb_tailroom(skb) < tailen) { + err =3D pskb_expand_head(skb, 0, tailen - skb_tailroom(skb), + GFP_ATOMIC); + if (err) + return err; + } + + if (!airoha_xfrm_tx_esp_offset(skb, x, &esp_offset)) + return -EINVAL; + + esph =3D (struct ip_esp_hdr *)(skb->data + esp_offset); + esph->seq_no =3D htonl(xo->seq.low); + + return eip93_ipsec_xmit(state->sa, skb, esp_offset, airoha_xfrm_tx_done, + skb); +} + +static void airoha_xfrm_flush_dev(struct net_device *dev) +{ + xfrm_dev_state_flush(dev_net(dev), dev, true); + xfrm_dev_policy_flush(dev_net(dev), dev, true); +} + +static void airoha_xfrm_link_change(struct net_device *dev) +{ + struct airoha_gdm_port *port =3D airoha_xfrm_dev_port(dev); + + if (!port || !(dev->hw_features & NETIF_F_HW_ESP) || + !atomic_read(&port->xfrm_state_count)) + return; + + netdev_dbg(dev, "carrier %s, preserving ESP HW offload SAs\n", + netif_carrier_ok(dev) ? "up" : "down"); +} + +#if IS_ENABLED(CONFIG_NET_DSA) +static void airoha_xfrm_dsa_attach_user(struct net_device *conduit, + struct net_device *user) +{ + netdev_features_t features =3D airoha_xfrm_dev_features(conduit); + + if (conduit->xfrmdev_ops !=3D &airoha_xfrmdev_ops || + !airoha_xfrm_dsa_user_matches_port(user, conduit)) + return; + + if (!(features & NETIF_F_HW_ESP)) + return; + + if (user->xfrmdev_ops && user->xfrmdev_ops !=3D &airoha_xfrmdev_ops) { + netdev_dbg(conduit, + "DSA user %s already has XFRM offload ops\n", + user->name); + return; + } + + user->xfrmdev_ops =3D &airoha_xfrmdev_ops; + user->hw_features |=3D features; + user->hw_enc_features |=3D features; + user->gso_partial_features |=3D features & NETIF_F_GSO_ESP; + netdev_dbg(user, "ESP HW offload available via %s\n", conduit->name); +} + +static void airoha_xfrm_dsa_detach_user(struct net_device *user) +{ + struct airoha_gdm_port *port; + bool active =3D false; + bool enabled; + + if (user->xfrmdev_ops !=3D &airoha_xfrmdev_ops || + !dsa_user_dev_check(user)) + return; + + enabled =3D user->features & NETIF_F_HW_ESP; + port =3D airoha_xfrm_dsa_dev_port(user); + if (port) + active =3D atomic_read(&port->xfrm_state_count); + + if (active) { + netdev_warn(user, "DSA detach with active ESP SAs, flushing\n"); + airoha_xfrm_flush_dev(user); + } + + user->wanted_features &=3D ~AIROHA_XFRM_FEATURES; + user->features &=3D ~AIROHA_XFRM_FEATURES; + user->hw_features &=3D ~AIROHA_XFRM_FEATURES; + user->hw_enc_features &=3D ~AIROHA_XFRM_FEATURES; + user->gso_partial_features &=3D ~NETIF_F_GSO_ESP; + user->xfrmdev_ops =3D NULL; + + if (active || enabled) + netdev_features_change(user); +} + +static void airoha_xfrm_dsa_feature_change(struct net_device *dev) +{ + struct airoha_gdm_port *port; + + if (dev->xfrmdev_ops !=3D &airoha_xfrmdev_ops || + !dsa_user_dev_check(dev) || (dev->features & NETIF_F_HW_ESP)) + return; + + port =3D airoha_xfrm_dsa_dev_port(dev); + if (port && atomic_read(&port->xfrm_state_count)) { + netdev_warn(dev, "DSA feature lost ESP SAs, flushing\n"); + airoha_xfrm_flush_dev(dev); + } +} +#endif + +static int airoha_xfrm_netdevice_event(struct notifier_block *nb, + unsigned long event, void *ptr) +{ + struct net_device *dev =3D netdev_notifier_info_to_dev(ptr); + + switch (event) { + case NETDEV_CHANGE: + airoha_xfrm_link_change(dev); + break; +#if IS_ENABLED(CONFIG_NET_DSA) + case NETDEV_CHANGEUPPER: { + struct netdev_notifier_changeupper_info *info =3D ptr; + + if (info->linking) + airoha_xfrm_dsa_attach_user(dev, info->upper_dev); + else + airoha_xfrm_dsa_detach_user(info->upper_dev); + break; + } + case NETDEV_FEAT_CHANGE: + airoha_xfrm_dsa_feature_change(dev); + break; + case NETDEV_UNREGISTER: + airoha_xfrm_dsa_detach_user(dev); + break; +#endif + default: + break; + } + + return NOTIFY_DONE; +} + +static struct notifier_block airoha_xfrm_netdev_notifier =3D { + .notifier_call =3D airoha_xfrm_netdevice_event, +}; + +static int airoha_xfrm_register_netdev_notifier(void) +{ + return register_netdevice_notifier(&airoha_xfrm_netdev_notifier); +} + +static void airoha_xfrm_unregister_netdev_notifier(void) +{ + unregister_netdevice_notifier(&airoha_xfrm_netdev_notifier); +} + +static void airoha_xfrm_drop_dev(struct net_device *dev, const char *reaso= n) +{ + struct airoha_gdm_port *port =3D airoha_xfrm_dev_port(dev); + bool advertised =3D dev->hw_features & AIROHA_XFRM_FEATURES; + bool enabled =3D dev->features & NETIF_F_HW_ESP; + bool active =3D false; + + if (port) + active =3D atomic_read(&port->xfrm_state_count); + + if (active) { + netdev_warn(dev, "%s, flushing ESP HW offload SAs\n", reason); + airoha_xfrm_flush_dev(dev); + } + + dev->wanted_features &=3D ~AIROHA_XFRM_FEATURES; + dev->features &=3D ~AIROHA_XFRM_FEATURES; + dev->hw_features &=3D ~AIROHA_XFRM_FEATURES; + dev->hw_enc_features &=3D ~AIROHA_XFRM_FEATURES; + dev->gso_partial_features &=3D ~NETIF_F_GSO_ESP; + + if (active || enabled || advertised) + netdev_features_change(dev); +} + +static void airoha_xfrm_drop_ipsec(struct eip93_ipsec *ipsec, + const char *reason) +{ + struct net_device *dev; + struct net *net; + + rtnl_lock(); + for_each_net(net) { + for_each_netdev(net, dev) { + struct airoha_gdm_port *port; + + port =3D airoha_xfrm_dev_port(dev); + if (!port || port->xfrm_ipsec !=3D ipsec) + continue; + + airoha_xfrm_drop_dev(dev, reason); + } + } + + for_each_net(net) { + for_each_netdev(net, dev) { + struct airoha_gdm_port *port; + + if (dev->xfrmdev_ops !=3D &airoha_xfrmdev_ops) + continue; + + if (airoha_xfrm_dsa_dev_port(dev)) + continue; + + port =3D netdev_priv(dev); + if (dev =3D=3D port->dev && port->xfrm_ipsec =3D=3D ipsec) { + eip93_ipsec_put(port->xfrm_ipsec); + port->xfrm_ipsec =3D NULL; + } + } + } + + for_each_net(net) { + for_each_netdev(net, dev) { + if (dev->xfrmdev_ops =3D=3D &airoha_xfrmdev_ops && + !(dev->hw_features & NETIF_F_HW_ESP)) + dev->xfrmdev_ops =3D NULL; + } + } + rtnl_unlock(); +} + +static int airoha_xfrm_ipsec_event(struct notifier_block *nb, + unsigned long event, void *ptr) +{ + switch (event) { + case EIP93_IPSEC_EVENT_REMOVE: + airoha_xfrm_drop_ipsec(ptr, "EIP93 provider removed"); + break; + case EIP93_IPSEC_EVENT_RESET: + airoha_xfrm_drop_ipsec(ptr, "EIP93 provider reset"); + break; + case EIP93_IPSEC_EVENT_DMA_ERROR: + airoha_xfrm_drop_ipsec(ptr, "EIP93 DMA error"); + break; + case EIP93_IPSEC_EVENT_CAPABILITY_LOSS: + airoha_xfrm_drop_ipsec(ptr, "EIP93 capability loss"); + break; + default: + break; + } + + return NOTIFY_DONE; +} + +static struct notifier_block airoha_xfrm_ipsec_notifier =3D { + .notifier_call =3D airoha_xfrm_ipsec_event, +}; + +int airoha_xfrm_register_notifier(void) +{ + int err; + + err =3D airoha_xfrm_register_netdev_notifier(); + if (err) + return err; + + err =3D eip93_ipsec_register_notifier(&airoha_xfrm_ipsec_notifier); + if (err) + airoha_xfrm_unregister_netdev_notifier(); + + return err; +} + +void airoha_xfrm_unregister_notifier(void) +{ + eip93_ipsec_unregister_notifier(&airoha_xfrm_ipsec_notifier); + airoha_xfrm_unregister_netdev_notifier(); +} +#else +void airoha_xfrm_build_netdev(struct net_device *dev) +{ +} + +void airoha_xfrm_teardown_netdev(struct net_device *dev) +{ +} + +netdev_features_t airoha_xfrm_fix_features(struct net_device *dev, + netdev_features_t features) +{ + return features & ~(NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP); +} + +int airoha_xfrm_set_features(struct net_device *dev, netdev_features_t fea= tures) +{ + return 0; +} + +bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb) +{ + return false; +} + +int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *= skb) +{ + return 0; +} + +int airoha_xfrm_register_notifier(void) +{ + return 0; +} + +void airoha_xfrm_unregister_notifier(void) +{ +} + +#endif --=20 2.53.0