From nobody Sun Feb 8 13:40:05 2026 Received: from OS0P286CU011.outbound.protection.outlook.com (mail-japanwestazon11010054.outbound.protection.outlook.com [52.101.228.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E20AA2D8DB7 for ; Thu, 23 Oct 2025 07:21:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.228.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761204076; cv=fail; b=rWZSiOd5+YHYwa+XJPEjFE1wfT2uSBIy8sH0aRn5/dY3gf47BkbON0jYUvzj+9Wwy3y8IudNU69rDUFa2KMvOuu4H6fHNbTSdFww2bwuiCSIU4IBVR5RI1M2peCBumS07tX9bNZ9XQeVECDr7M1DUgiAzfEF8P4tOlE6lBXKl34= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761204076; c=relaxed/simple; bh=iInewF/Nn96A9u8toqFdXz3dWKznFWoZrAdzcsdLgpc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=q53J7I1EBGOAxN99M6B06goWaCPozLNx7fMypULbRDWT/12Q34HzFXhHw4VKdV67K9Y5JXyN4Bh/98XNji+epTtIbkjWje3KR6KWvXQr0fSFZLslYyXXa8u5pmBJLSahWRW1edH8dxD1o3Fa5ZgBaTL+AGAEzd9+6jShPQvXKHM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp; spf=pass smtp.mailfrom=valinux.co.jp; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b=RQpecUQy; arc=fail smtp.client-ip=52.101.228.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b="RQpecUQy" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=IY3/mMTJuAcryPVdHWZlcbjAfSZH4mWtJf1dEdN1xBw34iPfxwFEFH9VSB9CnHFKGyGMzaJbC1g3td05EuJScWYqhYfi4J3hWrh7WerXKpiWvJ0AByCznaa/ART1OLIx/4+QVjquebOJSs9Thyc10lo2mnzIi9L59bSga26j7hfRn0UWk0F/o4Sepm5KqeYVyNntxeH06alZ0Zm2yiQjjfc4TPXVhXpr/Qsv57KGyVqnbRXtHNMtkczDi3SKzjzAEGavYbPSnW48/sE/B+1ZR+TLThGJ9f0OxWLf/6/EwI0dX91pKPT1pyIpJJICbmVXcTJ1zorw56eEeFDOxW7GXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qUCbXhyPzK9Ww9Pxs4mmsyS1S+y8p4UOzwnj9IMV2FU=; b=wnj2htDSHCUpA3n6ZAgX52VdJRpPDWyxVyoXeSNJOFjG1F2nbtZ/LigQuowtjyuYZrUWFtAwjiZ6CKun5wMH4BXTcnerUbc3LxjgIEcx4zHoWEHyx5/gkGUwNQ8PRciKYJQD+zPPc3PmvHh3nfg1zAVqaXEt2zf2b6WIoKhb0zCoeJtZstAeJv4c4eAf0600VyuVxVcpwGc7pqedUDb5d9Ptqt+gyxMLIXE2h+7I3+MiRytGgxkrWX5clTwuL8akQKH0rPj34Iwp2PoUc4QvyC/6W1501XNGP8+hCBYDL386vpx/C/W/GidAG9CfRQBkUxtJoMnyAeHdDsJOW9wONw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=valinux.co.jp; dmarc=pass action=none header.from=valinux.co.jp; dkim=pass header.d=valinux.co.jp; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=valinux.co.jp; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qUCbXhyPzK9Ww9Pxs4mmsyS1S+y8p4UOzwnj9IMV2FU=; b=RQpecUQykuSb4Ab9ynXJ85dypnjZ6fHfODZpmRMt1NIQb8/fAZrvq8HJO3tEEj9uvo85J+JigQtqPsgLUIQ7asPXxfMCy5cPLlO8qfYkf+/+tz6rqt+ZDNuJikVLHnzbye/CwFrEfMHA9WlxoaiMMMGawaht9KxBxDPv6PsnUrY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=valinux.co.jp; Received: from OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM (2603:1096:604:10d::7) by TYRP286MB4555.JPNP286.PROD.OUTLOOK.COM (2603:1096:405:1b0::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9253.13; Thu, 23 Oct 2025 07:21:11 +0000 Received: from OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM ([fe80::80f1:db56:4a11:3f7a]) by OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM ([fe80::80f1:db56:4a11:3f7a%5]) with mapi id 15.20.9253.011; Thu, 23 Oct 2025 07:21:11 +0000 From: Koichiro Den To: ntb@lists.linux.dev, linux-kernel@vger.kernel.org Cc: jdmason@kudzu.us, dave.jiang@intel.com, allenbh@gmail.com Subject: [PATCH 1/2] NTB: ntb_transport: Remove unused 'retries' field from ntb_queue_entry Date: Thu, 23 Oct 2025 16:21:04 +0900 Message-ID: <20251023072105.901707-2-den@valinux.co.jp> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251023072105.901707-1-den@valinux.co.jp> References: <20251023072105.901707-1-den@valinux.co.jp> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: TYCPR01CA0095.jpnprd01.prod.outlook.com (2603:1096:405:3::35) To OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM (2603:1096:604:10d::7) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: OS3P286MB0979:EE_|TYRP286MB4555:EE_ X-MS-Office365-Filtering-Correlation-Id: c8d7ab18-93f8-48ff-566d-08de1204be08 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|10070799003|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?xQpvoqSE9D3TLhys2MU45R1MRg6cCUbMo7KaDK8ConcuTcMibIjfP0DZQqHe?= =?us-ascii?Q?vyOS0P9zVlsUDmmvdPWxmCroqyYQWH2w3ILn1iLcQnkaf9exdEtxsb1BVeCn?= =?us-ascii?Q?fF+DCRurqxTw027q9gkIcReLc+2ISNx+na3nsLRZ3ZHa8N1I5xcVkmjvsmbO?= =?us-ascii?Q?Kt/d2yCekq0SOj6osvIH1EjlXL53YPBXwMrrQUxCYA6fjMJkJ2MptEDyJJ4Y?= =?us-ascii?Q?4tI2ph4klQltnXr+7b7MoGqLCsvobNdCtwIHd7qpRwKAcQOej4DYDwYJvXMJ?= =?us-ascii?Q?JlrLKiB8lzBbAtqLSlP/GlCXjIvApPG4n8kDjsrnWaylYSETI/zbJU81B4Ls?= =?us-ascii?Q?2pKqDhIbXmixghcv+U75IxDqB1MH1frUUV1SCWFsoQ7eGc9lrcC7DCMUb+5D?= =?us-ascii?Q?oOQXvHFZYWGDUFuJoZVU2QNu6/e/g3u0E8BZMhztBXR7p+H17z7hYQz3V3yK?= =?us-ascii?Q?BIVMZIQNOJbTvC/QnY1XrDjnWh4yAIShPp+rm5gqx4NwMJ1iM9SYv6qkn1Mc?= =?us-ascii?Q?M4sBQ3K9LeGFDDsvsYb8ou62rlsH5lyMx27hQKtiDWVkR/71C9Uc0HV0JgAW?= =?us-ascii?Q?nuqj8C6wRasEF8wtkE/V3Z1wF/7JZwl7CW8v34lUmy6KLhFldlE7EKZSDPWT?= =?us-ascii?Q?JkFUzbTY0gBydTsKcOoJqlSQn0UAb+wu2xmuTJsYEHcWl92Xa/IA4uxsFoOW?= =?us-ascii?Q?JdN/SxHZ6iyQ/Mu8ScvVz7YZg5CYOG4ZfFJOlvqNMIwd/qic4zJcsCqsim+m?= =?us-ascii?Q?jNeDqA5kMxBrXP1GW7vn3KKo3jBhn0XJ9gXF5KAu7EuB1FV0SsLDWFEm9lU5?= =?us-ascii?Q?/u8Nobfop6xH0FELPftxhWnTijPy3my7mREmV+cM/Uf41Zcy3bKNQxNEy5P5?= =?us-ascii?Q?6GQ6H1vVEVxHCCqCMtY8XxiSCNgdaZckeFuintnIkicBzQ6SNEa2d5C/NyPf?= =?us-ascii?Q?hQzcSZMetc9G5+JidoNWcUtWHROEfrf1gvU2CJfJTGy+zIMivDb9/weOKkr/?= =?us-ascii?Q?WTvJ8u9sYTSEDFckhVXljZtGBVx0okWRa+doIAtDwz+CWNxkXO6FP/r9gwHe?= =?us-ascii?Q?+xDEdfFJImUxzda0r5eivGE23C6a6OtX+daI4/TRGM/VWsnPMZdqhUFmmXHq?= =?us-ascii?Q?LbFpeCLCtZ7o5tZ3j62B8HFP2KYSca28TR5QZNBf4t7hERgMkHjieHiJaBWP?= =?us-ascii?Q?oMD/3h9Sclc9eIbijlLA4q4QZsvEeL50Bhm6DDtD+nPqkr40FXt14VWJz3Op?= =?us-ascii?Q?Q+IgvYTEmRWVj0t3uQcvAG4hyownLRdhvZ3VfWJJwM2d04xI1PJQmHICUTTY?= =?us-ascii?Q?NI+rB5RfY63yqtSE/AXrtODDrTExzOZF1S0dxKwrO1purwfgNn5Ap1jWX/VT?= =?us-ascii?Q?/hHROWcj0vuMsNxsE9Y0lp8Xi2CPwCvjPeIIk29VKEtkdIoMOebWKEitQQow?= =?us-ascii?Q?UTKcfW22J12o45xwcYmyKXnIfwJ6CHmX?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(366016)(10070799003)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dz/UiTNDx6VmG64UXKGTE73gvamXgfbNuYL76Usu+SU1JPOoSHqIwIwpnEQ0?= =?us-ascii?Q?nIuCUkSzSMDvQCKAKztNbQckHvw86bLZHE/OR/56u11roz89rfXWrAw2y9BD?= =?us-ascii?Q?jujvqpe71LsIhZjH3DhjoQ+6xqOo8H32lBYp0CdS5JEtl9OQccg4hlEekta2?= =?us-ascii?Q?B1MXTAWeJZPwpvwPkMUVFEDprNGCu96E5yjTwExU+MMK6Q7VQNs75m94P983?= =?us-ascii?Q?kB0HVglL45R8F59mXAnr6Q+38MRIW8KX17ttC22s7EraEbhv37pLiRGDGqE4?= =?us-ascii?Q?ooElqUWgsJXDm/XqWnWuXIhqT9MD0BjE9Cko4+IdKiEFv7SbOhFr5RkWfiFX?= =?us-ascii?Q?dfHTaCCI6hJxfErPWVJkk30HYgjGDOk1BhI5dOtcuavnbckahJD7rFwtV/k/?= =?us-ascii?Q?yhhSpIp2wWsVWU4Ei9JSd8wAg0nwy9HIoFSPQ1yd8qijnupP+SdE2tM1IaHT?= =?us-ascii?Q?S7FlwMMD4Ut57RjZUXp1UUkSDZ8skoxvOjRfuXKINIe9leDiDtq+xlnD4uX4?= =?us-ascii?Q?oIQMVG3ifeeKhLsPUjIqlLAyfN4J5hgPmu3oxTcUTee7sYzRgf5Zh2u2/AHb?= =?us-ascii?Q?+OK44RIPMrQWw8TwDtxKXet+6WSC2+ez/rK7hK2I32qnd/bGwJyIp8TSZj/W?= =?us-ascii?Q?B9yLD6CwXAkTt+5d2dYX7Z7HsviR4Amp/hxLtJDnatowm3fOtM6mtC2RRvOd?= =?us-ascii?Q?JivOmePuvMbh9A0QAzW9Z7QWwJvSEDhbrB66ldIwl0j1Mt4JS3P4rLTV6dJi?= =?us-ascii?Q?i09i5muJKBdD+7JtmVGh2YqNlxirlAZhijRg9FP2nnOeQysr+arYKSyQPpAL?= =?us-ascii?Q?dp81s6t3SZWl58YtHRi6E17p86oYy4tQMS78wzdeLpI44tbfhUy9AOjFv6um?= =?us-ascii?Q?0wuXIyskzflrQBd4v4IeWZ0Mj4h256AAukNwzx4u7qh+dderDx4ur8dUXTC2?= =?us-ascii?Q?ybURot40pHs21T5zEvp934FoTeR6weAW6KMLPnaaj/BKC8hFn0pxN7/nI5JS?= =?us-ascii?Q?Rn2hG5WW45FbPJawIDYLRqWqOsmcCLYS4ql32hy4NyPSSi9kcvAvdD9U+MDc?= =?us-ascii?Q?bO+SUzLxzSfAtZjzMLbS8SCfbYoQWszl32zgSNJFnp5goKdD/dTCezM4TdZE?= =?us-ascii?Q?jscT0xAUtLPkTOF5JUGyD64hYuTxzh4jSd8g2KbCGAMvYlTvQessQ2Kz9Hcm?= =?us-ascii?Q?lJ5RAPg9GE5jKKOES8sEgCYKYmmPTphKBH/7QcuqTBfqw+IaLje+wzSx7fGk?= =?us-ascii?Q?/eoYgsAntZuPWj/V3da8E8vMjeLCVa7kvbRAkdYWsbL2lJi6nQMxmnbdaCzf?= =?us-ascii?Q?uzvIqyKZEN0YMI89fyg2imD1R7X5lN9Ho4bAsifGWN9Yg7X9SuW8xP+IeFoU?= =?us-ascii?Q?SgO2u8D2vJ4xpWIl7e6ecQX/cghq74YO0puYtkIVkOk21CZJBym07qAOTfM7?= =?us-ascii?Q?MUCdGjQkvqfFQ6K08jdfsjF8rdeqaoiX1NWhXAx5KYsGG32mqgL7b8VXxuFp?= =?us-ascii?Q?N1RndPWRUjQ2TBs045LcFE65SgS2+z9xq98C2XN5KgpLrnXaoncNnpvR2k3w?= =?us-ascii?Q?G8xMnGFmYbDkhsFGR0EYKbik6Nh/5L/F3H599AmlvIMx4hjN/NKfzrg0Zk4Q?= =?us-ascii?Q?av681mkOB5b36oto0+saW1k=3D?= X-OriginatorOrg: valinux.co.jp X-MS-Exchange-CrossTenant-Network-Message-Id: c8d7ab18-93f8-48ff-566d-08de1204be08 X-MS-Exchange-CrossTenant-AuthSource: OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2025 07:21:11.1604 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a57bee8-f73d-4c5f-a4f7-d72c91c8c111 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Nk6l16B2xTpAVOsGbt4QgkxIVl7blwcyqqpXxVv6H3SnA2G0I7Ymc0WNM5eArJHXXwJj7W4wj/XTvPWb7OAfkg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYRP286MB4555 Content-Type: text/plain; charset="utf-8" Drop the unused field 'retries' from struct ntb_queue_entry for simplicity's sake. Signed-off-by: Koichiro Den Reviewed-by: Dave Jiang > --- --- drivers/ntb/ntb_transport.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c index eb875e3db2e3..39b2398b95a6 100644 --- a/drivers/ntb/ntb_transport.c +++ b/drivers/ntb/ntb_transport.c @@ -113,7 +113,6 @@ struct ntb_queue_entry { void *buf; unsigned int len; unsigned int flags; - int retries; int errors; unsigned int tx_index; unsigned int rx_index; @@ -1630,9 +1629,7 @@ static void ntb_async_rx(struct ntb_queue_entry *entr= y, void *offset) if (res < 0) goto err; =20 - if (!entry->retries) - qp->rx_async++; - + qp->rx_async++; return; =20 err: @@ -1910,9 +1907,7 @@ static void ntb_async_tx(struct ntb_transport_qp *qp, if (res < 0) goto err; =20 - if (!entry->retries) - qp->tx_async++; - + qp->tx_async++; return; =20 err: @@ -2273,7 +2268,6 @@ int ntb_transport_rx_enqueue(struct ntb_transport_qp = *qp, void *cb, void *data, entry->buf =3D data; entry->len =3D len; entry->flags =3D 0; - entry->retries =3D 0; entry->errors =3D 0; entry->rx_index =3D 0; =20 @@ -2323,7 +2317,6 @@ int ntb_transport_tx_enqueue(struct ntb_transport_qp = *qp, void *cb, void *data, entry->len =3D len; entry->flags =3D 0; entry->errors =3D 0; - entry->retries =3D 0; entry->tx_index =3D 0; =20 rc =3D ntb_process_tx(qp, entry); --=20 2.48.1 From nobody Sun Feb 8 13:40:05 2026 Received: from OS0P286CU011.outbound.protection.outlook.com (mail-japanwestazon11010054.outbound.protection.outlook.com [52.101.228.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9FEC2D978A for ; Thu, 23 Oct 2025 07:21:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.228.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761204078; cv=fail; b=cTzq3KMXOsYzxTxsyI4xqtjR30gir1KhFsN4VN0rtVYblhBMZSmfwjPpqcnDRNIvTHnOdeNT93hjktcse5yLdKWLnwAPQvRKLkslMtYlNSHyfylquN+ritozUbP2ewVqw3EEcPzrINdmbcafF6VRAUOjdzExYaOwP9TTTs5MhEo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761204078; c=relaxed/simple; bh=S8A1G+FZbiNFQnCj3WPHDH27l1ELIcOu1L55zaOvxg8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=dg0DyxRwwfkp/LnwG7e3X3wXaXkZ/uOAkA6VAt3LJtjq9MIPsIGBdXqcdSp9oAROQOirc24BM/TQAUXNXZYevinSTQRZJjpjfdx1ovoqyMluaS97xNIo4i+kAmMC6wHcZsjxYmX/qYaoEwsMf+dnOLxaF1tO31jZPQ25/oZ8ITI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp; spf=pass smtp.mailfrom=valinux.co.jp; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b=ULICoEhO; arc=fail smtp.client-ip=52.101.228.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b="ULICoEhO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cO+9khdq4QcDeDrN69m3NIcWEe0KgLDRkL9ay2CBK0Z27HIHwbREyooDXrIK7N6fm5Xl7z95ylc9HQxOWH7faarAEJ3Ufv2ZyrNbE7yC2lEy4zU6Q+wYC/fDZYoC7H5hSPQPApfVaXfkKikbi2g5A1GoDSIQvd2EFNuUAidBYxmVf5+fEHayWo/+O/2UwMMLYtRP84PIfEA+tzaARK7nN2B2FTwvOpoBQGnYlp3zHED+dFGeIQmlU0ZGpkDd7alW8r9ghAqCyUKYuXnZuN/EYhdEH8PZrGKRsL77qw6ZaiiKBlIcbxmLCnGOi98rQeNnN/1nfgvU6oZqxqpppLmyQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lnyxGvV49oAOws+yZpz1074x60qxvFwVxHI59rl1yQ4=; b=TzEQ/2gkiJ15eaD19QNLJWFDYxTJUDsSR2JfkH7Hf7qPg0AJ9TZU22vmqKp+PclGC/69HDmGXp5xaqVkmdkXs8lAudHQ0lnpdTqV6InWviE6l1Jsr8dT6H3m2tTUDpu75ijHTNgv0e2Uz2uezPFTYooeBuMlf9ufEen29BtR5LAxLEam5RySGdHv3ndSdZmUmGSgBIJDe7T40mJw2CA1st/pKgTzHZWILvr9MN73hv25RP+7zQKXPBcRff4QfAkEUaskWC67nNU5EqrgwFjI16ZO3iTliLkrTF5GJgX07p6KYQ9GEt2SX/zUGi5UOg2M7DZF7Jl2vpRErBQbmP0Qzg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=valinux.co.jp; dmarc=pass action=none header.from=valinux.co.jp; dkim=pass header.d=valinux.co.jp; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=valinux.co.jp; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lnyxGvV49oAOws+yZpz1074x60qxvFwVxHI59rl1yQ4=; b=ULICoEhOdTIshTd3eX842nbu0TNh4Up7EkoEUfNes7W7r3SdPrRmYzl1hEyDDQJKAZI/wXORQN7I5yB5pmWFP83CN2GsA1z3BCkdB+59X+TPmJ1+DEJOERSA+r22kyNQknNal9HvHzIWSVIwWycuC4AkYcPuLCFrtUKhdf1uZFU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=valinux.co.jp; Received: from OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM (2603:1096:604:10d::7) by TYRP286MB4555.JPNP286.PROD.OUTLOOK.COM (2603:1096:405:1b0::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9253.13; Thu, 23 Oct 2025 07:21:12 +0000 Received: from OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM ([fe80::80f1:db56:4a11:3f7a]) by OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM ([fe80::80f1:db56:4a11:3f7a%5]) with mapi id 15.20.9253.011; Thu, 23 Oct 2025 07:21:12 +0000 From: Koichiro Den To: ntb@lists.linux.dev, linux-kernel@vger.kernel.org Cc: jdmason@kudzu.us, dave.jiang@intel.com, allenbh@gmail.com Subject: [PATCH 2/2] NTB: ntb_transport: Add 'tx_memcpy_offload' module option Date: Thu, 23 Oct 2025 16:21:05 +0900 Message-ID: <20251023072105.901707-3-den@valinux.co.jp> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251023072105.901707-1-den@valinux.co.jp> References: <20251023072105.901707-1-den@valinux.co.jp> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: TYCPR01CA0185.jpnprd01.prod.outlook.com (2603:1096:400:2b0::11) To OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM (2603:1096:604:10d::7) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: OS3P286MB0979:EE_|TYRP286MB4555:EE_ X-MS-Office365-Filtering-Correlation-Id: 095023e4-67a3-41b7-4352-08de1204beb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|10070799003|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?YHPfKLkYuXi020DFoD91+mHeibg1bZ/sX5sPBk6RXft+9x4302vRMZvhL22q?= =?us-ascii?Q?VAX4lLWKs4pLMv+DzaGkwWWOjSB/rkZLXlYeBKjaeybV7Lrj2MNszQ2anv0Y?= =?us-ascii?Q?1UUYTAZy+6Zm1BiSt9hLnFXIRi66FlxIbjnTwYx4ZcWazUjMs6rTqBYBHd6H?= =?us-ascii?Q?aczhOw+QqlU5mo67iy6lGlnVQyBmdumowlJhkdzdIvO+X+TyhcGPGndFg4Jl?= =?us-ascii?Q?IaBc5K+gnt8q/KxW8DzZ8p0ma27Vvy3ptoZZZ55h6tvJj3NRrgH1jOsSSn2a?= =?us-ascii?Q?RcC1WKtxHNo4G8/H7B3P/BvAzjeLuVzCDHhwy7bxFY5fw5p9ukFBsdvfudS7?= =?us-ascii?Q?x7RytLvlULFlphcBkhpayGD7SUIcQUE8dL6mrUOnszS9IS/qp8yXCzxirsUB?= =?us-ascii?Q?jsbgO+sz8dXwEBe74md/IZ3VhPV1Tycv5qV4IgpGKZ8gpHaMtw49NRy8WBjZ?= =?us-ascii?Q?txNZAHZ1SQlo0uO/3h6/Ce5JqRp5IfVm4XFEkav32ELKFEN+8nQTrEfSMepT?= =?us-ascii?Q?nfh1tt7WZYomqvZbvm0zrlRa6d1swOW8EgbSNnut0fzn11ZIe5oXCOXLJSIP?= =?us-ascii?Q?Naf45/tNaxJvouD+8eo4H+SeGCbcLlJYVOkai5qZ4+KSIUwFv8Y+X4UyFpGt?= =?us-ascii?Q?YjrUMojhPm8xZqlrJcwUYntvgxwAWoxvFVY/OArUNQUw3sLN9oA0eHwQDBA+?= =?us-ascii?Q?nxR4YfRI9vziLguyXI1EyiR6sN4qytx83TE4hIhMspi76crjq/go90OHQvP8?= =?us-ascii?Q?ks5M8MOVvMevCbjNln9XX5ehqvo+jyezNUA2/Ir/uzKvD7oQcd/UkAymldA4?= =?us-ascii?Q?iqM5h6EGP/dla7aH2yj7ZwijhzBzX7ryMnF9K8VeedJM8viv+OhqeW8C5ptg?= =?us-ascii?Q?I0PxZl82aFsneG6wc/ZRSSFNSBx9rvBCxevB1+WOQyzBjZnG7Pj0SDHq+qNg?= =?us-ascii?Q?26NPQ+/L7tEr8rQ7fGQ/OiILyqP4CvbYKjTO2/JszECndHigvfMnvpyRSdya?= =?us-ascii?Q?27vqVKWA9RiEUf7+WcZ7H4WkqgU6pqHJ2qebNSFp3yWqzOOdd5ncK1FhpNLt?= =?us-ascii?Q?Zh51fYmy71rPf8kX/EvXANm4zNaEnZbe5rM1Wi+0qbLvC9NSvRzrQRL++B7Y?= =?us-ascii?Q?l39ZOtJ7si42vcMjO0JtcutQIG3CzIARdwPxM8RPPkW5Fhjp5Vur6G2Aj12K?= =?us-ascii?Q?cv3ICWqIeAstBnyyU8bNNLcKzX1GOL+Kk7ZiRgyU7qvzpjPOtdzARQLpoF0A?= =?us-ascii?Q?xIsVL/+6WjVMMRqSmO50uyxyUL5ZI6UBjY07lAJVkyz8RDHsvFRPbjpmaGxR?= =?us-ascii?Q?X8u9mKZ06fUbbkAYnS4rsdPsSy7AY3xigBfKRduzAD8WNxW1Bp6KtQ7y8TYM?= =?us-ascii?Q?p/MW2tmOo7Q4qXqAAqLiwCnE07Nc6Ovbe4a0XOCSrE+poNir4q3i54woCkLP?= =?us-ascii?Q?s1Fu4Puwfu8vPao+BP29/Zfcf54zivVx?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(366016)(10070799003)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LxTemL92UlzNEuA6YVI+BDWZlUYW9YT3pxc3or1p61IexBI/1zSoBGd3ILdC?= =?us-ascii?Q?YtMmPInPdXxHI0XPcjp7wLdWuksG2ynD8yXgi/Xa9aP8ocOXJrfi5/12S7SU?= =?us-ascii?Q?FZA8PVZB6IOzcJh72348b+07jVJ/xauPDUgNwuMwmQmyz62JilVbtwZjgbTN?= =?us-ascii?Q?/iqML8OLMr9oynPyXZXyGS5fOrFnv4xqUq4TgQmwgFOIboWnx4D5EV5WjVfY?= =?us-ascii?Q?jJdFGfqdQdr7UAEiD2Au7vpHYWuY1GlLsp2XFoTeKY7BS+xqFwiHGZx6WF6M?= =?us-ascii?Q?ibHI5QFXOXM6cX7seCiM26XgO70kFVHDH97KJc5BS1wB/Pa3IuPZKPjv+zGq?= =?us-ascii?Q?bCZntyg4MASFik3mS9AnWtw7/qn0SO2lCpSMDA1aQn3hp42bdQIkkJRQ2OYM?= =?us-ascii?Q?GY6NWFYf66oj+hp4TFsP+/3BrQJWu3ql37eIeLs8NMA/cViLEHFYM9Gwqa/S?= =?us-ascii?Q?PMr3M+J2WDXzj0BtuBLrKSICk/jKe2hUhfRmzJh+ykh2LaSNZvwjxszxRiC0?= =?us-ascii?Q?D6YPSdqeClr7AdvePUrYunsBJEB9UztS/aDJBBwsxje9J1uqolu6ntDyBatZ?= =?us-ascii?Q?Rj1t9qEiOmNYjXX7Sxwnu1F2JVTKXb1osZlnXwV8WV9pUc9/TL3j6gpY2n1s?= =?us-ascii?Q?ePV/VMGoZsq67oVLEdWyWNUtff/mmhCXmmwHdsfWif05RuWnZGowlUKbXlfM?= =?us-ascii?Q?rAClE89FBa+exiru7Uzdosx9eHvhknoSBs9Z6PgjvrfFQp4/RAfN8yB6O8dT?= =?us-ascii?Q?mpVjL4lQVAu92FGlZaXDdb6AUMs+7XN6nqAP/2XieS7MbArpssSMQPs/bw/P?= =?us-ascii?Q?yy1kHYtM8aYKlZWN/X6w93djNHPjKSjs16PthbbWWadylLiGVhWh5pu+GNl/?= =?us-ascii?Q?o8b9mrXAwYRVzKl3P0CQawScRUTgVH3SrRJrr0u1ObsTXmrMv5SqxIwdCa0v?= =?us-ascii?Q?+CMNNYuHo7x+R8Y9AX4cZMe+cF4ecgd0V1GPYegh8UXPPco9jklWT3h4pRdK?= =?us-ascii?Q?tmYsAdzz0MUIDGFX1dJJXZiYyFF73jFmzsRH35dUPVX8LoroCdtrVb98GBLD?= =?us-ascii?Q?CiXOrpkDkoWeuM5k1NfG8HMfTD9WoP1NT5JCGjlqHqBLdnQCewOns9TlKlaA?= =?us-ascii?Q?ou+/UPQOJwD3rDsOMvrZyiJgOK5dmpYagjcNWYGULkphhgmEtxmzsJTQI498?= =?us-ascii?Q?MVe2Wq+HzNhkWJ1ZHTSH0/Zs3ttvFO3acu6HZ6La350vlXikyV3BRy1of6jz?= =?us-ascii?Q?Jxx0dpFw7dl8GT8b+qbAInNkmHQq3Xw/+MCHOLlKAc2doKC0K3mm66W6Mo7B?= =?us-ascii?Q?vje0zWd4jdqpVKhLxvIMnxWD/LvweK6vK3y86r+MC8dijqsZRBNyoSX+z1jl?= =?us-ascii?Q?L4nFYkNzhf/sEy9kI71ob6Guc7Ctc48HZR32E6VBwNCFz4wx2XP7HSQXEsWW?= =?us-ascii?Q?S8N/BtQrVQQb/P1ZlKj9/kGSeW1XveA5h53o/NgV8EwD8z73I1bRLaUUh8b0?= =?us-ascii?Q?KxP6dggaOrhJpX7x81kFnsGBuZncos+Hm5PD9gf0pAQaAg3qv4LW5ifRqqYm?= =?us-ascii?Q?83ItLPxTssgB1dklTjyBWfjZmi70+Rmml850my2OD+zStFEnj/zqYmOrdAKS?= =?us-ascii?Q?3AznJEk75XhXlJBL9XgNVjE=3D?= X-OriginatorOrg: valinux.co.jp X-MS-Exchange-CrossTenant-Network-Message-Id: 095023e4-67a3-41b7-4352-08de1204beb7 X-MS-Exchange-CrossTenant-AuthSource: OS3P286MB0979.JPNP286.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2025 07:21:12.2981 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a57bee8-f73d-4c5f-a4f7-d72c91c8c111 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YfMPuD/duJyEYYSwTdJ4TY3Ec+7tlK5WbrcETqlge7c7FkcZmgS/68zJBTyz5Yu28+hzPaVdTnx4hkDC3N69fg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYRP286MB4555 Content-Type: text/plain; charset="utf-8" Some platforms (e.g. R-Car S4) do not gain from using a DMAC on TX path in ntb_transport and end up CPU-bound on memcpy_toio(). Add a module parameter 'tx_memcpy_offload' that moves the TX memcpy_toio() and descriptor writes to a per-QP kernel thread. It is disabled by default. This change also fixes a rare ordering hazard in ntb_tx_copy_callback(), that was observed on R-Car S4 once throughput improved with the new module parameter: the DONE flag write to the peer MW, which is WC mapped, could be observed after the DB/MSI trigger. Both operations are posted PCIe MWr (often via different OB iATUs), so WC buffering and bridges may reorder visibility. Insert dma_mb() to enforce store->load ordering and then read back hdr->flags to flush the posted write before ringing the doorbell / issuing MSI. While at it, update tx_index with WRITE_ONCE() at the earlier possible location to make ntb_transport_tx_free_entry() robust. Signed-off-by: Koichiro Den --- drivers/ntb/ntb_transport.c | 104 ++++++++++++++++++++++++++++++++++-- 1 file changed, 100 insertions(+), 4 deletions(-) diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c index 39b2398b95a6..f4ed616c6ab8 100644 --- a/drivers/ntb/ntb_transport.c +++ b/drivers/ntb/ntb_transport.c @@ -54,12 +54,14 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include "linux/ntb.h" #include "linux/ntb_transport.h" =20 @@ -100,6 +102,10 @@ module_param(use_msi, bool, 0644); MODULE_PARM_DESC(use_msi, "Use MSI interrupts instead of doorbells"); #endif =20 +static bool tx_memcpy_offload; +module_param(tx_memcpy_offload, bool, 0644); +MODULE_PARM_DESC(tx_memcpy_offload, "Offload TX memcpy_toio() to a kernel = thread"); + static struct dentry *nt_debugfs_dir; =20 /* Only two-ports NTB devices are supported */ @@ -148,7 +154,9 @@ struct ntb_transport_qp { void (*tx_handler)(struct ntb_transport_qp *qp, void *qp_data, void *data, int len); struct list_head tx_free_q; + struct list_head tx_offl_q; spinlock_t ntb_tx_free_q_lock; + spinlock_t ntb_tx_offl_q_lock; void __iomem *tx_mw; phys_addr_t tx_mw_phys; size_t tx_mw_size; @@ -199,6 +207,9 @@ struct ntb_transport_qp { int msi_irq; struct ntb_msi_desc msi_desc; struct ntb_msi_desc peer_msi_desc; + + struct task_struct *tx_offload_thread; + wait_queue_head_t tx_offload_wq; }; =20 struct ntb_transport_mw { @@ -284,7 +295,13 @@ static int ntb_async_tx_submit(struct ntb_transport_qp= *qp, static void ntb_memcpy_tx(struct ntb_queue_entry *entry, void __iomem *off= set); static int ntb_async_rx_submit(struct ntb_queue_entry *entry, void *offset= ); static void ntb_memcpy_rx(struct ntb_queue_entry *entry, void *offset); +static int ntb_tx_memcpy_kthread(void *data); + =20 +static inline bool ntb_tx_offload_enabled(struct ntb_transport_qp *qp) +{ + return tx_memcpy_offload && qp && qp->tx_offload_thread; +} =20 static int ntb_transport_bus_match(struct device *dev, const struct device_driver *drv) @@ -1254,11 +1271,13 @@ static int ntb_transport_init_queue(struct ntb_tran= sport_ctx *nt, =20 spin_lock_init(&qp->ntb_rx_q_lock); spin_lock_init(&qp->ntb_tx_free_q_lock); + spin_lock_init(&qp->ntb_tx_offl_q_lock); =20 INIT_LIST_HEAD(&qp->rx_post_q); INIT_LIST_HEAD(&qp->rx_pend_q); INIT_LIST_HEAD(&qp->rx_free_q); INIT_LIST_HEAD(&qp->tx_free_q); + INIT_LIST_HEAD(&qp->tx_offl_q); =20 tasklet_init(&qp->rxc_db_work, ntb_transport_rxc_db, (unsigned long)qp); @@ -1784,6 +1803,13 @@ static void ntb_tx_copy_callback(void *data, =20 iowrite32(entry->flags | DESC_DONE_FLAG, &hdr->flags); =20 + /* + * Make DONE flag visible before DB/MSI. WC + posted MWr may reorder + * across iATU/bridge (platform-dependent). Order and flush here. + */ + dma_mb(); + ioread32(&hdr->flags); + if (qp->use_msi) ntb_msi_peer_trigger(qp->ndev, PIDX, &qp->peer_msi_desc); else @@ -1804,7 +1830,7 @@ static void ntb_tx_copy_callback(void *data, ntb_list_add(&qp->ntb_tx_free_q_lock, &entry->entry, &qp->tx_free_q); } =20 -static void ntb_memcpy_tx(struct ntb_queue_entry *entry, void __iomem *off= set) +static void ntb_memcpy_tx_on_stack(struct ntb_queue_entry *entry, void __i= omem *offset) { #ifdef ARCH_HAS_NOCACHE_UACCESS /* @@ -1822,6 +1848,54 @@ static void ntb_memcpy_tx(struct ntb_queue_entry *en= try, void __iomem *offset) ntb_tx_copy_callback(entry, NULL); } =20 +static int ntb_tx_memcpy_kthread(void *data) +{ + struct ntb_transport_qp *qp =3D data; + struct ntb_queue_entry *entry, *tmp; + const int resched_nr =3D 64; + LIST_HEAD(local_list); + void __iomem *offset; + int processed =3D 0; + + while (!kthread_should_stop()) { + spin_lock_irq(&qp->ntb_tx_offl_q_lock); + wait_event_interruptible_lock_irq_timeout(qp->tx_offload_wq, + kthread_should_stop() || + !list_empty(&qp->tx_offl_q), + qp->ntb_tx_offl_q_lock, 5*HZ); + list_splice_tail_init(&qp->tx_offl_q, &local_list); + spin_unlock_irq(&qp->ntb_tx_offl_q_lock); + + list_for_each_entry_safe(entry, tmp, &local_list, entry) { + list_del(&entry->entry); + offset =3D qp->tx_mw + qp->tx_max_frame * entry->tx_index; + ntb_memcpy_tx_on_stack(entry, offset); + if (++processed >=3D resched_nr) { + cond_resched(); + processed =3D 0; + } + } + cond_resched(); + } + + return 0; +} + +static void ntb_memcpy_tx(struct ntb_queue_entry *entry, void __iomem *off= set) +{ + struct ntb_transport_qp *qp =3D entry->qp; + + if (WARN_ON_ONCE(!qp)) + return; + + if (ntb_tx_offload_enabled(qp)) { + ntb_list_add(&qp->ntb_tx_offl_q_lock, &entry->entry, + &qp->tx_offl_q); + wake_up(&qp->tx_offload_wq); + } else + ntb_memcpy_tx_on_stack(entry, offset); +} + static int ntb_async_tx_submit(struct ntb_transport_qp *qp, struct ntb_queue_entry *entry) { @@ -1894,6 +1968,9 @@ static void ntb_async_tx(struct ntb_transport_qp *qp, hdr =3D offset + qp->tx_max_frame - sizeof(struct ntb_payload_header); entry->tx_hdr =3D hdr; =20 + WARN_ON_ONCE(!ntb_transport_tx_free_entry(qp)); + WRITE_ONCE(qp->tx_index, (qp->tx_index + 1) % qp->tx_max_entry); + iowrite32(entry->len, &hdr->len); iowrite32((u32)qp->tx_pkts, &hdr->ver); =20 @@ -1934,9 +2011,6 @@ static int ntb_process_tx(struct ntb_transport_qp *qp, =20 ntb_async_tx(qp, entry); =20 - qp->tx_index++; - qp->tx_index %=3D qp->tx_max_entry; - qp->tx_pkts++; =20 return 0; @@ -2033,6 +2107,20 @@ ntb_transport_create_queue(void *data, struct device= *client_dev, qp->tx_handler =3D handlers->tx_handler; qp->event_handler =3D handlers->event_handler; =20 + init_waitqueue_head(&qp->tx_offload_wq); + if (tx_memcpy_offload) { + qp->tx_offload_thread =3D kthread_run(ntb_tx_memcpy_kthread, qp, + "ntb-txcpy/%s/%u", + pci_name(ndev->pdev), qp->qp_num); + if (IS_ERR(qp->tx_offload_thread)) { + dev_warn(&nt->ndev->dev, + "tx memcpy offload thread creation failed: %ld; falling back to inlin= e copy\n", + PTR_ERR(qp->tx_offload_thread)); + qp->tx_offload_thread =3D NULL; + } + } else + qp->tx_offload_thread =3D NULL; + dma_cap_zero(dma_mask); dma_cap_set(DMA_MEMCPY, dma_mask); =20 @@ -2140,6 +2228,11 @@ void ntb_transport_free_queue(struct ntb_transport_q= p *qp) =20 qp->active =3D false; =20 + if (qp->tx_offload_thread) { + kthread_stop(qp->tx_offload_thread); + qp->tx_offload_thread =3D NULL; + } + if (qp->tx_dma_chan) { struct dma_chan *chan =3D qp->tx_dma_chan; /* Putting the dma_chan to NULL will force any new traffic to be @@ -2203,6 +2296,9 @@ void ntb_transport_free_queue(struct ntb_transport_qp= *qp) while ((entry =3D ntb_list_rm(&qp->ntb_tx_free_q_lock, &qp->tx_free_q))) kfree(entry); =20 + while ((entry =3D ntb_list_rm(&qp->ntb_tx_offl_q_lock, &qp->tx_offl_q))) + kfree(entry); + qp->transport->qp_bitmap_free |=3D qp_bit; =20 dev_info(&pdev->dev, "NTB Transport QP %d freed\n", qp->qp_num); --=20 2.48.1