From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0FD5C433F5 for ; Wed, 16 Mar 2022 07:12:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348099AbiCPHNM (ORCPT ); Wed, 16 Mar 2022 03:13:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354174AbiCPHMw (ORCPT ); Wed, 16 Mar 2022 03:12:52 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6505DE74; Wed, 16 Mar 2022 00:11:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414699; x=1678950699; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BLFuTRhqE482sqaHRif7NDKXQY4BaFJ4C/qrheqYfJs=; b=Jv5MP3dlRFp8xrNGv1GrZIu2sytx29ZnFEpf2UHNS73fb2HxBvziY7oE j0gD2uGrG9Ty+ehURzFcjIbwhoutcTMMt6MM6e1eGo3tIk3s9eI0Eu4nG TDGtj7LrH6iSm+Mg5Zv9OqI01SIkitG3ZZMROCPhpHDyu4oxN47xdYlxC Xt/z8B7eM2yOpE7BRJoR3HIA4/CeWb7uqmLrtEKPo1Sb+0yB7FT7Q+K9B du4llyl6hv+okUbMfK39hdO8POF4Lps0UITI2HT+WOrsmqPX3rPQBrluT a2ipnaR6AQclNUCawo7KrB33UXaIgsGhdNIkFJRIyfY7QAT9waIilzY46 A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289023" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289023" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538324" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:35 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Matthew Gerlach , Tianfei Zhang Subject: [PATCH v6 1/6] fpga: dfl: Allow ports without local bar space. Date: Wed, 16 Mar 2022 03:08:09 -0400 Message-Id: <20220316070814.1916017-2-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Matthew Gerlach In OFS, each PR slot (AFU) has one port device which include Port control, Port user clock control and Port errors. In legacy model, the AFU MMIO space was connected with Port device, so from port device point of view, there is a bar space associated with this port device. But in "Multiple VFs per PR slot" model, the AFU MMIO space was not connected with Port device. The BarID (3bits field) in PORTn_OFFSET register indicates which PCI bar space associated with this port device, the value 0b111 (FME_HDR_NO_PORT_BAR) means that no PCI bar for this port device. --- v3: add PCI bar number checking with PCI_STD_NUM_BARS. v2: use FME_HDR_NO_PORT_BAR instead of PCI_STD_NUM_BARS. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei Zhang --- drivers/fpga/dfl-pci.c | 7 +++++++ drivers/fpga/dfl.h | 1 + 2 files changed, 8 insertions(+) diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c index 4d68719e608f..2e9abeca3625 100644 --- a/drivers/fpga/dfl-pci.c +++ b/drivers/fpga/dfl-pci.c @@ -258,6 +258,13 @@ static int find_dfls_by_default(struct pci_dev *pcidev, */ bar =3D FIELD_GET(FME_PORT_OFST_BAR_ID, v); offset =3D FIELD_GET(FME_PORT_OFST_DFH_OFST, v); + if (bar >=3D PCI_STD_NUM_BARS || + bar =3D=3D FME_HDR_NO_PORT_BAR) { + dev_dbg(&pcidev->dev, "skipping port without local BAR space %d\n", + bar); + continue; + } + start =3D pci_resource_start(pcidev, bar) + offset; len =3D pci_resource_len(pcidev, bar) - offset; =20 diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index 53572c7aced0..1fd493e82dd8 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -91,6 +91,7 @@ #define FME_HDR_PORT_OFST(n) (0x38 + ((n) * 0x8)) #define FME_HDR_BITSTREAM_ID 0x60 #define FME_HDR_BITSTREAM_MD 0x68 +#define FME_HDR_NO_PORT_BAR 7 =20 /* FME Fab Capability Register Bitfield */ #define FME_CAP_FABRIC_VERID GENMASK_ULL(7, 0) /* Fabric version ID */ --=20 2.26.2 From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D0CCC433F5 for ; Wed, 16 Mar 2022 07:13:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354175AbiCPHNY (ORCPT ); Wed, 16 Mar 2022 03:13:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354231AbiCPHMz (ORCPT ); Wed, 16 Mar 2022 03:12:55 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C09A5DE54; Wed, 16 Mar 2022 00:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414702; x=1678950702; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BbcJfJHFs7h3DBzP//dTgYkN4sqD2gOtwRAt4y/7sbQ=; b=ISA3UHjm0e4IDet9v7Q7ALfvOPkAdmXpCWfFQBxCUngtvv7m3QTT0qy1 +6w/qWQ/7xtroFC8PPIIABNhJ/xXrrc/NSAsQDdsc7+8mLaOHyy67W5tT P8luulIQRAJ4kzlhzy2Z1dyWF0EIKVg05mUjO3UdxVGBZw0WBAmUfuA66 gILqDZ8vFgunbsDTsE4vGt065XU7sB1oPGW767Vpib9/QvSPRxMOPfNIN obEfxbgf3UUvA9dr9E9ryNecTVpM4Z3M2Rr9FyAzZY+91V5ssDgla6jU4 JB4HZ6miSxQKa1DAAOFZnsH1Fza2BUJNPHFoke8d36U703TRcd2XMqjOL A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289028" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289028" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538338" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:38 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Tianfei zhang , Matthew Gerlach Subject: [PATCH v6 2/6] fpga: dfl: tracking port conntected with AFU Date: Wed, 16 Mar 2022 03:08:10 -0400 Message-Id: <20220316070814.1916017-3-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tianfei zhang Introducing flags in dfl_fpga_cdev to track extensions or new features discovered during DFL enumeration. It uses some lowest bits of flags to track the port status which the AFU was connected to port device or not. In legacy model, the AFU was connected to Port device, but in "multiple VFs per PR slot" model, the AFU or PR slot without connected to Port device directly. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei zhang --- drivers/fpga/dfl.c | 11 ++++++++++- drivers/fpga/dfl.h | 12 ++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index 599bb21d86af..712c53363fda 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1124,8 +1124,10 @@ static void build_info_complete(struct build_feature= _devs_info *binfo) static int parse_feature_fiu(struct build_feature_devs_info *binfo, resource_size_t ofst) { + struct dfl_fpga_cdev *cdev =3D binfo->cdev; int ret =3D 0; u32 offset; + u32 port; u16 id; u64 v; =20 @@ -1160,8 +1162,15 @@ static int parse_feature_fiu(struct build_feature_de= vs_info *binfo, v =3D readq(binfo->ioaddr + NEXT_AFU); =20 offset =3D FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v); - if (offset) + if (offset) { + if (dfh_id_to_type(id) =3D=3D PORT_ID) { + port =3D FIELD_GET(PORT_CAP_PORT_NUM, + readq(binfo->ioaddr + PORT_HDR_CAP)); + cdev->flags |=3D dfl_feat_port_connect_afu(port); + } + return parse_feature_afu(binfo, offset); + } =20 dev_dbg(binfo->dev, "No AFUs detected on FIU %d\n", id); =20 diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index 1fd493e82dd8..bc56b7e8c01b 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -461,6 +461,16 @@ int dfl_fpga_enum_info_add_irq(struct dfl_fpga_enum_in= fo *info, unsigned int nr_irqs, int *irq_table); void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info *info); =20 +/* + * Bitfields in flags of dfl_fpga_cdev. + * + * 0 - (DFL_PORT_CONNECT_BITS -1): AFU was connected with Port device. + * DFL_PORT_CONNECT_BITS - 63: reserved. + */ +#define dfl_feat_port_connect_afu(port) (BIT_ULL(port)) +#define DFL_PORT_CONNECT_BITS MAX_DFL_FPGA_PORT_NUM +#define DFL_FEAT_PORT_CONNECT_MASK ((1UL << (DFL_PORT_CONNECT_BITS)) - 1) + /** * struct dfl_fpga_cdev - container device of DFL based FPGA * @@ -470,6 +480,7 @@ void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info = *info); * @lock: mutex lock to protect the port device list. * @port_dev_list: list of all port feature devices under this container d= evice. * @released_port_num: released port number under this container device. + * @flags: extensions discovered during DFL enumeration. */ struct dfl_fpga_cdev { struct device *parent; @@ -478,6 +489,7 @@ struct dfl_fpga_cdev { struct mutex lock; struct list_head port_dev_list; int released_port_num; + u64 flags; }; =20 struct dfl_fpga_cdev * --=20 2.26.2 From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71285C433FE for ; Wed, 16 Mar 2022 07:12:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354243AbiCPHN2 (ORCPT ); Wed, 16 Mar 2022 03:13:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354247AbiCPHM6 (ORCPT ); Wed, 16 Mar 2022 03:12:58 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39D035DE74; Wed, 16 Mar 2022 00:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414705; x=1678950705; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=82RRYaqdq5m//xBFiiE6/hsoZBDz6vsg38N2TZZZ+hQ=; b=SfYA5E9IaBSag5LmC5H38xKat5RbkGZk9AHPnZJylivVjXBVON2M3J4A 9a+YG7qXn16v51GRJDPKomIKVJemEPbqf1e4fa/UBUNpUJahBDF5Lfy5w iu5+GDfw05aKT05eeaQpQjzSTX9sQaRLCniE7/0RNtAMxLc9M6RSIbRVt 9FkkEb42d347opE8U+DuBTjdkwhHjooZQqkqLR/BN95LgUnP5vg9TedAY 0auc17uH352rhbNCB+bMpG2V4vn43uVTVW1myu/R6E/U2E0qP3qNZ5Ehl 7i569qwispARbkhzFBB3hNbU9/8vJGyXnXqZlhTgM5NfTx3DLR6HDrhzv g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289037" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289037" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538353" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:42 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Tianfei zhang , Matthew Gerlach Subject: [PATCH v6 3/6] fpga: dfl: check released_port_num and num_vfs for legacy model Date: Wed, 16 Mar 2022 03:08:11 -0400 Message-Id: <20220316070814.1916017-4-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tianfei zhang In OFS legacy model, there is 1:1 mapping for Port device and VF, so it need to check the number of released port match the number of VFs or not. But in "Multiple VFs per PR slot" model, there is 1:N mapping for the Port device and VFs. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei zhang --- drivers/fpga/dfl.c | 10 ++++++---- drivers/fpga/dfl.h | 2 ++ 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index 712c53363fda..b95b29c5c81d 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1707,11 +1707,13 @@ int dfl_fpga_cdev_config_ports_vf(struct dfl_fpga_c= dev *cdev, int num_vfs) =20 mutex_lock(&cdev->lock); /* - * can't turn multiple ports into 1 VF device, only 1 port for 1 VF - * device, so if released port number doesn't match VF device number, - * then reject the request with -EINVAL error code. + * In the OFS legacy model, it can't turn multiple ports into 1 VF + * device, because only 1 port conneced to 1 VF device, so if released + * port number doesn't match VF device number, then reject the request + * with -EINVAL error code. */ - if (cdev->released_port_num !=3D num_vfs) { + if ((dfl_has_port_connected_afu(cdev) && + cdev->released_port_num !=3D num_vfs)) { ret =3D -EINVAL; goto done; } diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index bc56b7e8c01b..83c2c50975e5 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -471,6 +471,8 @@ void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info = *info); #define DFL_PORT_CONNECT_BITS MAX_DFL_FPGA_PORT_NUM #define DFL_FEAT_PORT_CONNECT_MASK ((1UL << (DFL_PORT_CONNECT_BITS)) - 1) =20 +#define dfl_has_port_connected_afu(cdev) ((cdev)->flags & DFL_FEAT_PORT_CO= NNECT_MASK) + /** * struct dfl_fpga_cdev - container device of DFL based FPGA * --=20 2.26.2 From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ECF4C433EF for ; Wed, 16 Mar 2022 07:12:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352103AbiCPHNP (ORCPT ); Wed, 16 Mar 2022 03:13:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354265AbiCPHNB (ORCPT ); Wed, 16 Mar 2022 03:13:01 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 548115DE54; Wed, 16 Mar 2022 00:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414708; x=1678950708; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iLyZDvemV1Z1JW/SXRxgwjuzRQL+/0fI6UJa6N3g8s0=; b=B0+PRNBPqdVA6arFi49I1gvnscUC3fCYQrHTM3Skib2D2H35ze7n9XMG 75C+53ZjDZG7A+zLpQ9/io2eukmzbT1vWPAMV0fzI2c1V2/Jhy5W1SHT3 +7JX9dqgsxnep4J5LYvAexQNl96easQ7kd5gPYz4Oic1TUAh7QDIM3yCU 5cNYqEBuNSUdsvGxqcvFy77msnUOQSRxB9cStjog8EtnyCf+JV32Mc7Sw f+cbgJFjjB/uHkiflMW4FRUu4HBkpXL77WRz56BzNSNe8VTIIuOjT+XvO fTfs+TgD7wZpb3R+aXxs+VmmWle3otRzAIUt8S0s7Nqtsg/Q3s9If/49B g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289052" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289052" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538374" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:45 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Tianfei zhang , Matthew Gerlach Subject: [PATCH v6 4/6] fpga: dfl: configure port access mode for afu connected with port Date: Wed, 16 Mar 2022 03:08:12 -0400 Message-Id: <20220316070814.1916017-5-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tianfei zhang In legacy model, we should set AfuAccessCtrl (Bit 55) in PORTn_OFFSET register to switch VF and PF for AFU. But in "multiple VFs per PR slot" model, the PF/VF mux hardware unit will statically configure the funciton mapping without set the AfuAccessCtrl by software. This patch check the port status in dfl_fpga_cdev->flags before configure the port access mode. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei zhang --- drivers/fpga/dfl.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index b95b29c5c81d..71e0725b6be0 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1666,6 +1666,17 @@ static void config_port_access_mode(struct device *f= me_dev, int port_id, #define config_port_vf_mode(dev, id) config_port_access_mode(dev, id, true) #define config_port_pf_mode(dev, id) config_port_access_mode(dev, id, fals= e) =20 +static int dfl_check_port_connect_afu(struct device *dev, u64 flags) +{ + void __iomem *base; + int port; + + base =3D dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER); + port =3D FIELD_GET(PORT_CAP_PORT_NUM, readq(base + PORT_HDR_CAP)); + + return flags & dfl_feat_port_connect_afu(port); +} + /** * dfl_fpga_cdev_config_ports_pf - configure ports to PF access mode * @@ -1683,7 +1694,9 @@ void dfl_fpga_cdev_config_ports_pf(struct dfl_fpga_cd= ev *cdev) if (device_is_registered(&pdata->dev->dev)) continue; =20 - config_port_pf_mode(cdev->fme_dev, pdata->id); + /* configure port access mode for AFU connected to Port device */ + if (dfl_check_port_connect_afu(&pdata->dev->dev, cdev->flags)) + config_port_pf_mode(cdev->fme_dev, pdata->id); } mutex_unlock(&cdev->lock); } @@ -1722,7 +1735,9 @@ int dfl_fpga_cdev_config_ports_vf(struct dfl_fpga_cde= v *cdev, int num_vfs) if (device_is_registered(&pdata->dev->dev)) continue; =20 - config_port_vf_mode(cdev->fme_dev, pdata->id); + /* configure port access mode for AFU connected to Port device */ + if (dfl_check_port_connect_afu(&pdata->dev->dev, cdev->flags)) + config_port_vf_mode(cdev->fme_dev, pdata->id); } done: mutex_unlock(&cdev->lock); --=20 2.26.2 From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDBA9C433F5 for ; Wed, 16 Mar 2022 07:12:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354255AbiCPHNa (ORCPT ); Wed, 16 Mar 2022 03:13:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354274AbiCPHNE (ORCPT ); Wed, 16 Mar 2022 03:13:04 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58B045DE54; Wed, 16 Mar 2022 00:11:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414711; x=1678950711; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RWgY9AC+lYBXFFxLtUrfMxojVXpyTuSrrkNnLAQetlM=; b=YXUhN6Um8u8khJcmILJdE39pl8/qK5QGGBq9fm3k43vO1gPWRCUd/QLk 1wf1TYeRkevM8opmQSSeWzR1lxNgHIMWnzzR4zn8p/utf5JpfGHsa15sk I3uasEsGyo3ICyshdOhQk1dF4lwYk1NM3/HmJR4p8IRxDyQUVU/qPhY8s txXQ7SOUD1PMV2OveAoHgPOd3HYXMhDkGzBBHsitjt4czXG6G8/e6vGzp X06M/VV5MYqJC7oKOfMdyV59hmviSi+mqTBBy5WBysGlghIIxXLndh13p 8CnD+lFMtGG9PhjZqHph8xT6D6pqRHlzVj0+Brkp8w1qCEmL5b+byJD+1 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289060" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289060" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538396" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:48 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Matthew Gerlach , Tianfei Zhang Subject: [PATCH v6 5/6] fpga: dfl: support PF/VF starting with DFH Date: Wed, 16 Mar 2022 03:08:13 -0400 Message-Id: <20220316070814.1916017-6-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Matthew Gerlach In OFS, it allows several PFs and VFs in static region or PR region. Those PFs and VFs managed by DFL or a specific device, like virtio-net device. Those PFs and VFs which managed by DFL can start with DFH, and leverage VFIO to expose to an application or assign to a VM. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei Zhang --- drivers/fpga/dfl-pci.c | 2 ++ drivers/fpga/dfl.c | 22 +++++++++++++--------- drivers/fpga/dfl.h | 7 +++++++ 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c index 2e9abeca3625..7d8b53330152 100644 --- a/drivers/fpga/dfl-pci.c +++ b/drivers/fpga/dfl-pci.c @@ -275,6 +275,8 @@ static int find_dfls_by_default(struct pci_dev *pcidev, len =3D pci_resource_len(pcidev, 0); =20 dfl_fpga_enum_info_add_dfl(info, start, len); + } else if (dfl_feature_is_afu(base)) { + dev_info(&pcidev->dev, "find AFU\n"); } else { ret =3D -ENODEV; } diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index 71e0725b6be0..db676f7482ec 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -900,9 +900,11 @@ static void build_info_free(struct build_feature_devs_= info *binfo) dfl_id_free(feature_dev_id_type(binfo->feature_dev), binfo->feature_dev->id); =20 - list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) { - list_del(&finfo->node); - kfree(finfo); + if (!list_empty(&binfo->sub_features)) { + list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) { + list_del(&finfo->node); + kfree(finfo); + } } } =20 @@ -1444,12 +1446,14 @@ dfl_fpga_feature_devs_enumerate(struct dfl_fpga_enu= m_info *info) * start enumeration for all feature devices based on Device Feature * Lists. */ - list_for_each_entry(dfl, &info->dfls, node) { - ret =3D parse_feature_list(binfo, dfl->start, dfl->len); - if (ret) { - remove_feature_devs(cdev); - build_info_free(binfo); - goto unregister_region_exit; + if (!list_empty(&info->dfls)) { + list_for_each_entry(dfl, &info->dfls, node) { + ret =3D parse_feature_list(binfo, dfl->start, dfl->len); + if (ret) { + remove_feature_devs(cdev); + build_info_free(binfo); + goto unregister_region_exit; + } } } =20 diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index 83c2c50975e5..08edaeeb7f80 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -421,6 +421,13 @@ static inline bool dfl_feature_is_port(void __iomem *b= ase) (FIELD_GET(DFH_ID, v) =3D=3D DFH_ID_FIU_PORT); } =20 +static inline bool dfl_feature_is_afu(void __iomem *base) +{ + u64 v =3D readq(base + DFH); + + return (FIELD_GET(DFH_TYPE, v) =3D=3D DFH_TYPE_AFU); +} + static inline u8 dfl_feature_revision(void __iomem *base) { return (u8)FIELD_GET(DFH_REVISION, readq(base + DFH)); --=20 2.26.2 From nobody Mon Jun 22 22:26:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06CBC433EF for ; Wed, 16 Mar 2022 07:12:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354273AbiCPHNd (ORCPT ); Wed, 16 Mar 2022 03:13:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354281AbiCPHNI (ORCPT ); Wed, 16 Mar 2022 03:13:08 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 415555DE54; Wed, 16 Mar 2022 00:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647414714; x=1678950714; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l15JadKf9Nz32702pJxV1seWJTMzNF6fJsTjk92xnng=; b=cJ6vvZ5SwaC2rb2qK5T5e02fDovibkzkqKUxuet3IT5R0WATh2eSUW48 PjvpSqTKOhixrJLJZCOHfrburGjKrJ8B9XdvPsbLpqGKa0Nr1QOJjAi6F 3iIb8RsWNOWlXZ01fXYlqTDDAVHaDCZ2GV4RNf/8lrPnpxT1HydTc4JFT yYJLUsYKDrpTiQzvUVdUSxa7VejxuBbd+0YXGYzFZM/J2rCHUKQvJUdg8 WlwLjOwYWQm8w4Im4nWAMKANQYpeqebkZ0D4+752Hsq2l/bTR8FcwPI65 xQ0wmR6L1gExY+sEwIzEGRSZ3SbHs9LiGQI4/RchlPlZ5bFujPSCB6Hxk A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="281289067" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="281289067" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 00:11:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="646538406" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by orsmga004.jf.intel.com with ESMTP; 16 Mar 2022 00:11:51 -0700 From: Tianfei Zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rdunlap@infradead.org Cc: corbet@lwn.net, Tianfei zhang Subject: [PATCH v6 6/6] Documentation: fpga: dfl: add description of OFS Date: Wed, 16 Mar 2022 03:08:14 -0400 Message-Id: <20220316070814.1916017-7-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220316070814.1916017-1-tianfei.zhang@intel.com> References: <20220316070814.1916017-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tianfei zhang This patch adds description about OFS support for DFL. --- v6: fix documentation with Randy's comment. v5: fix documentation with Matthew and Randy's comment. v4: add description about access the AFU on "multiple VFs per PR slot" model. v3: change IOFS to OFS in documentation. v2: * Fixs some typos. * Adds more detail description about the models of AFU access which support= ed in OFS. Signed-off-by: Tianfei zhang --- Documentation/fpga/dfl.rst | 114 +++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst index ef9eec71f6f3..93f262fe7b8c 100644 --- a/Documentation/fpga/dfl.rst +++ b/Documentation/fpga/dfl.rst @@ -556,6 +556,120 @@ new DFL feature via UIO direct access, its feature id= should be added to the driver's id_table. =20 =20 +Open FPGA Stack +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Open FPGA Stack (OFS) is a collection of RTL and open source software prov= iding +interfaces to access the instantiated RTL easily in an FPGA. OFS leverages= the +DFL for the implementation of the FPGA RTL design. + +OFS designs allow for the arrangement of software interfaces across multip= le +PCIe endpoints. Some of these interfaces may be PFs defined in the static = region +that connect to interfaces in an IP that is loaded via Partial Reconfigura= tion (PR). +And some of these interfaces may be VFs defined in the PR region that can = be +reconfigured by the end-user. Furthermore, these PFs/VFs may use DFLs such= that +features may be discovered and accessed in user space (with the aid of a g= eneric +kernel driver like vfio-pci). The diagram below depicts an example design = with two +PFs and two VFs. In this example, it will export the management functions = via PF0, +PF1 will bind with virtio-net driver presenting itself as a network interf= ace to +the OS. The other functions, VF0 and VF1, leverage VFIO to export the MMIO= space +to an application or assign to a VM. +:: + + +-----------------+ +--------------+ +-------------+ +------------+ + | FPGA Management | | VirtIO | | User App | | Virtual | + | App | | App | | | | Machine | + +--------+--------+ +------+-------+ +------+------+ +-----+------+ + | | | | + +--------+--------+ +------+-------+ +------+------+ | + | DFL Driver | |VirtIO driver | | VFIO | | + +--------+--------+ +------+-------+ +------+------+ | + | | | | + | | | | + +--------+--------+ +------+-------+ +------+------+ +----+------+ + | PF0 | | PF1 | | PF0_VF0 | | PF0_VF1 | + +-----------------+ +--------------+ +-------------+ +-----------+ + +As accelerators are specialized hardware, they are typically limited in the +number installed in a given system. Many use cases require them to be shar= ed +across multiple software contexts or threads of software execution, either +through partitioning of individual dedicated resources, or virtualization = of +shared resources. OFS provides several models to share the AFU resources v= ia +PR mechanism and hardware-based virtualization schemes. + +1. Legacy model. + With legacy model FPGA cards like Intel PAC N3000 or N5000, there is + a notion that the boundary between the AFU and the shell is also the un= it of + PR for those FPGA platforms. This model is only able to handle a + single context, because it only has one PR engine, and one PR region wh= ich + has an associated Port device. +2. Multiple VFs per PR slot. + In this model, available AFU resources may allow instantiation of many = VFs + which have a dedicated PCIe function with their own dedicated MMIO spac= e, or + partition a region of MMIO space on a single PCIe function. Intel PAC N= 6000 + card has implemented this model. + In this model, the AFU/PR slot was not connected to port device. For DF= L's view, + the Next_AFU pointer in FIU feature header of port device points to NUL= L in this + model, so in AFU driver perspective, there is no AFU MMIO region manage= d by + AFU driver. On the other hand, each VF can start with an AFU feature he= ader without + being connected to a FIU Port feature header. + +In multiple VFs per PR slot model, the port device can still be accessed u= sing +ioctls API which expose /dev/dfl-port.h device nodes, like port reset, get +port info, whose APIs were mentioned in AFU section in this documentation.= But +it cannot access the AFU MMIO space via AFU ioctl APIs like DFL_FPGA_PORT_= DMA_MAP +because there is no AFU MMIO space managed in the AFU driver. Users can ac= cess +the AFU resource by creating VF devices via PCIe SRIOV interface, and then= access +the VF via VFIO driver or assign the VF to VM. + +In multiple VFs per PR slot model, the steps to enable VFs are compatible = with +legacy mode which are mentioned in "FPGA virtualization - PCIe SRIOV" sect= ion +in this documentation. + +OFS provides the diversity for accessing the AFU resource to RTL developer. +An IP designer may choose to add more than one PF for interfacing with IP +on the FPGA and choose different model to access the AFU resource. + +There is one reference architecture design using the "Multiple VFs per PR = slot" +model for OFS as illustrated below. In this reference design, it exports t= he +FPGA management functions via PF0. PF1 will bind with virtio-net driver +presenting itself as a network interface to the OS. PF2 will bind to the +vfio-pci driver allowing the user space software to discover and interface +with the specific workload like diagnostic test. To access the AFU resourc= e, +it uses SR-IOV to partition workload interfaces across various VFs. +:: + + +----------------------+ + | PF/VF mux/demux | + +--+--+-----+------+-+-+ + | | | | | + +------------------------+ | | | | + PF0 | +---------+ +-+ | | + +---+---+ | +---+----+ | | + | DFH | | | DFH | | | + +-------+ +-----+----+ +--------+ | | + | FME | | VirtIO | | Test | | | + +---+---+ +----------+ +--------+ | | + | PF1 PF2 | | + | | | + | +----------+ | + | | ++ + | | | + | | PF0_VF0 | PF0_VF1 + | +-----------------+-----------+------------+ + | | +-----+-----------+--------+ | + | | | | | | | + | | +------+ | +--+ -+ +--+---+ | | + | | | Port | | | DFH | | DFH | | | + +-----------+ +------+ | +-----+ +------+ | | + | | | DEV | | DEV | | | + | | +-----+ +------+ | | + | | PR Slot | | + | +--------------------------+ | + | Port Gasket | + +------------------------------------------+ + + Open discussion =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfigur= ation --=20 2.26.2