From nobody Fri Nov 29 19:33:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1632402532; cv=none; d=zohomail.com; s=zohoarc; b=QnRK0ec0+m9ZF7r0eYpM3onqYuGhbaRvtU97vXsan4xRyp16a3Vj6fLDmQ5KLgXnLAlU7/l1rk3zJ8ApB3RNbp1RcqkJ6maWXAchGmPEJilf9zx58dTMuOi1We50rbE9vIPDknLo+C2JlJ7y7OcUvABe5fTw29yJLgyufm88eE0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1632402532; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=01d6SaTi/ep3sxri+AlZsSZXy8XnKOJzH2RZEoLRhnE=; b=ZukxB1nLu6zspeLReKgAtetlkRcGL9nIv7nHuGefxho/AZUdhJoheXqCfpO19lYsITyq39fnVp3kc8VC5R1Ab//Ai2krYMBLnfSL+N36mmeyCzZbrtRcLmAuKHV7ncROcESBEeDr4krG0E7YqMUQdgHi2nNtArLAkUfk71GDG+4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1632402532371904.6287511915875; Thu, 23 Sep 2021 06:08:52 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.194212.346030 (Exim 4.92) (envelope-from ) id 1mTOSr-0004VK-Rs; Thu, 23 Sep 2021 13:08:37 +0000 Received: by outflank-mailman (output) from mailman id 194212.346030; Thu, 23 Sep 2021 13:08:37 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOSr-0004VD-NJ; Thu, 23 Sep 2021 13:08:37 +0000 Received: by outflank-mailman (input) for mailman id 194212; Thu, 23 Sep 2021 13:08:36 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHu-0004it-7y for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:57:18 +0000 Received: from mail-lf1-x12e.google.com (unknown [2a00:1450:4864:20::12e]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id bf96aaab-a758-4d13-82cb-0f9b9f285ee2; Thu, 23 Sep 2021 12:55:16 +0000 (UTC) Received: by mail-lf1-x12e.google.com with SMTP id b20so26425942lfv.3 for ; Thu, 23 Sep 2021 05:55:16 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:14 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bf96aaab-a758-4d13-82cb-0f9b9f285ee2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=01d6SaTi/ep3sxri+AlZsSZXy8XnKOJzH2RZEoLRhnE=; b=EWpEK98V3gn+BXUIniS4aeCA9MHYJ+uraGzPRHWI6TQV1B/vOoZeupB+37cr+IGN5l ecgShTHEtU2cFr68IVOvI7ZjrPH0mELx7q5M2RfFA8RxYf62iGZGZgB4d9huCMUKwJfI rGF75wr4M7zdy8Vi/3omiimpqMmoiPB++DcA182+C+cCOS5SNWsaGte1VMc4ISIdMYeU rvNSAtt5UvAYcglcHlgkOy6E/D+dEtx4joYE3SnnrHq3wzNfOPDbQQ5ep1vQAXvSRdWv ZwRm1UayVuL8hQeFccvHM3BPFjlSdpDw6ZPFeeWU6FplECWsRyCoua3q5zY1abpy29BS TyoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=01d6SaTi/ep3sxri+AlZsSZXy8XnKOJzH2RZEoLRhnE=; b=nlcDXV1u1cnc+4YFn2ba3tN+MGMyo7eUUp1AuyM5rohwKnRgY+7oW0X6RtD5eUJ7+e AqYMP9BBVaOlTKaGkRt4R54QFKzSbbNwPwpicQTodRd7BiSTQXxLV5kzi8lpDOm6qwkQ ye4PgQrSTQHy5gaIOcl/x+3T3bQi75+LN/sx/Ug7N1k8u6vrZ/mH1FbfnKjfcOx3Xfzx Fb5FwOgfOUnzqqAtUPZTyHdzcWB3sWimRMyKFLrSSLZrk/sAeVS1mm0kTRfM2kG4D+hx EQtoqiFas0VSEZSQqJBoWg5wSrk/+sAqHLBv5mWR5pZMJoQtcyDXc43gDsYdbIAPJnt4 K88w== X-Gm-Message-State: AOAM530ziUhSgaeuaJ6B9MN3eXhcEwn5DSVjiDSnr4jS0ZrYc0ctve/1 zZt8XsdKY4miS6JmE6ymVZBk41ACHtlJ2g== X-Google-Smtp-Source: ABdhPJx3puHxJXs6dd2LzzWCWWYJzbY4YRnuivwUaxA1/mtGOwJpGs6ZnDTtx7CE1RfoyWW9sphKRw== X-Received: by 2002:a2e:6e08:: with SMTP id j8mr4786466ljc.247.1632401715286; Thu, 23 Sep 2021 05:55:15 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 10/11] vpci: Add initial support for virtual PCI bus topology Date: Thu, 23 Sep 2021 15:55:00 +0300 Message-Id: <20210923125501.234252-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1632402533890100001 Content-Type: text/plain; charset="utf-8" From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- New in v2 --- xen/common/domain.c | 1 + xen/drivers/passthrough/pci.c | 57 +++++++++++++++++++++++++++++++++++ xen/drivers/vpci/vpci.c | 18 +++++++++-- xen/include/xen/pci.h | 18 +++++++++++ xen/include/xen/sched.h | 6 ++++ 5 files changed, 97 insertions(+), 3 deletions(-) diff --git a/xen/common/domain.c b/xen/common/domain.c index 40d67ec34232..b80ff2e5e2e6 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -601,6 +601,7 @@ struct domain *domain_create(domid_t domid, =20 #ifdef CONFIG_HAS_PCI INIT_LIST_HEAD(&d->pdev_list); + INIT_LIST_HEAD(&d->vdev_list); #endif =20 /* All error paths can depend on the above setup. */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index e1da283d73ad..4552ace855e0 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -833,6 +833,63 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) return ret; } =20 +static struct vpci_dev *pci_find_virtual_device(const struct domain *d, + const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + list_for_each_entry ( vdev, &d->vdev_list, list ) + if ( vdev->pdev =3D=3D pdev ) + return vdev; + return NULL; +} + +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + ASSERT(!pci_find_virtual_device(d, pdev)); + + /* Each PCI bus supports 32 devices/slots at max. */ + if ( d->vpci_dev_next > 31 ) + return -ENOSPC; + + vdev =3D xzalloc(struct vpci_dev); + if ( !vdev ) + return -ENOMEM; + + /* We emulate a single host bridge for the guest, so segment is always= 0. */ + *(u16*) &vdev->seg =3D 0; + /* + * The bus number is set to 0, so virtual devices are seen + * as embedded endpoints behind the root complex. + */ + *((u8*) &vdev->bus) =3D 0; + *((u8*) &vdev->devfn) =3D PCI_DEVFN(d->vpci_dev_next++, 0); + + vdev->pdev =3D pdev; + vdev->domain =3D d; + + pcidevs_lock(); + list_add_tail(&vdev->list, &d->vdev_list); + pcidevs_unlock(); + + return 0; +} + +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + pcidevs_lock(); + vdev =3D pci_find_virtual_device(d, pdev); + if ( vdev ) + list_del(&vdev->list); + pcidevs_unlock(); + xfree(vdev); + return 0; +} + /* Caller should hold the pcidevs_lock */ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, uint8_t devfn) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 0cdc0c3f75c4..a40a138a14f7 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -90,24 +90,36 @@ int __hwdom_init vpci_add_handlers(struct pci_dev *pdev) /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, const struct pci_dev *dev) { +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + int rc; +#endif + /* It only makes sense to assign for hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; =20 #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT - return vpci_bar_add_handlers(d, dev); -#else - return 0; + rc =3D vpci_bar_add_handlers(d, dev); + if ( rc ) + return rc; #endif + + return pci_add_virtual_device(d, dev); } =20 /* Notify vPCI that device is de-assigned from guest. */ int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) { + int rc; + /* It only makes sense to de-assign from hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; =20 + rc =3D pci_remove_virtual_device(d, dev); + if ( rc ) + return rc; + #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT return vpci_bar_remove_handlers(d, dev); #else diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 2b2dfb6f1b49..35ae1d093921 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -136,6 +136,22 @@ struct pci_dev { struct vpci *vpci; }; =20 +struct vpci_dev { + struct list_head list; + /* Physical PCI device this virtual device is connected to. */ + const struct pci_dev *pdev; + /* Virtual SBDF of the device. */ + const union { + struct { + uint8_t devfn; + uint8_t bus; + uint16_t seg; + }; + pci_sbdf_t sbdf; + }; + struct domain *domain; +}; + #define for_each_pdev(domain, pdev) \ list_for_each_entry(pdev, &(domain)->pdev_list, domain_list) =20 @@ -165,7 +181,9 @@ int pci_add_segment(u16 seg); const unsigned long *pci_get_ro_map(u16 seg); int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *, nodeid_t node); +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev); int pci_remove_device(u16 seg, u8 bus, u8 devfn); +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev= ); int pci_ro_device(int seg, int bus, int devfn); int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn= ); struct pci_dev *pci_get_pdev(int seg, int bus, int devfn); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 28146ee404e6..d304c7ebe766 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,12 @@ struct domain =20 #ifdef CONFIG_HAS_PCI struct list_head pdev_list; + struct list_head vdev_list; + /* + * Current device number used by the virtual PCI bus topology + * to assign a unique SBDF to a passed through virtual PCI device. + */ + int vpci_dev_next; #endif =20 #ifdef CONFIG_HAS_PASSTHROUGH --=20 2.25.1