From nobody Sat May 11 12:44:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1683669679; cv=none; d=zohomail.com; s=zohoarc; b=aBgUb2N55DWSNh8vcpBaNqHkjY66L2tPoJr5KSevnoKGYuCVDawVb13If1iNoytOrEPleJl+ABMLBlJ8eWvFRRZaBCFzqLNEVOUfZsqGf/66pu+VnC4RMAekDpG9VQPxZUnbS5PtBat3zbTRyraqXTxxBROMdK0yj5I8gJ69HPM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683669679; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=CwClyvExvhUOcjO77Dya0a7V972SdH0wCG7mSU3dEQ8=; b=FusGPMOUQQWUvU5scVdiJIPNslZwl+T49kwI6hoJ6A2sVkp2yhJqxR+JbM1I8z6U7fsqWfoexoWpH5EKJUp2wK+NbHXJd/15xA+lMhMX1qVv9fNgvTYjkt1LRwInayLJX7+vqhMnEPWbt/9Atjo8RklMDW1c+obEmA4DE1NCHhg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683669679344636.7705101485606; Tue, 9 May 2023 15:01:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwVNo-0004NU-S5; Tue, 09 May 2023 18:00:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNh-0004L9-6C for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNf-0003e7-5n for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:24 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-111-p_66oQo3PZipr3LCaqAdQQ-1; Tue, 09 May 2023 18:00:18 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 17AEB886461; Tue, 9 May 2023 22:00:18 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.18.185]) by smtp.corp.redhat.com (Postfix) with ESMTP id 332462166B26; Tue, 9 May 2023 22:00:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683669621; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CwClyvExvhUOcjO77Dya0a7V972SdH0wCG7mSU3dEQ8=; b=bh0XpjL2FvEAAPVEBG4dISr3jFyVU0rGJs6HDA3BAmxbG3GMkmUbUoQC2vKdqXf3jxpH5K 373+nRgUYG5hsFEoIel+1U/Ylf2lbgPVgeUYu4gVvl2C9bVrLc71EBT/Qoy9vaar9QEXrz YT6sL1LiRmkTdZutaVkpqF4isW+IjJs= X-MC-Unique: p_66oQo3PZipr3LCaqAdQQ-1 From: Alex Williamson To: qemu-devel@nongnu.org Cc: Minwoo Im , Alex Williamson , Klaus Jensen Subject: [PULL 1/3] vfio/pci: add support for VF token Date: Tue, 9 May 2023 15:59:21 -0600 Message-Id: <20230509215923.3186420-2-alex.williamson@redhat.com> In-Reply-To: <20230509215923.3186420-1-alex.williamson@redhat.com> References: <20230509215923.3186420-1-alex.williamson@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1683669681566100007 Content-Type: text/plain; charset="utf-8" From: Minwoo Im VF token was introduced [1] to kernel vfio-pci along with SR-IOV support [2]. This patch adds support VF token among PF and VF(s). To passthu PCIe VF to a VM, kernel >=3D v5.7 needs this. It can be configured with UUID like: -device vfio-pci,host=3DDDDD:BB:DD:F,vf-token=3D,... [1] https://lore.kernel.org/linux-pci/158396393244.5601.1029743072496402575= 3.stgit@gimli.home/ [2] https://lore.kernel.org/linux-pci/158396044753.5601.1480487068117478970= 9.stgit@gimli.home/ Cc: Alex Williamson Signed-off-by: Minwoo Im Reviewed-by: Klaus Jensen Link: https://lore.kernel.org/r/20230320073522epcms2p48f682ecdb73e0ae1a4850= ad0712fd780@epcms2p4 Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 13 ++++++++++++- hw/vfio/pci.h | 1 + 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ec9a854361ac..cf27f28936cb 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2856,6 +2856,8 @@ static void vfio_realize(PCIDevice *pdev, Error **err= p) int groupid; int i, ret; bool is_mdev; + char uuid[UUID_FMT_LEN]; + char *name; =20 if (!vbasedev->sysfsdev) { if (!(~vdev->host.domain || ~vdev->host.bus || @@ -2936,7 +2938,15 @@ static void vfio_realize(PCIDevice *pdev, Error **er= rp) goto error; } =20 - ret =3D vfio_get_device(group, vbasedev->name, vbasedev, errp); + if (!qemu_uuid_is_null(&vdev->vf_token)) { + qemu_uuid_unparse(&vdev->vf_token, uuid); + name =3D g_strdup_printf("%s vf_token=3D%s", vbasedev->name, uuid); + } else { + name =3D vbasedev->name; + } + + ret =3D vfio_get_device(group, name, vbasedev, errp); + g_free(name); if (ret) { vfio_put_group(group); goto error; @@ -3268,6 +3278,7 @@ static void vfio_instance_init(Object *obj) =20 static Property vfio_pci_dev_properties[] =3D { DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), + DEFINE_PROP_UUID_NODEFAULT("vf-token", VFIOPCIDevice, vf_token), DEFINE_PROP_STRING("sysfsdev", VFIOPCIDevice, vbasedev.sysfsdev), DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevic= e, vbasedev.pre_copy_dirty_page_tracking, diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 177abcc8fb67..2674476d6c77 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -137,6 +137,7 @@ struct VFIOPCIDevice { VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; PCIHostDeviceAddress host; + QemuUUID vf_token; EventNotifier err_notifier; EventNotifier req_notifier; int (*resetfn)(struct VFIOPCIDevice *); --=20 2.39.2 From nobody Sat May 11 12:44:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1683669705; cv=none; d=zohomail.com; s=zohoarc; b=GP75viafU0LPULpFT0tI22/iHPNbl76JBvEjDeLrK3cVKo39WsqXW0fjWsBSTyV09o6UqP0alFXeyXv7WdFdjRtQ6hq28wC3jJS8vBSSNXm7BTXtt/O7bIysidKzerTODAnnCAYjcix0fUUr1RHNwp/MLglNDDGvttIbFUi1dl0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683669705; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=8ihVEjBV3U5cCb9uxrle17sGu8lcKCiiO/F9zdoLXP8=; b=i6DDq3Dtd9lHANBxpKhokzQDrC5mPJ4BzA4N5uzzDoandZvcJnLqnlgpbRqP96eXSIxrwiwwrqPVq70Ur2+NC9VvSG3QJiPH4Y6esNUBjSEmgJFm3RGIJatveC9anGsBPp4cHUvXaraR8/1TEIT4yvbpsQRevvI1+Spn+ipGSAc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683669705718245.27971286092884; Tue, 9 May 2023 15:01:45 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwVNq-0004Ni-Be; Tue, 09 May 2023 18:00:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNl-0004N4-2a for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:29 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNg-0003ez-9K for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:26 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-269-DVmswm3MNV-QLg8A4gVv3Q-1; Tue, 09 May 2023 18:00:20 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 42A1F857E61; Tue, 9 May 2023 22:00:20 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.18.185]) by smtp.corp.redhat.com (Postfix) with ESMTP id 042002166B26; Tue, 9 May 2023 22:00:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683669623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ihVEjBV3U5cCb9uxrle17sGu8lcKCiiO/F9zdoLXP8=; b=Si3dBzfBeplcdlep3h1F4ZxSscADXBuoB6v2HrV3Usr825pneLIBUP7MaH5JHQqGJf5jAE OFJpLvKQXU4wJUdTHMcNwZnrtmRFn0dFRQVBirCaP9xAg3c0ULzC0CsrlosE8naytaj2gi ojy2YwF9bqBWtr9puJwhy+PtTPCbdRQ= X-MC-Unique: DVmswm3MNV-QLg8A4gVv3Q-1 From: Alex Williamson To: qemu-devel@nongnu.org Cc: Avihai Horon , Alex Williamson Subject: [PULL 2/3] vfio/migration: Skip log_sync during migration SETUP state Date: Tue, 9 May 2023 15:59:22 -0600 Message-Id: <20230509215923.3186420-3-alex.williamson@redhat.com> In-Reply-To: <20230509215923.3186420-1-alex.williamson@redhat.com> References: <20230509215923.3186420-1-alex.williamson@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1683669706988100003 Content-Type: text/plain; charset="utf-8" From: Avihai Horon Currently, VFIO log_sync can be issued while migration is in SETUP state. However, doing this log_sync is at best redundant and at worst can fail. Redundant -- all RAM is marked dirty in migration SETUP state and is transferred only after migration is set to ACTIVE state, so doing log_sync during migration SETUP is pointless. Can fail -- there is a time window, between setting migration state to SETUP and starting dirty tracking by RAM save_live_setup handler, during which dirty tracking is still not started. Any VFIO log_sync call that is issued during this time window will fail. For example, this error can be triggered by migrating a VM when a GUI is active, which constantly calls log_sync. Fix it by skipping VFIO log_sync while migration is in SETUP state. Fixes: 758b96b61d5c ("vfio/migrate: Move switch of dirty tracking into vfio= _memory_listener") Signed-off-by: Avihai Horon Link: https://lore.kernel.org/r/20230403130000.6422-1-avihaih@nvidia.com Signed-off-by: Alex Williamson --- hw/vfio/common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 4d01ea351515..78358ede2764 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -478,7 +478,8 @@ static bool vfio_devices_all_dirty_tracking(VFIOContain= er *container) VFIODevice *vbasedev; MigrationState *ms =3D migrate_get_current(); =20 - if (!migration_is_setup_or_active(ms->state)) { + if (ms->state !=3D MIGRATION_STATUS_ACTIVE && + ms->state !=3D MIGRATION_STATUS_DEVICE) { return false; } =20 --=20 2.39.2 From nobody Sat May 11 12:44:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1683669711; cv=none; d=zohomail.com; s=zohoarc; b=mFng0x/Y4oj5OmPTewmOn77N9F95JZZG7696aS07KPjBTEre/lx5mmnRwTfct5R3Od6KNOmele1NAwof4EZkdDv/W3dA90l6Qqvx5F3ym/OCmvN5G4qxOb32FhGJjghfVbu1Az+T19L8h33oBEEOFk//p+5IvnabdgwbyiinHVc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683669711; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TWU5I3m/v9fM0xCSwQq11XJvPmPoMyRIcKp6zQGroCI=; b=YE2amPIoLmTsXheS42fdgVKqVzinfB4vTqxM9PRkIXmNqSc4HZr5N+KSsNVuWTE3kH5Qo7+VRACm5wpEIxks1aotIxfRMysRSWosGdtXykR6P/8D27XBXzkDqaGAscriwjSrCDpQpJ2NhLbFRqjJhsNh8qrH40Rl4iNks/5SX+U= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683669711231273.82314435361786; Tue, 9 May 2023 15:01:51 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwVNt-0004OM-Ai; Tue, 09 May 2023 18:00:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNl-0004N5-3E for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:29 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwVNg-0003f8-Sz for qemu-devel@nongnu.org; Tue, 09 May 2023 18:00:27 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-439-4xPctDaiMJiuOlm9SS0EYg-1; Tue, 09 May 2023 18:00:22 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 60C83185A7A2 for ; Tue, 9 May 2023 22:00:22 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.18.185]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FE612166B26; Tue, 9 May 2023 22:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683669623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TWU5I3m/v9fM0xCSwQq11XJvPmPoMyRIcKp6zQGroCI=; b=UiwaO9VzGKZqO1xgv8ptgP6wIrUB9CjLbQQhsIxKbxjUu6WcacWZvEz4CTTXs3oQOXZsD5 zySCMBv3lmxm6gDXNMw+UrcFrwLjN0Mn2i+AQ8ZtMvsIhUvxG8OmTPgVtCBHngH9/k7o8r kXo2EOQGf1sar3HcHOTHOc5Q0MqvKsA= X-MC-Unique: 4xPctDaiMJiuOlm9SS0EYg-1 From: Alex Williamson To: qemu-devel@nongnu.org Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 3/3] vfio/pci: Static Resizable BAR capability Date: Tue, 9 May 2023 15:59:23 -0600 Message-Id: <20230509215923.3186420-4-alex.williamson@redhat.com> In-Reply-To: <20230509215923.3186420-1-alex.williamson@redhat.com> References: <20230509215923.3186420-1-alex.williamson@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1683669713025100003 The PCI Resizable BAR (ReBAR) capability is currently hidden from the VM because the protocol for interacting with the capability does not support a mechanism for the device to reject an advertised supported BAR size. However, when assigned to a VM, the act of resizing the BAR requires adjustment of host resources for the device, which absolutely can fail. Linux does not currently allow us to reserve resources for the device independent of the current usage. The only writable field within the ReBAR capability is the BAR Size register. The PCIe spec indicates that when written, the device should immediately begin to operate with the provided BAR size. The spec however also notes that software must only write values corresponding to supported sizes as indicated in the capability and control registers. Writing unsupported sizes produces undefined results. Therefore, if the hypervisor were to virtualize the capability and control registers such that the current size is the only indicated available size, then a write of anything other than the current size falls into the category of undefined behavior, where we can essentially expose the modified ReBAR capability as read-only. This may seem pointless, but users have reported that virtualizing the capability in this way not only allows guest software to expose related features as available (even if only cosmetic), but in some scenarios can resolve guest driver issues. Additionally, no regressions in behavior have been reported for this change. A caveat here is that the PCIe spec requires for compatibility that devices report support for a size in the range of 1MB to 512GB, therefore if the current BAR size falls outside that range we revert to hiding the capability. Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/r/20230505232308.2869912-1-alex.williamson@re= dhat.com Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index cf27f28936cb..bf27a3990564 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2066,6 +2066,54 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uin= t8_t pos, Error **errp) return 0; } =20 +static int vfio_setup_rebar_ecap(VFIOPCIDevice *vdev, uint16_t pos) +{ + uint32_t ctrl; + int i, nbar; + + ctrl =3D pci_get_long(vdev->pdev.config + pos + PCI_REBAR_CTRL); + nbar =3D (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIF= T; + + for (i =3D 0; i < nbar; i++) { + uint32_t cap; + int size; + + ctrl =3D pci_get_long(vdev->pdev.config + pos + PCI_REBAR_CTRL + (= i * 8)); + size =3D (ctrl & PCI_REBAR_CTRL_BAR_SIZE) >> PCI_REBAR_CTRL_BAR_SH= IFT; + + /* The cap register reports sizes 1MB to 128TB, with 4 reserved bi= ts */ + cap =3D size <=3D 27 ? 1U << (size + 4) : 0; + + /* + * The PCIe spec (v6.0.1, 7.8.6) requires HW to support at least o= ne + * size in the range 1MB to 512GB. We intend to mask all sizes ex= cept + * the one currently enabled in the size field, therefore if it's + * outside the range, hide the whole capability as this virtualiza= tion + * trick won't work. If >512GB resizable BARs start to appear, we + * might need an opt-in or reservation scheme in the kernel. + */ + if (!(cap & PCI_REBAR_CAP_SIZES)) { + return -EINVAL; + } + + /* Hide all sizes reported in the ctrl reg per above requirement. = */ + ctrl &=3D (PCI_REBAR_CTRL_BAR_SIZE | + PCI_REBAR_CTRL_NBAR_MASK | + PCI_REBAR_CTRL_BAR_IDX); + + /* + * The BAR size field is RW, however we've mangled the capability + * register such that we only report a single size, ie. the current + * BAR size. A write of an unsupported value is undefined, theref= ore + * the register field is essentially RO. + */ + vfio_add_emulated_long(vdev, pos + PCI_REBAR_CAP + (i * 8), cap, ~= 0); + vfio_add_emulated_long(vdev, pos + PCI_REBAR_CTRL + (i * 8), ctrl,= ~0); + } + + return 0; +} + static void vfio_add_ext_cap(VFIOPCIDevice *vdev) { PCIDevice *pdev =3D &vdev->pdev; @@ -2139,9 +2187,13 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev) case 0: /* kernel masked capability */ case PCI_EXT_CAP_ID_SRIOV: /* Read-only VF BARs confuse OVMF */ case PCI_EXT_CAP_ID_ARI: /* XXX Needs next function virtualization= */ - case PCI_EXT_CAP_ID_REBAR: /* Can't expose read-only */ trace_vfio_add_ext_cap_dropped(vdev->vbasedev.name, cap_id, ne= xt); break; + case PCI_EXT_CAP_ID_REBAR: + if (!vfio_setup_rebar_ecap(vdev, next)) { + pcie_add_capability(pdev, cap_id, cap_ver, next, size); + } + break; default: pcie_add_capability(pdev, cap_id, cap_ver, next, size); } --=20 2.39.2