From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201380; cv=none; d=zohomail.com; s=zohoarc; b=V9mOr9GHxc8g6gon+vYTl5NR+wJ5n3x0xQMp1QDzi2MB+uzipgkGwdiTtdAaAKcu+rBHZLtENQMXh5DQASO5EeGGqM+STaVLmwuz2ixWSPIM4ifOwpZhaCpD1Ul4l8bsNfFuXKAE9LxnpO3Q6Ls2AAtdkDxdBM0MlKePU4e/JJw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201380; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=JXW8+TkkbO/zdqSD0tRYgROfejMENktX+Aqzp4oymc4=; b=kBF3Ig6j+5HJ5TlBRQoh/gRBvLykU3VB+JfBhCKjiNfMPapfSQXPN1plS9NeJrcTSF6AncvPU3fcX0bfR7L1aZSltb2iQyemWgmJtD0/pYePhchzPpcVmtCRKbpLpxSPZ96Krot3neS9fIAL9ZM53VOO4YtiRyGWAsysdtJY2VI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201380917472.4238262367504; Thu, 11 Jun 2026 11:09:40 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpk-0006zl-Lx; Thu, 11 Jun 2026 14:08:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpj-0006yW-1P for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:51 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpf-0002FQ-L2 for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:50 -0400 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-490be03d47bso1269735e9.0 for ; Thu, 11 Jun 2026 11:08:46 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201325; x=1781806125; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JXW8+TkkbO/zdqSD0tRYgROfejMENktX+Aqzp4oymc4=; b=Q3Gvyof7JJ4mEldhfO5aPt7O93qyNMLt8lvG7Kej1v3bwqEVHo08JO9N8JuEQjlPA8 GrnoUurl8NbVNHiPMjvccJTFCGrB+NkI7cMCZI0VZEuoCm7vheU0K1QJd6GelpZIM3i4 HpZvWDsNmMiO1sgmQeegH0xQu2MEFs38f/fH8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201325; x=1781806125; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=JXW8+TkkbO/zdqSD0tRYgROfejMENktX+Aqzp4oymc4=; b=Xx1s8D/Cfwplxdvw/iq28oAFmtdpeGA6wPab/I2U5mDZCFK4ufejATBNN1al5VDpVk UtGn6ABnz3ko9HAOi/S5LAP5QmQo8hi5IcYAQwukO8Yp0wSnItBP/mE5y3yRx8mGFoZr 3zxZRBAd+tT5DB1Inkx/1CRq8P9vpGbl7VXWt1KMLWIiVTOdDZhLCwdsz+2UnmPB0abv rCAR2AYBi9Tc55FtxdCa5dCaPfj2Wp4Qp6f7DEtT9L55XK0PMZby0e7ZAuY1jsnBv7yR x4Xvtye7c2bw51ejfIMSLHkv9GPRg2erg4AZYCsATU01VT0uOunl4CK5BKTrIClB+xkQ Jctg== X-Gm-Message-State: AOJu0YzouGeeMyKIfHx9ikVzzGdBhXXiQesT6tmZm+S5nHdJlyaEbhYV CBtU2sB6xZhXmuMuflUdHmhfhlBwHglsRs/l9GjCrv3LinNgxqlaJg+RkPvLwuCMIawlQx6C6s7 qLxbhPu0= X-Gm-Gg: Acq92OGq79r+zz9ArD1Oofga86hlgCmVGDbKtRJcxgwwiXirlpfC5S+RfwLbieNlC62 geU6IK9bN+qwc4Dejjt1gXQYbNDzYB1B7dhiC0sfGqaOSOIdQTyY1ACdy1pM8dxK4y/EK3AlIKg ifDJzE8lDi39PV461EqnGQ6q/31Ad11Gz2SNn6qczJ+2OLbATpEgYWv46tP6jZgjB6A97Y5Eob5 G2ZH3OLsbDN98O8KiMr7WAnEyTSrhu8f+YAvz+h+e17KMZhjX1Zq53XEly0TCbXyJPl6iLAcoCs g1x4oV6yYiVefKyUOJT0K3WUuaf2W+RfSi7fpjTtu37Do+Sr9K7fHqzkfRLLtVOD1HkYD+FNEpH sL5bFh+xMf/F4zg02/ZJiQALiQUGeKmjiVDzRBfRJNQcxVOOsu95xwbS+6t+oKOl03zuqXopQP3 7flem7+P1ngfuLJWjM3UCR5F2dXedTdZG1OHlFCriIz7ieyoGc583YQHHnDPOzRH8ueM9EO4p6L xndoPBKYvt87PnDCQoRuVA= X-Received: by 2002:a05:600c:8682:b0:490:c2a3:3303 with SMTP id 5b1f17b1804b1-490e56248d5mr35200145e9.34.1781201324811; Thu, 11 Jun 2026 11:08:44 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn Subject: [PATCH v10 1/8] tests/functional/migration: add VM launch/configure hooks Date: Thu, 11 Jun 2026 20:08:34 +0200 Message-ID: <20260611180842.6390-2-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x333.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201383029158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce configure_machine, launch_source_vm and assert_dest_vm methods to allow child classes to override some pieces of source/dest VMs creation, start and check logic. Reviewed-by: Peter Xu Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- tests/functional/migration.py | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/tests/functional/migration.py b/tests/functional/migration.py index 3b7674af3b6..4344e03be41 100644 --- a/tests/functional/migration.py +++ b/tests/functional/migration.py @@ -40,19 +40,36 @@ def assert_migration(self, src_vm, dst_vm): self.assertEqual(dst_vm.cmd('query-status')['status'], 'running') self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate= ') =20 + # Can be overridden by subclasses to configure both source/dest VMs. + def configure_machine(self, vm): + vm.add_args('-nodefaults') + + # Can be overridden by subclasses to prepare the source VM before + # migration, e.g. by running some workload inside the source VM + # to see if it continues to run properly after migration. + def launch_source_vm(self, vm): + vm.launch() + + # Can be overridden by subclasses to check the destination VM after + # migration, e.g. by checking if the workload is still running after + # migration. + def assert_dest_vm(self, vm): + pass + def migrate_vms(self, dst_uri, src_uri, dst_vm, src_vm): dst_vm.qmp('migrate-incoming', uri=3Ddst_uri) src_vm.qmp('migrate', uri=3Dsrc_uri) self.assert_migration(src_vm, dst_vm) + self.assert_dest_vm(dst_vm) =20 def migrate(self, dst_uri, src_uri=3DNone): dst_vm =3D self.get_vm('-incoming', 'defer', name=3D"dst-qemu") - dst_vm.add_args('-nodefaults') + self.configure_machine(dst_vm) dst_vm.launch() =20 src_vm =3D self.get_vm(name=3D"src-qemu") - src_vm.add_args('-nodefaults') - src_vm.launch() + self.configure_machine(src_vm) + self.launch_source_vm(src_vm) =20 if src_uri is None: src_uri =3D dst_uri --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201439; cv=none; d=zohomail.com; s=zohoarc; b=mrkMY0gaIPDKZd+HdCR3kUaPJodH4cEI97ldIGV+8STAFArkvuu4mljxyrAAPG3+UANZMqqb7hgkb7yuniIzo50MjZKAgclakjqIAuIBQhAjtdZsmsC5ALl7Jd4uEsiZzcQky4NsuoBfRSbbWVg24v6Csfk1qAoUg2/gXv4N6G4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201439; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=0TDXVKCogGTId0T5xX3GpZdgySNzo5x6/mGX55G7OmA=; b=bYyDx9qYwOO4Ejjik8oraTojRXcsecA/WzL3hp+MdGwhdGW5BsZ28hGYTeRUrxvTM6LWrszpBy7ofJL9SVQBj4Zkzyhwquzw4Q9jw+dgtJABS8KepvPLCX9ymfJykgXaYDI+Scxmu01+AaP8M0DKOvdh5XMMk5rkeXfI3rmgCuk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201439968353.58619482034703; Thu, 11 Jun 2026 11:10:39 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpm-00071A-B3; Thu, 11 Jun 2026 14:08:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpj-0006yv-Oi for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:51 -0400 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpg-0002Fn-4I for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:51 -0400 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-490a76757e5so720215e9.2 for ; Thu, 11 Jun 2026 11:08:47 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201326; x=1781806126; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0TDXVKCogGTId0T5xX3GpZdgySNzo5x6/mGX55G7OmA=; b=X+54Lo4/JWboVxI04e3pJhIJRLGpQW57sRcsGU15YEOzLQA2K+T5LCA/V1R2ooV80n ExV1jNBBU24//gxNqgOn3QsUEfRXDKr7gu484aQV2uhprecOdOds00gtB4DzUSqv+g+2 nLauKQtdq38SGoLQRrUk5gZo1ebsjkwXg1PTM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201326; x=1781806126; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=0TDXVKCogGTId0T5xX3GpZdgySNzo5x6/mGX55G7OmA=; b=gopnZCH8ES7vS7kaLZmy10Tk3HVybOyS1amBIDovCwXGc9ZWireLAsFsiPomHBCY6w DNDInMdfOYmbEKPzfJ7Fw9kkSHseENWurzpsmGQvYklVfY4YsyDBqC9fVkdBY6tz8VOv fTXh1eiKxEJ5tryAX3mK4Wj4q6bWdhDQL48VmjhkIJn2lc3943qZnRDERs+z77Fiw70z 9nhsxlMrOodA47VsgxhGnz9U/sIcXFaD4UbY40PnCJeDVvu0TMhNr2z7Ap/gY8aiH5zj 0qZ7x6uWloGMcgNhlXvigEHummpFZ9pomsX9kYgEYjmAP+PO+JuchzkaBo5eWMkzMqsd y+Ag== X-Gm-Message-State: AOJu0YyyL/mjkeKF2QJF5aFXRtrwHxyYXRFSwsb34qFXYON3QmpAkra4 4HkY69d+5HhtZ/bLBC53HOBV1DF7PaI6LxtxwA83b109mW8l5MJOp0l75W/E+QhcEWg9RMhgCUi mYc0y8gc= X-Gm-Gg: Acq92OFPnS7SxAUjfRJjoACg7/TqhfP+rQSXj+YnJDtbKkwAItRZNp5j3ia+Z4gyN+t V8Zs54NApNMmSilOv8i+JZnQ7fTENHaKoYPv5QIuT8Rjz1x9RDWr6wnCnhb+8ViuVBCg6mCbyEx ylWhxsj5vdxTDoK3Q9ZIPErkOInZvCIbE3nuTl6YeA3kZ3CXw5rtVHxLPKfk6sJHRTA2cehdGIb vxce1M+KaCbRTviFyMO1QdUWF6ygpth8TOkEpmcvGUAP/4Icng5mqpKed1SbkcOIKVG6fvBATee rqY3XDIL4zhJt9uOIh2qaAtAB3fYjwkx96UClA41wWkvDjRjp7cPvoy3o46A3y39iy1r+7ab2pV alwBbysJzkfsAvTzPGT/YVnIUzqXz8pfR8+l9G+/zLhnEisOi9eWaTbykkP1J3hYzcz0+uzGu3Q UdciVuAQ+5Pm1mqDs96KTwFnGnodjeflK9eFYFPRtc2NtgtytjGjTYDbAG3upKo+wqhCj2gbmVy huW/kvlaty/s+2PFhU42Ks= X-Received: by 2002:a05:600c:3f1b:b0:490:b642:ce31 with SMTP id 5b1f17b1804b1-490e55b7020mr54298925e9.2.1781201325963; Thu, 11 Jun 2026 11:08:45 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v10 2/8] hw/nvme: add migration blockers for non-supported cases Date: Thu, 11 Jun 2026 20:08:35 +0200 Message-ID: <20260611180842.6390-3-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::334; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x334.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201440965158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Let's block migration for cases we don't support: - SR-IOV - CMB - PMR - SPDM No functional changes here, because NVMe migration is not supported at all as of this commit. Reviewed-by: Klaus Jensen Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- v10: - fix "../hw/nvme/ctrl.c:9385:13: error: label followed by a declaratio= n " [ thanks to Fabiano Rosas ] v9: - check-patch trivial fixes migration blockers fixup --- hw/nvme/ctrl.c | 212 +++++++++++++++++++++++++++++++++++++++++++ hw/nvme/nvme.h | 3 + include/block/nvme.h | 12 +++ 3 files changed, 227 insertions(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 815f39173c8..b4a9728e06b 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -209,6 +209,7 @@ #include "hw/pci/msix.h" #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" +#include "migration/blocker.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -252,6 +253,7 @@ static const bool nvme_feature_support[NVME_FID_MAX] = =3D { [NVME_COMMAND_SET_PROFILE] =3D true, [NVME_FDP_MODE] =3D true, [NVME_FDP_EVENTS] =3D true, + /* if you add something here, please update nvme_set_migration_blocker= s() */ }; =20 static const uint32_t nvme_feature_cap[NVME_FID_MAX] =3D { @@ -4603,6 +4605,7 @@ static uint16_t nvme_io_mgmt_send(NvmeCtrl *n, NvmeRe= quest *req) return 0; case NVME_IOMS_MO_RUH_UPDATE: return nvme_io_mgmt_send_ruh_update(n, req); + /* if you add something here, please update nvme_set_migration_blocker= s() */ default: return NVME_INVALID_FIELD | NVME_DNR; }; @@ -7522,6 +7525,10 @@ static uint16_t nvme_security_receive(NvmeCtrl *n, N= vmeRequest *req) =20 static uint16_t nvme_directive_send(NvmeCtrl *n, NvmeRequest *req) { + /* + * When adding a new dtype handling here, + * please also update nvme_set_migration_blockers(). + */ return NVME_INVALID_FIELD | NVME_DNR; } =20 @@ -9233,6 +9240,205 @@ static void nvme_init_ctrl(NvmeCtrl *n, PCIDevice *= pci_dev) } } =20 +#define BLOCKER_FEATURES_MAX_LEN 256 + +static inline void nvme_add_blocker_feature(char *blocker_features, + const char *feature) +{ + if (strlen(blocker_features) > 0) { + g_strlcat(blocker_features, ", ", BLOCKER_FEATURES_MAX_LEN); + } + g_strlcat(blocker_features, feature, BLOCKER_FEATURES_MAX_LEN); +} + +static bool nvme_set_migration_blockers(NvmeCtrl *n, PCIDevice *pci_dev, + Error **errp) +{ + uint64_t unsupported_cap, cap =3D ldq_le_p(&n->bar.cap); + char blocker_features[BLOCKER_FEATURES_MAX_LEN] =3D ""; + bool adm_cmd_security_checked =3D false; + bool cmd_io_mgmt_checked =3D false; + bool cmd_zone_checked =3D false; + + /* + * Idea of this function is simple, we iterate over all Command Sets a= nd + * for each supported command we provide a special handling logic to + * determine if we should block migration or not. + * + * For instance, we have NVME_ADM_CMD_NS_ATTACHMENT and it is always + * available to the guest, but if there is only 1 namespace, then it is + * safe to allow migration, but if there are more, then we need to blo= ck + * migration because we don't handle this in migration code yet. + */ + for (int opcode =3D 0; opcode < ARRAY_SIZE(n->cse.acs); opcode++) { + /* Is command supported? */ + if (!n->cse.acs[opcode]) { + continue; + } + + switch (opcode) { + case NVME_ADM_CMD_DELETE_SQ: + case NVME_ADM_CMD_CREATE_SQ: + case NVME_ADM_CMD_GET_LOG_PAGE: + case NVME_ADM_CMD_DELETE_CQ: + case NVME_ADM_CMD_CREATE_CQ: + case NVME_ADM_CMD_IDENTIFY: + case NVME_ADM_CMD_ABORT: + case NVME_ADM_CMD_SET_FEATURES: + case NVME_ADM_CMD_GET_FEATURES: + case NVME_ADM_CMD_ASYNC_EV_REQ: + case NVME_ADM_CMD_DBBUF_CONFIG: + case NVME_ADM_CMD_FORMAT_NVM: + case NVME_ADM_CMD_DIRECTIVE_SEND: + case NVME_ADM_CMD_DIRECTIVE_RECV: + break; + case NVME_ADM_CMD_NS_ATTACHMENT: { + int namespaces_num =3D 0; + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + namespaces_num++; + } + + if (namespaces_num > 1) { + nvme_add_blocker_feature(blocker_features, + "Namespace Attachment"); + } + + break; + } + case NVME_ADM_CMD_VIRT_MNGMT: + if (n->params.sriov_max_vfs) { + nvme_add_blocker_feature(blocker_features, "SR-IOV"); + } + + break; + case NVME_ADM_CMD_SECURITY_SEND: + case NVME_ADM_CMD_SECURITY_RECV: + if (adm_cmd_security_checked) { + break; + } + + if (pci_dev->spdm_port) { + nvme_add_blocker_feature(blocker_features, "SPDM"); + } + + adm_cmd_security_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < ARRAY_SIZE(n->cse.iocs.nvm); opcode++)= { + if (!n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_FLUSH: + case NVME_CMD_WRITE: + case NVME_CMD_READ: + case NVME_CMD_COMPARE: + case NVME_CMD_WRITE_ZEROES: + case NVME_CMD_DSM: + case NVME_CMD_VERIFY: + case NVME_CMD_COPY: + break; + case NVME_CMD_IO_MGMT_RECV: + case NVME_CMD_IO_MGMT_SEND: + if (cmd_io_mgmt_checked) { + break; + } + + /* check for NVME_IOMS_MO_RUH_UPDATE */ + if (n->subsys->params.fdp.enabled) { + nvme_add_blocker_feature(blocker_features, "FDP"); + } + + cmd_io_mgmt_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < ARRAY_SIZE(n->cse.iocs.zoned); opcode+= +) { + /* + * If command isn't supported or we have the same command + * in n->cse.iocs.nvm, then we can skip it here. + */ + if (!n->cse.iocs.zoned[opcode] || n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_ZONE_APPEND: + case NVME_CMD_ZONE_MGMT_SEND: + case NVME_CMD_ZONE_MGMT_RECV: + if (cmd_zone_checked) { + break; + } + + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + if (ns->params.zoned) { + nvme_add_blocker_feature(blocker_features, + "Zoned Namespace"); + break; + } + } + + cmd_zone_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + /* + * Try our best to explicitly detect all not supported caps, + * to let users know what features cause migration to be blocked, + * but in case we miss handling here, everything else will be + * covered by unsupported_cap check. + */ + if (NVME_CAP_CMBS(cap)) { + nvme_add_blocker_feature(blocker_features, "CMB"); + cap &=3D ~((uint64_t)CAP_CMBS_MASK << CAP_CMBS_SHIFT); + } + + if (NVME_CAP_PMRS(cap)) { + nvme_add_blocker_feature(blocker_features, "PMR"); + cap &=3D ~((uint64_t)CAP_PMRS_MASK << CAP_PMRS_SHIFT); + } + + unsupported_cap =3D cap & ~NVME_MIGRATION_SUPPORTED_CAP_BITS; + if (unsupported_cap) { + nvme_add_blocker_feature(blocker_features, "unknown capability"); + } + + assert(n->migration_blocker =3D=3D NULL); + if (strlen(blocker_features) > 0) { + error_setg(&n->migration_blocker, + "Migration is not supported for %s", blocker_features); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + + return true; +} + static int nvme_init_subsys(NvmeCtrl *n, Error **errp) { int cntlid; @@ -9338,6 +9544,10 @@ static void nvme_realize(PCIDevice *pci_dev, Error *= *errp) =20 n->subsys->namespaces[ns->params.nsid] =3D ns; } + + if (!nvme_set_migration_blockers(n, pci_dev, errp)) { + return; + } } =20 static void nvme_exit(PCIDevice *pci_dev) @@ -9390,6 +9600,8 @@ static void nvme_exit(PCIDevice *pci_dev) } =20 memory_region_del_subregion(&n->bar0, &n->iomem); + + migrate_del_blocker(&n->migration_blocker); } =20 static const Property nvme_props[] =3D { diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 5ef3ebee29e..05aee24a15c 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -668,6 +668,9 @@ typedef struct NvmeCtrl { =20 /* Socket mapping to SPDM over NVMe Security In/Out commands */ int spdm_socket; + + /* Migration-related stuff */ + Error *migration_blocker; } NvmeCtrl; =20 typedef enum NvmeResetType { diff --git a/include/block/nvme.h b/include/block/nvme.h index e4e7be51205..17a7c7818d7 100644 --- a/include/block/nvme.h +++ b/include/block/nvme.h @@ -141,6 +141,18 @@ enum NvmeCapMask { #define NVME_CAP_SET_CMBS(cap, val) \ ((cap) |=3D (uint64_t)((val) & CAP_CMBS_MASK) << CAP_CMBS_SHIFT) =20 +#define NVME_MIGRATION_SUPPORTED_CAP_BITS ( \ + ((uint64_t)CAP_MQES_MASK << CAP_MQES_SHIFT) \ + | ((uint64_t)CAP_CQR_MASK << CAP_CQR_SHIFT) \ + | ((uint64_t)CAP_AMS_MASK << CAP_AMS_SHIFT) \ + | ((uint64_t)CAP_TO_MASK << CAP_TO_SHIFT) \ + | ((uint64_t)CAP_DSTRD_MASK << CAP_DSTRD_SHIFT) \ + | ((uint64_t)CAP_NSSRS_MASK << CAP_NSSRS_SHIFT) \ + | ((uint64_t)CAP_CSS_MASK << CAP_CSS_SHIFT) \ + | ((uint64_t)CAP_MPSMIN_MASK << CAP_MPSMIN_SHIFT) \ + | ((uint64_t)CAP_MPSMAX_MASK << CAP_MPSMAX_SHIFT) \ +) + enum NvmeCapCss { NVME_CAP_CSS_NCSS =3D 1 << 0, NVME_CAP_CSS_IOCSS =3D 1 << 6, --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201391; cv=none; d=zohomail.com; s=zohoarc; b=OuQUtUO3xfLARFI0koD8b5XyYAox/xHSs/tTEdls9uRznt/AhTPCBgLjF1LQyxOk5AsAnVDNN+LILd1qXq2Lx+ihq5Uo2ZEDAx9gDRT597y0ZX3nbK0BYoM+Sad1Eoh9e1l5h1Q4QW/UvPoqlVSB4K5jiwVap1QvTevPX/+5PwM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201391; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Nza7GQuI/kpEISd3D+sr2iIcoZ5Psm5cIZXdAsY0e74=; b=O0Q4gfHTFqNkblEYfckp6xyAYwfHxR9/lbRO5j7RUE5ydBz+VFU+EJok2xyFu8iB9tib+bCxb5tAbwvO2fEvKcvXWq4m9kAlvrLlPc083i2fODkTgoDh66KKPh+3XAGMcFemxVWtsBHTckcRSNp/cbnomS+A87PvLC6wWqBPH5s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201391859978.2297784268445; Thu, 11 Jun 2026 11:09:51 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpn-00071p-26; Thu, 11 Jun 2026 14:08:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpk-0006zy-UK for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:52 -0400 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjph-0002Gr-JX for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:52 -0400 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-490b12270b3so694365e9.1 for ; Thu, 11 Jun 2026 11:08:49 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201328; x=1781806128; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nza7GQuI/kpEISd3D+sr2iIcoZ5Psm5cIZXdAsY0e74=; b=es6sH7piJH0eVoCr/EUOclhMu5nqnDMqNYKkUJgU1Wp5IuH+IQ16mCHz8dG4Hi3ior /toGX3qg6HqBNiGfuQ5CH7kIiJujY9rZCSGwp+gWXxzQR3w2A4pCs85VNDr1rFg0EHDF Z7keUXOInsNQVu/SrTdvCg4G1orMX/pBWut28= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201328; x=1781806128; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Nza7GQuI/kpEISd3D+sr2iIcoZ5Psm5cIZXdAsY0e74=; b=IwR6G7IaIVMUlC/h2aODcJyuoRap5hesBLsxwQKZADs/gZ5ZA4IndSwGEey6C/xEw9 gxwc6xEbLH7Jv0q+K2MV10wt9J1XijD92xXF4Z4MfmiN3rbWYm8XslQC2hbiQ+IE2+K4 WL5qcaAjyeF0kdPPOHaeluia8Gt7nu00JDPDRdaLEWrM4YeRIyxwJySC6g0o70ta2yAs n7+aOhxRhLqnRILtW28pVxhcDWV/wznVt/mh25BIubu5b/6vMwh6CfiB0p6eXju/V7gK frxVCxZPQ5oGdhm+iadGcnp2HWsGn5BNzDvepoiOZFqNB++kj6J6cecFj6oiDaGDiQYd vFJw== X-Gm-Message-State: AOJu0Yy0hS5ujryRZCxSR44XEbKFYZvsb+W48wlWvGo7r2NpFXh0gsgO D+w2IUwEE4C75DS/uw6qKKGJH4tFtv5S2Dt9sS7DbR3qT7ULSUVsoxYgCb6r15ClvN7y71/GAZB PMweZMoo= X-Gm-Gg: Acq92OEqmtdAdEXCgAZJt4RGPurLyqkizkOQuPB3BjJuLcfO04V8Qwgcgk4hHIsdc4f UTXW0QBj8zhaBezbXizxNKyB377lu+77G9vLw4H+ymJ+qrODdRIlVi1A9G+slxBj0IUjJvHaJeL ICPlCiGKNsdsw+68PnFmMtUR6mahVw+yTz++HgjgUe+pJlPmc7yar23qSDbE6akUgAuIpL5tYjW DTZYygnVooomtslazJWqzs6BLkhPlbSRKwHuRqGXbesnbfQPuOp1ntPOmtm2RHW/NokaUh45ZT9 dDDaN8cLGWyfxYtxE6mK0pQ3cyl9buWF6Hd6nyagdblsJ9iw8jdW8f1GmXHKrP0eu8nLqq8dKVU dx3qyZITDmpKdIw3Otwbv7WceAhLJS2qqrMhoaA43Y4ODQCisOLDoO/WVDHDKbN1CdHlpnVDX8O gTb6iY1aLs/6J9qajOO75mwsH9MrRGpF+xxEfNW9I7D4IVw3V3W9kd0pn7pRncDcJey29igSaOx 78uN5m/mpjvcFRijlUM7devGNJP+x8ryA== X-Received: by 2002:a05:600c:4fd0:b0:490:be78:1861 with SMTP id 5b1f17b1804b1-490e55a5d64mr51998895e9.4.1781201327069; Thu, 11 Jun 2026 11:08:47 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v10 3/8] hw/nvme: split nvme_init_sq/nvme_init_cq into helpers Date: Thu, 11 Jun 2026 20:08:36 +0200 Message-ID: <20260611180842.6390-4-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201392504158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn We will make a benefit from this split in later patches. Reviewed-by: Klaus Jensen Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 59 +++++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index b4a9728e06b..dcdb9bd66f8 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -4856,18 +4856,14 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t sqid, uint16_t cqid, uint16_t size) +static void __nvme_init_sq(NvmeSQueue *sq) { + NvmeCtrl *n =3D sq->ctrl; + uint16_t sqid =3D sq->sqid; + uint16_t cqid =3D sq->cqid; int i; NvmeCQueue *cq; =20 - sq->ctrl =3D n; - sq->dma_addr =3D dma_addr; - sq->sqid =3D sqid; - sq->size =3D size; - sq->cqid =3D cqid; - sq->head =3D sq->tail =3D 0; sq->io_req =3D g_new0(NvmeRequest, sq->size); =20 QTAILQ_INIT(&sq->req_list); @@ -4897,6 +4893,18 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, n->sq[sqid] =3D sq; } =20 +static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t sqid, uint16_t cqid, uint16_t size) +{ + sq->ctrl =3D n; + sq->dma_addr =3D dma_addr; + sq->sqid =3D sqid; + sq->size =3D size; + sq->cqid =3D cqid; + sq->head =3D sq->tail =3D 0; + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5557,25 +5565,16 @@ static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t cqid, uint16_t vector, uint16_t size, - uint16_t irq_enabled) +static void __nvme_init_cq(NvmeCQueue *cq) { + NvmeCtrl *n =3D cq->ctrl; PCIDevice *pci =3D PCI_DEVICE(n); + uint16_t cqid =3D cq->cqid; =20 - if (msix_present(pci) && irq_enabled) { - msix_vector_use(pci, vector); + if (msix_present(pci) && cq->irq_enabled) { + msix_vector_use(pci, cq->vector); } =20 - cq->ctrl =3D n; - cq->cqid =3D cqid; - cq->size =3D size; - cq->dma_addr =3D dma_addr; - cq->phase =3D 1; - cq->irq_enabled =3D irq_enabled; - cq->vector =3D vector; - cq->head =3D cq->tail =3D 0; - QTAILQ_INIT(&cq->req_list); QTAILQ_INIT(&cq->sq_list); if (n->dbbuf_enabled) { cq->db_addr =3D n->dbbuf_dbs + (cqid << 3) + (1 << 2); @@ -5592,6 +5591,22 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, &DEVICE(cq->ctrl)->mem_reentrancy_guard); } =20 +static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t cqid, uint16_t vector, uint16_t size, + uint16_t irq_enabled) +{ + cq->ctrl =3D n; + cq->cqid =3D cqid; + cq->size =3D size; + cq->dma_addr =3D dma_addr; + cq->phase =3D 1; + cq->irq_enabled =3D irq_enabled; + cq->vector =3D vector; + cq->head =3D cq->tail =3D 0; + QTAILQ_INIT(&cq->req_list); + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201380; cv=none; d=zohomail.com; s=zohoarc; b=GC62jkFzWhDY2sltXeKYVUGLVV8Che3/1yie8A4MiXBdEb1bCPtLK4rjTKfOIFm2HuX1+9Aa8vkqelYjKliO4DLUfYhnrazvtvz36r5q2pxkFkuYYuGZZmLejonqqfqyMwVaeuA9QDZUG2Zeb8xLILGwkudWKhxhSKYyPAAFWRw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201380; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PEkWfPanh7q59TrWlqY0t/+OQLGLErZaOJuxqlzy6nE=; b=b6fou1/yQHYltvTfySF8gRo5XDeX3z1r0O2tkmf9fFq9FB+GfIu3TrtfgxJ6ixuXkT2NSdh5JIEk9sQdBu7+kxQNKPm96cdGC5qFvVDcBMYRDmfHtCKoSo1E0WGJLN02ddkQUBSOwFq8NxR7wTSR9q6arKS7EK3JOMqJr9mFDTs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201380194102.12618551435241; Thu, 11 Jun 2026 11:09:40 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpn-000727-AX; Thu, 11 Jun 2026 14:08:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpl-000703-39 for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:53 -0400 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjph-0002H1-TZ for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:52 -0400 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-490c1915793so872095e9.2 for ; Thu, 11 Jun 2026 11:08:49 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201328; x=1781806128; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PEkWfPanh7q59TrWlqY0t/+OQLGLErZaOJuxqlzy6nE=; b=E8GgoU3UYxtEYUQHfU9Is0oaROwXUJklr5gwEXAnsNVRFakDi4fz0eMG37MTzi4DY7 kcfmj7/D6Ad2Bj2oHftHseJ4krY6gJ6bqKgVPXZFvSiDJTdkaWHAFE3JYzkF0eUB4wC+ xNAuZzKDoQwF1kb3HKqvG/ApX1b2uv4odhmM8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201328; x=1781806128; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=PEkWfPanh7q59TrWlqY0t/+OQLGLErZaOJuxqlzy6nE=; b=ZY2IeSJPcALmI2wIlq3OCl9R6LieZJGjg4UzQ4VRXnpcG1qBOT5HX8Nwlm8c0ac8a9 lOvWzaTQiAtlSd0LfM25CNzXqrSJ/TilHpM9EGDrHU8aVOQI9kJ1+fmyvSKD5X+40qOr lXVHxQu0RqALoyvKHGDawiawUb3v1WTINvHPn0j/rn1pXBWYWHj0FoHEZFTfckz7CG9T 9jFBIomKrrSfDcf4YQWNayH8mkYVWhDH3WB58ep+O68GzdZP9lqql8650WW1FAFNavIK MTqw6RuZH2en4c3CqWYPB/0fmLrNXfBxIlOyOaRoPstU21FXR9Kax29l6fUalVCE/owt 5e3Q== X-Gm-Message-State: AOJu0YxUqF1XW2AKY0TFGiUB7MFPOALvce2n21hxI4UOkyv7anVmcWgC 4ybWGzSZiUXZpV501s9/QLVhgB7t1WecHYwIEnKZcstQsZis960/kGA/mFdGQ8ofmayg1/pSCHR l0idF29U= X-Gm-Gg: Acq92OG/SzeheTdeCSDIkgO3VqOIHt641C8Um+8NZov9txGpQzo05ldZHa7kP+o/KUQ s++vJbT5xTDmBx8t5z9n9alhqn4T34nnQggM5Izp2SnNtDEgGrCVnq/fAFb7YKm6zHE7RY/QCPm eLLZ8VT6UNpA0/4xwIj24uRyzmFsanwpk9VyN71cA7PPADINVfl8prbSYJNxmkvsqtGLruQFy9d 2AdkZuV2hHgbT4boHpMQSzm4ZK1wKxm+0MfS9WuZtRg8EK0hAkftv3Jdlus0P1R+wcdkuZy+CdI xbdDmzSHgRCg1bm/tFT2rQshJGhMR1it/QVYONXJzYd3JQSXnAgcL605DJA/Ke3WoQpSWCFp3D/ MLtNQJDCzdP1T5/eG5oybHFOLgFWPcalWx3N+QqVUOdUC+MXYHNC94VOzbv5t5b7T2nrcpIDWq9 Fy6BQZ1wl/qAqHT9UyuwKpRaoQhAtaVOsYpqjrbLadP+wGWmxS3c8sqsoQre7r0VFfd0j5Be8Rv ql1rW6SxK6FChg+Zx7yWop9NGnrBqiRJA== X-Received: by 2002:a05:600c:8902:b0:490:e1cd:ea25 with SMTP id 5b1f17b1804b1-490e55fee98mr51634245e9.23.1781201328261; Thu, 11 Jun 2026 11:08:48 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v10 4/8] hw/nvme: set CQE.sq_id earlier in nvme_process_sq Date: Thu, 11 Jun 2026 20:08:37 +0200 Message-ID: <20260611180842.6390-5-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201382743158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of filling req->cqe.sq_id in nvme_post_cqes, let's set it earlier in nvme_process_sq. This shouldn't cause any issues, because req->cqe.sq_id never changes during lifetime of req. This will help us for migration support. Reviewed-by: Klaus Jensen Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index dcdb9bd66f8..4040d24073b 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1522,7 +1522,6 @@ static void nvme_post_cqes(void *opaque) =20 sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); - req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.sq_head =3D cpu_to_le16(sq->head); addr =3D cq->dma_addr + (cq->tail << NVME_CQES); ret =3D pci_dma_write(PCI_DEVICE(n), addr, (void *)&req->cqe, @@ -7852,6 +7851,7 @@ static void nvme_process_sq(void *opaque) QTAILQ_REMOVE(&sq->req_list, req, entry); QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); nvme_req_clear(req); + req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.cid =3D cmd.cid; memcpy(&req->cmd, &cmd, sizeof(NvmeCmd)); =20 --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201425; cv=none; d=zohomail.com; s=zohoarc; b=PxtGb0kjL8Js8kBaoYRmURs28IFj7RgIFeRTfVuvv5xC+vLPCvULBho1awOrfQkBPTyWZPYn5PzQBpF1AnZk+HAuWHmoTSTgiLpAbwqeUjPhptHvW2VyjpUNd5nfQBheNcyCOuRRpau2aTRuLyueBjjNE3wnYgiLAqC3FVRJlSM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201425; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=zCPk+l34ucyP8qr+raic5pyiQuMSsix62CIWyGJoUqc=; b=ZRK/I5jVoWU6a+sT1sAUddj2svMLMZ3y7SpBiCQdIXNNhRMWnoyHPN6R0lar6PywRyEQZff8XWSZW4mXSxHZvMAGu5iUfkb1ewLuqOiCTx2llHoHvpztGfDzfDo960uuohYRMhwkAmpLWVXNGXSuG2wfjcEiTWMf0qe+usNLScM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201425909308.5438913877849; Thu, 11 Jun 2026 11:10:25 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpn-00072R-Hp; Thu, 11 Jun 2026 14:08:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpl-00070D-8e for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:53 -0400 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpi-0002Hi-SX for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:52 -0400 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-490ae94a89eso937545e9.1 for ; Thu, 11 Jun 2026 11:08:50 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201329; x=1781806129; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zCPk+l34ucyP8qr+raic5pyiQuMSsix62CIWyGJoUqc=; b=SRu6oUPsAKATLhVKWZ7NaL5iBzJDzyB4YsIj8zJjJyW+ySdQZ5REoRSZo7MHWwVGYw yRIGUFupbh83ZKxxeLKK4PZVt+3btg90QpxWo/EW7zfb5M8yjvbFckOB5vTpD7n/36/8 77vhEvvkWL+nCloHzzAncA/BA/5Zy9yj2UxoY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201329; x=1781806129; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=zCPk+l34ucyP8qr+raic5pyiQuMSsix62CIWyGJoUqc=; b=F485uQGSiJyMgIyTHPBtB6gUzseV6fDojmBLcX/frraq3+mxdOoQSCEgp1JAA5u1Ut gZdm79hin3yEHj8edBz+eL10aNcc9FvMSROtkF1xLAh6r46Y2IXBfkBuHrcj4olKlogQ YduTiOr9/Usfc2SeZxvFUcCOBwjJrbWcqA+6nvkVvfYzMFVoRC3eLj8+LX+oEIAdaG0b zjtv3Anst3wrWleNeCfuY/jIsE72dr11GX5c1dFuGIQ60A2yFiIK7MxUQ4BnopFrzX4l QMXY0wUICHZwyZgRhtDze9l72ornPRV1XSkx2iOrzXspgiLvgIAKs6yoncqBkvrLRJzB 8Smw== X-Gm-Message-State: AOJu0Yx1Nb67FiQNWdryPMxt+FTWlBfhnzERH/GFtifrF49zRmbHzuig PaHSDj3nclj4IwLpKgy1mbwgCIesmSbkT7Uhdxi8DOYNCkdKxaBKvzzrWr/1hileO8pJuQCdfsv 2rENKk1U= X-Gm-Gg: Acq92OHNf0eV+c4aBSz9+fnB2EWO6JNpVAkMIWtiOP5iOIWok0WKDUGsXPCy9ndMnbz lMlGfSBjLdojhAcnHIJSOT1xcwBPLlIIMpT+wHOFe8+UOhrdgp763CEKRvIArAYM66an7cWeQcR Yof5DIM1CkvOrLRRK3Q8NMqhxo0CH8MvSqayZcqBewh8h2m9DbT6QNdNOvkAHmGBmoTxW54m0/J 0EO4XIJWsPRpxQXTXywHwEEXhkvHxDPrbgqJvQSu3NuOpy3BisSlZmhEOt3m+XHrvwWrXLIpg31 rEQCfxK5D0KPqBjSLrxWd3mfWFFF2Z2xp2hhp47AFn2ZKN1H+3onXQiPy3utJWn9AZj3dupL6SN ndEBjQ2LVkQjHS7zPNd0pFguV2MKPd1WH1RHyobLZfoUK+MOu6UHws51UEMAfle00qAzoJg3n9u VB1OunJOvDg0oDK0G08p8pkZUep+RdUCYMRsgAsXijRacDwx8ouBlZthrQCKkuZaqC5RBzflKnB T5G5idGL0ap2KPxKQ1ix2k= X-Received: by 2002:a05:600c:4691:b0:48f:e249:4094 with SMTP id 5b1f17b1804b1-490e560b2d5mr54393125e9.18.1781201329469; Thu, 11 Jun 2026 11:08:49 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v10 5/8] hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completion Date: Thu, 11 Jun 2026 20:08:38 +0200 Message-ID: <20260611180842.6390-6-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::334; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x334.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201426379158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of unmapping req->sg in nvme_post_cqes(), we can do it earlier in nvme_enqueue_req_completion(). When req completion is enqueued we don't need to access req->sg anymore. We only care about req->sq, req->cqe and req->status. Reviewed-by: Klaus Jensen Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 4040d24073b..8856cae6e65 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1536,7 +1536,6 @@ static void nvme_post_cqes(void *opaque) QTAILQ_REMOVE(&cq->req_list, req, entry); =20 nvme_inc_cq_tail(cq); - nvme_sg_unmap(&req->sg); =20 if (QTAILQ_EMPTY(&sq->req_list) && !nvme_sq_empty(sq)) { qemu_bh_schedule(sq->bh); @@ -1566,6 +1565,8 @@ static void nvme_enqueue_req_completion(NvmeCQueue *c= q, NvmeRequest *req) req->status, req->cmd.opcode); } =20 + nvme_sg_unmap(&req->sg); + QTAILQ_REMOVE(&req->sq->out_req_list, req, entry); QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); =20 --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201379; cv=none; d=zohomail.com; s=zohoarc; b=a3Iu4Y6WbWmZdvIuDSXXQsd9VQcf8k+QqDJLM2vG+yhhXnBwU1a4d4pNIvt0jUnSgQoCAfYt5kRXyECiYVIaReA/bzWVeDBoT0ydtdpRJy+mLgTiPCmFw4wYp+yge6rOzZ3DMyr1JOFwN8j6J0Akbh71OwFX6DMmFHoZ9Azup9o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201379; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=UvVf8rQ2RkZT0OdCkOdgzME+7vR/B7/RTUCzmRaHAWw=; b=OejVVuKB+95szIz5KeRxV0COzhBLX46FkE1kG45VQHTNIgVZUP0alIwmKeqcg5+e62WBPxhkYyv+jXtzmyghxNM7rLAOsyuutrJiaAGXYjgNBdKthwfcgs+9eJHLLzVyKCUdhqG5Y5EvGqIE5WcNuO+WcunQv5It1eYb4CXegAM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201379589856.8495501862615; Thu, 11 Jun 2026 11:09:39 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpq-00074a-4E; Thu, 11 Jun 2026 14:08:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpo-00073O-RU for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:56 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpk-0002Im-Cx for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:56 -0400 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-490b4a8e28bso807305e9.1 for ; Thu, 11 Jun 2026 11:08:51 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201331; x=1781806131; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UvVf8rQ2RkZT0OdCkOdgzME+7vR/B7/RTUCzmRaHAWw=; b=XBFSMqHe5/xxmCs4XLYvxo0qTTlQwd7RfozMuYFqfqZgtBegKcNcVR1R/vi4HjVX4w CoJchDb8YT2BXE7h/9/QL0ixcmlGApbB1qWhscNlvMUDIUZE4f+DWWlZvPdO5/anTaWP K0usxVrX048GULPzOnlPZ+51ykVdGgU0Yuf2Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201331; x=1781806131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UvVf8rQ2RkZT0OdCkOdgzME+7vR/B7/RTUCzmRaHAWw=; b=Z8Oc8Zf9vSgAd3uvuHNu+YfhKClDA07+JzkD8me0QrN+eB7DJ4y6YVFNAvib6ZFcnw FGZG9ALixnH8J5Ps5s9ayMbPGno2AVgH9Ou4qhM585RCIRi48+tsxAqSKxwzoZ3P0Pki 8Av2cuscbhohorfjwsQJGABe05yBI89L1llIMY7YOMK+Zz0bivA5PZqA0ThUo4NZ7OVI 96lS70RSME+0bAtHM3s6htUvA9smXEzp3OvHblCMVh6e6dLEWK+1Po7YdNjcY6KKy9KN FXy3RssYlqUQMlMgFwlGzE5DNwH+ljNyzH8rEGSQAwnr2n0ISMXu0HrRphssoB3ypqLQ G2sw== X-Gm-Message-State: AOJu0YwXMqmaY5/ZiwLAi3mrAgkgns9WRKCXyQdaHPo7OuE43XkGlsdg FAlbFfvVZHS0oYPEwGVuyzhptvlLrPRYJKJROAxq9uGZxEy50tqKkDtnRBtp2PCRlNxryYvnrjD giTNFCZk= X-Gm-Gg: Acq92OH3Y04N1+FmlZJjQ35poZRLtEHhI+Le+w8lNc6VjOzviNFcK/Z4Wd8nj4zCKnV umnnz+dNQqqfvVMW5+UdsFE8l/3SUKHlYjkP+RwYv1ON7auEOR5Nxv/yAY+hq7EW062m3sl4vTk /4ZXX7YyeNisMAUBZU+kHcPCnaJcx7kgvHNdRs3cI3bHmXlxsZ947pdC/Y9sVCe/hDSwUvSab7D BhDUFuHWKD1ok1QlvnoVLF7jO1U6E0sDo5S/Chzeh2/QRGPspXrL66zecwcErCOMR68cEHJ0HKp Xp54ABoLuPipP5B5oKORmIR6u874RHbxivDTUQd0RwsxbHEJ/2eW6U41OliTq7LfbWZP/VG1/Ip lfzJzNTw8sZlodcaf7GJJymU9xXAhSWeuxXO0VDR3Yp5eYuSyL9zOOdimWlBq5pXigxCBf+tpHn JSfIS9Vmh61XjTHkPfnP58mseDx1yxk9KsyLWsbW0BYMatIeJVALE6pHoiJ2RSc7aVXf9ajcBTK IzND8UdKfV/QwyN4OcbRqU= X-Received: by 2002:a05:600c:3e15:b0:490:48df:2793 with SMTP id 5b1f17b1804b1-490e561eff9mr53184085e9.26.1781201330789; Thu, 11 Jun 2026 11:08:50 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn Subject: [PATCH v10 6/8] hw/nvme: add basic live migration support Date: Thu, 11 Jun 2026 20:08:39 +0200 Message-ID: <20260611180842.6390-7-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201382756158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn It has some limitations: - only one NVMe namespace is supported - SMART counters are not preserved - CMB is not supported - PMR is not supported - SPDM is not supported - SR-IOV is not supported Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn Reviewed-by: Fabiano Rosas --- v9: - trivial check-patch fixes v7: - renamed copy_cq_req_list to move_cq_req_list - validate incoming migration stream better v6: - handle full CQ case v2: - AERs are now fully supported --- hw/nvme/ctrl.c | 760 ++++++++++++++++++++++++++++++++++++++++++- hw/nvme/ns.c | 164 ++++++++++ hw/nvme/nvme.h | 9 + hw/nvme/trace-events | 10 + 4 files changed, 934 insertions(+), 9 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 8856cae6e65..fc4c35f42a0 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -210,6 +210,7 @@ #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" #include "migration/blocker.h" +#include "migration/qemu-file-types.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -1520,6 +1521,18 @@ static void nvme_post_cqes(void *opaque) break; } =20 + /* + * Here we take the following fields from NvmeRequest structure + * and write cqe to the guest RAM based on them: + * - req->sq + * - req->status + * - req->cqe + * + * If you change this code and more fields from NvmeRequest are + * used, please make sure that you have handled this in: + * nvme_vmstate_request and nvme_ctrl_pre_save(). + */ + sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); req->cqe.sq_head =3D cpu_to_le16(sq->head); @@ -4905,6 +4918,25 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_sq(sq); } =20 +static void nvme_restore_sq(NvmeSQueue *sq_from) +{ + NvmeCtrl *n =3D sq_from->ctrl; + NvmeSQueue *sq =3D sq_from; + + if (sq_from->sqid =3D=3D 0) { + sq =3D &n->admin_sq; + sq->ctrl =3D n; + sq->dma_addr =3D sq_from->dma_addr; + sq->sqid =3D sq_from->sqid; + sq->size =3D sq_from->size; + sq->cqid =3D sq_from->cqid; + sq->head =3D sq_from->head; + sq->tail =3D sq_from->tail; + } + + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5607,6 +5639,39 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_cq(cq); } =20 +static void move_cq_req_list(NvmeCQueue *cq_to, NvmeCQueue *cq_from) +{ + NvmeRequest *req, *next; + + QTAILQ_FOREACH_SAFE(req, &cq_from->req_list, entry, next) { + QTAILQ_REMOVE(&cq_from->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq_to->req_list, req, entry); + } +} + +static void nvme_restore_cq(NvmeCQueue *cq_from) +{ + NvmeCtrl *n =3D cq_from->ctrl; + NvmeCQueue *cq =3D cq_from; + + if (cq_from->cqid =3D=3D 0) { + cq =3D &n->admin_cq; + cq->ctrl =3D n; + cq->cqid =3D cq_from->cqid; + cq->size =3D cq_from->size; + cq->dma_addr =3D cq_from->dma_addr; + cq->phase =3D cq_from->phase; + cq->irq_enabled =3D cq_from->irq_enabled; + cq->vector =3D cq_from->vector; + cq->head =3D cq_from->head; + cq->tail =3D cq_from->tail; + QTAILQ_INIT(&cq->req_list); + move_cq_req_list(cq, cq_from); + } + + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; @@ -7297,7 +7362,7 @@ static uint16_t nvme_dbbuf_config(NvmeCtrl *n, const = NvmeRequest *req) n->dbbuf_eis =3D eis_addr; n->dbbuf_enabled =3D true; =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { NvmeSQueue *sq =3D n->sq[i]; NvmeCQueue *cq =3D n->cq[i]; =20 @@ -7741,7 +7806,7 @@ static int nvme_atomic_write_check(NvmeCtrl *n, NvmeC= md *cmd, /* * Walk the queues to see if there are any atomic conflicts. */ - for (i =3D 1; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 1; i < n->num_queues; i++) { NvmeSQueue *sq; NvmeRequest *req; NvmeRwCmd *req_rw; @@ -7811,6 +7876,12 @@ static void nvme_process_sq(void *opaque) NvmeCmd cmd; NvmeRequest *req; =20 + /* + * We don't want to have a race with nvme_ctrl_pre_save(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + if (n->dbbuf_enabled) { nvme_update_sq_tail(sq); } @@ -7928,12 +7999,12 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) nvme_ns_drain(ns); } =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->sq[i] !=3D NULL) { nvme_free_sq(n->sq[i], n); } } - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->cq[i] !=3D NULL) { nvme_free_cq(n->cq[i], n); } @@ -8603,6 +8674,8 @@ static bool nvme_check_params(NvmeCtrl *n, Error **er= rp) params->max_ioqpairs =3D params->num_queues - 1; } =20 + n->num_queues =3D params->max_ioqpairs + 1; + if (n->namespace.blkconf.blk && n->subsys) { error_setg(errp, "subsystem support is unavailable with legacy " "namespace ('drive' property)"); @@ -8776,8 +8849,8 @@ static void nvme_init_state(NvmeCtrl *n) n->conf_msix_qsize =3D n->params.msix_qsize; } =20 - n->sq =3D g_new0(NvmeSQueue *, n->params.max_ioqpairs + 1); - n->cq =3D g_new0(NvmeCQueue *, n->params.max_ioqpairs + 1); + n->sq =3D g_new0(NvmeSQueue *, n->num_queues); + n->cq =3D g_new0(NvmeCQueue *, n->num_queues); n->temperature =3D NVME_TEMPERATURE; n->features.temp_thresh_hi =3D NVME_TEMPERATURE_WARNING; n->starttime_ms =3D qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); @@ -9012,7 +9085,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) } =20 if (n->params.msix_exclusive_bar && !pci_is_vf(pci_dev)) { - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, 0, NULL, N= ULL); + bar_size =3D nvme_mbar_size(n->num_queues, 0, NULL, NULL); memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n, "nv= me", bar_size); pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | @@ -9024,7 +9097,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) /* add one to max_ioqpairs to account for the admin queue pair */ if (!pci_is_vf(pci_dev)) { nr_vectors =3D n->params.msix_qsize; - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, + bar_size =3D nvme_mbar_size(n->num_queues, nr_vectors, &msix_table_offset, &msix_pba_offset); } else { @@ -9757,9 +9830,678 @@ static uint32_t nvme_pci_read_config(PCIDevice *dev= , uint32_t address, int len) return pci_default_read_config(dev, address, len); } =20 +static const VMStateDescription nvme_vmstate_cqe =3D { + .name =3D "nvme-cqe", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(result, NvmeCqe), + VMSTATE_UINT32(dw1, NvmeCqe), + VMSTATE_UINT16(sq_head, NvmeCqe), + VMSTATE_UINT16(sq_id, NvmeCqe), + VMSTATE_UINT16(cid, NvmeCqe), + VMSTATE_UINT16(status, NvmeCqe), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr_sgl =3D { + .name =3D "nvme-request-cmd-dptr-sgl", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(addr, NvmeSglDescriptor), + VMSTATE_UINT32(len, NvmeSglDescriptor), + VMSTATE_UINT8_ARRAY(rsvd, NvmeSglDescriptor, 3), + VMSTATE_UINT8(type, NvmeSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr =3D { + .name =3D "nvme-request-cmd-dptr", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(prp1, NvmeCmdDptr), + VMSTATE_UINT64(prp2, NvmeCmdDptr), + VMSTATE_STRUCT(sgl, NvmeCmdDptr, 0, + nvme_vmstate_cmd_dptr_sgl, NvmeSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd =3D { + .name =3D "nvme-request-cmd", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(opcode, NvmeCmd), + VMSTATE_UINT8(flags, NvmeCmd), + VMSTATE_UINT16(cid, NvmeCmd), + VMSTATE_UINT32(nsid, NvmeCmd), + VMSTATE_UINT64(res1, NvmeCmd), + VMSTATE_UINT64(mptr, NvmeCmd), + VMSTATE_STRUCT(dptr, NvmeCmd, 0, nvme_vmstate_cmd_dptr, NvmeCmdDpt= r), + VMSTATE_UINT32(cdw10, NvmeCmd), + VMSTATE_UINT32(cdw11, NvmeCmd), + VMSTATE_UINT32(cdw12, NvmeCmd), + VMSTATE_UINT32(cdw13, NvmeCmd), + VMSTATE_UINT32(cdw14, NvmeCmd), + VMSTATE_UINT32(cdw15, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static bool nvme_req_pre_load(void *opaque, Error **errp) +{ + memset(opaque, 0x0, sizeof(NvmeRequest)); + return true; +} + +static const VMStateDescription nvme_vmstate_request =3D { + .name =3D "nvme-request", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_req_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(status, NvmeRequest), + VMSTATE_STRUCT(cqe, NvmeRequest, 0, nvme_vmstate_cqe, NvmeCqe), + VMSTATE_STRUCT(cmd, NvmeRequest, 0, nvme_vmstate_cmd, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_bar =3D { + .name =3D "nvme-bar", + .minimum_version_id =3D 1, + .version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(cap, NvmeBar), + VMSTATE_UINT32(vs, NvmeBar), + VMSTATE_UINT32(intms, NvmeBar), + VMSTATE_UINT32(intmc, NvmeBar), + VMSTATE_UINT32(cc, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd24, NvmeBar, 4), + VMSTATE_UINT32(csts, NvmeBar), + VMSTATE_UINT32(nssr, NvmeBar), + VMSTATE_UINT32(aqa, NvmeBar), + VMSTATE_UINT64(asq, NvmeBar), + VMSTATE_UINT64(acq, NvmeBar), + VMSTATE_UINT32(cmbloc, NvmeBar), + VMSTATE_UINT32(cmbsz, NvmeBar), + VMSTATE_UINT32(bpinfo, NvmeBar), + VMSTATE_UINT32(bprsel, NvmeBar), + VMSTATE_UINT64(bpmbl, NvmeBar), + VMSTATE_UINT64(cmbmsc, NvmeBar), + VMSTATE_UINT32(cmbsts, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd92, NvmeBar, 3492), + VMSTATE_UINT32(pmrcap, NvmeBar), + VMSTATE_UINT32(pmrctl, NvmeBar), + VMSTATE_UINT32(pmrsts, NvmeBar), + VMSTATE_UINT32(pmrebs, NvmeBar), + VMSTATE_UINT32(pmrswtp, NvmeBar), + VMSTATE_UINT32(pmrmscl, NvmeBar), + VMSTATE_UINT32(pmrmscu, NvmeBar), + VMSTATE_UINT8_ARRAY(css, NvmeBar, 484), + VMSTATE_END_OF_LIST() + }, +}; + +static bool nvme_cqueue_pre_load(void *opaque, Error **errp) +{ + NvmeCQueue *cq =3D opaque; + + QTAILQ_INIT(&cq->req_list); + return true; +} + +static const VMStateDescription nvme_vmstate_cqueue =3D { + .name =3D "nvme-cq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_cqueue_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(phase, NvmeCQueue), + VMSTATE_UINT16(cqid, NvmeCQueue), + VMSTATE_UINT16(irq_enabled, NvmeCQueue), + VMSTATE_UINT32(head, NvmeCQueue), + VMSTATE_UINT32(tail, NvmeCQueue), + VMSTATE_UINT32(vector, NvmeCQueue), + VMSTATE_UINT32(size, NvmeCQueue), + VMSTATE_UINT64(dma_addr, NvmeCQueue), + + VMSTATE_QTAILQ_V(req_list, NvmeCQueue, 1, nvme_vmstate_request, + NvmeRequest, entry), + + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_squeue =3D { + .name =3D "nvme-sq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(sqid, NvmeSQueue), + VMSTATE_UINT16(cqid, NvmeSQueue), + VMSTATE_UINT32(head, NvmeSQueue), + VMSTATE_UINT32(tail, NvmeSQueue), + VMSTATE_UINT32(size, NvmeSQueue), + VMSTATE_UINT64(dma_addr, NvmeSQueue), + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event_result =3D { + .name =3D "nvme-async-event-result", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(event_type, NvmeAerResult), + VMSTATE_UINT8(event_info, NvmeAerResult), + VMSTATE_UINT8(log_page, NvmeAerResult), + VMSTATE_UINT8(resv, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event =3D { + .name =3D "nvme-async-event", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_STRUCT(result, NvmeAsyncEvent, 0, + nvme_vmstate_async_event_result, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_hbs =3D { + .name =3D "nvme-hbs", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(acre, NvmeHostBehaviorSupport), + VMSTATE_UINT8(etdas, NvmeHostBehaviorSupport), + VMSTATE_UINT8(lbafee, NvmeHostBehaviorSupport), + VMSTATE_UINT8(rsvd3, NvmeHostBehaviorSupport), + VMSTATE_UINT16(cdfe, NvmeHostBehaviorSupport), + VMSTATE_UINT8_ARRAY(rsvd6, NvmeHostBehaviorSupport, 506), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_atomic =3D { + .name =3D "nvme-atomic", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(atomic_max_write_size, NvmeAtomic), + VMSTATE_UINT64(atomic_boundary, NvmeAtomic), + VMSTATE_UINT64(atomic_nabo, NvmeAtomic), + VMSTATE_BOOL(atomic_writes, NvmeAtomic), + VMSTATE_END_OF_LIST() + } +}; + +static bool pre_save_validate_aer_req(NvmeRequest *req, Error **errp) +{ + /* + * Can't use assert() here, because we don't want + * to just crash QEMU when user requests a migration. + */ + if (!(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ)) { + error_setg(errp, "req->cmd.opcode (%u) !=3D NVME_ADM_CMD_ASYNC_EV_= REQ", + req->cmd.opcode); + return false; + } + + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->sq =3D=3D &req->sq->ctrl->admin_sq)) { + error_setg(errp, "req->sq !=3D &req->sq->ctrl->admin_sq"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool pre_save_validate_cq_req(NvmeRequest *req, Error **errp) +{ + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool nvme_ctrl_pre_save(void *opaque, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_pre_save_enter(n); + + /* + * We don't want to have a race with nvme_process_sq(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + + /* cancel all SQ processing BHs */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) { + continue; + } + + qemu_bh_cancel(sq->bh); + } + + /* drain all IO */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns; + + ns =3D nvme_ns(n, i); + if (!ns) { + continue; + } + + trace_pci_nvme_pre_save_ns_drain(n, i); + nvme_ns_drain(ns); + } + + /* + * Now, we should take care of AERs. + * + * 1. Save all queued events (n->aer_queue). + * This is done automatically, see nvme_vmstate VMStateDescription. + * Here we only need to print them for debugging purpose. + * 2. Go over outstanding AER requests (n->aer_reqs) and check they are + * all have expected opcode (NVME_ADM_CMD_ASYNC_EV_REQ) and other f= ields. + * + * We must be really careful here, because in case of further + * QEMU NVMe changes, we may break migration without noticing it, or w= orse, + * introduce silent data corruption during migration. + */ + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_pre_save_aer(event->result.event_type, + event->result.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeRequest *req =3D n->aer_reqs[i]; + + if (!pre_save_validate_aer_req(req, errp)) { + return false; + } + } + + /* + * Make sure that all in-flight IO requests + * (except NVME_ADM_CMD_ASYNC_EV_REQ) are processed. + */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req; + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) { + continue; + } + + trace_pci_nvme_pre_save_sq_out_req_check(n, i, + sq->head, sq->tail, sq->s= ize); + + QTAILQ_FOREACH(req, &sq->out_req_list, entry) { + assert(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ); + } + } + + /* wait when all IO requests completions are written to guest memory */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) { + continue; + } + + qemu_bh_cancel(cq->bh); + /* this should empty cq->req_list unless CQ is full */ + nvme_post_cqes(cq); + + trace_pci_nvme_pre_save_cq_req_check(n, i, + cq->head, cq->tail, cq->size); + + if (!QTAILQ_EMPTY(&cq->req_list)) { + NvmeRequest *req; + + assert(nvme_cq_full(cq)); + + QTAILQ_FOREACH(req, &cq->req_list, entry) { + trace_pci_nvme_pre_save_cq_unposted_cqe( + n, i, nvme_cid(req), + nvme_nsid(req->ns), + le32_to_cpu(req->cqe.result), + le32_to_cpu(req->cqe.dw1), + req->status, req->cmd.opcode); + if (!pre_save_validate_cq_req(req, errp)) { + return false; + } + } + } + } + + for (uint32_t nsid =3D 0; nsid <=3D NVME_MAX_NAMESPACES; nsid++) { + NvmeNamespace *ns =3D n->namespaces[nsid]; + + if (!ns) { + continue; + } + + if (ns !=3D &n->namespace) { + error_setg(errp, + "only one NVMe namespace is supported for migration= "); + return false; + } + } + + return true; +} + +static bool nvme_ctrl_post_load(void *opaque, int version_id, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_post_load_enter(n); + + /* restore CQs first */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) { + continue; + } + + if (cq->cqid !=3D i) { + error_setg(errp, "inconsistent migration stream (cq->cqid !=3D= i)"); + return false; + } + + cq->ctrl =3D n; + nvme_restore_cq(cq); + trace_pci_nvme_post_load_restore_cq(n, i, cq->head, cq->tail, cq->= size); + + if (i =3D=3D 0) { + /* + * Admin CQ lives in n->admin_cq, we don't need + * memory allocated for it in get_ptrs_array_entry() anymore. + * + * nvme_restore_cq() also takes care of: + * n->cq[0] =3D &n->admin_cq; + * so n->cq[0] remains valid. + */ + g_free(cq); + } + } + + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) { + continue; + } + + if (sq->sqid !=3D i) { + error_setg(errp, "inconsistent migration stream (sq->sqid !=3D= i)"); + return false; + } + + if (!n->cq[sq->cqid]) { + error_setg(errp, + "inconsistent migration stream (n->cq[sq->cqid] is = NULL)"); + return false; + } + + sq->ctrl =3D n; + nvme_restore_sq(sq); + trace_pci_nvme_post_load_restore_sq(n, i, sq->head, sq->tail, sq->= size); + + if (i =3D=3D 0) { + /* same as for CQ */ + g_free(sq); + } + } + + /* restore cq->req_list-s */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req_from, *next; + typeof_field(NvmeCQueue, req_list) req_list; + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq || QTAILQ_EMPTY(&cq->req_list)) { + continue; + } + + /* + * We use nvme_vmstate_request VMStateDescription to save/restore + * NvmeRequest structures, but tricky thing here is that + * memory for each cq->req_list item is allocated separately + * during restore. It doesn't work for us. We need to take + * an existing NvmeRequest structure from SQ's req_list pool + * and fill it with data from the newly allocated one (req_from). + * Then, we can safely release allocated memory for it. + */ + + /* make a copy of cq->req_list (QTAILQ head) and clean cq->req_lis= t */ + QTAILQ_INIT(&req_list); + QTAILQ_FOREACH_SAFE(req_from, &cq->req_list, entry, next) { + QTAILQ_REMOVE(&cq->req_list, req_from, entry); + QTAILQ_INSERT_TAIL(&req_list, req_from, entry); + } + QTAILQ_INIT(&cq->req_list); + + QTAILQ_FOREACH_SAFE(req_from, &req_list, entry, next) { + uint16_t sqid =3D le16_to_cpu(req_from->cqe.sq_id); + NvmeRequest *req; + NvmeSQueue *sq; + + assert(!nvme_check_sqid(n, sqid)); + sq =3D n->sq[sqid]; + + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + QTAILQ_REMOVE(&req_list, req_from, entry); + g_free(req_from); + } + + qemu_bh_schedule(cq->bh); + } + + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_post_load_aer(event->result.event_type, + event->result.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeSQueue *sq =3D &n->admin_sq; + NvmeRequest *req_from =3D n->aer_reqs[i]; + NvmeRequest *req; + + /* Idea here is the same as for "restore cq->req_list-s" step */ + + /* take an NvmeRequest struct from SQ */ + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + n->aer_reqs[i] =3D req; + g_free(req_from); + } + + /* + * We need to attach namespaces (currently, only one namespace is + * supported for migration). + * This logic comes from nvme_start_ctrl(). + */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + + if (!ns || (!ns->params.shared && ns->ctrl !=3D n)) { + continue; + } + + if (nvme_csi_supported(n, ns->csi) && !ns->params.detached) { + if (!ns->attached || ns->params.shared) { + nvme_attach_ns(n, ns); + } + } + } + + /* schedule SQ processing */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) { + continue; + } + + qemu_bh_schedule(sq->bh); + } + + return true; +} + static const VMStateDescription nvme_vmstate =3D { .name =3D "nvme", - .unmigratable =3D 1, + .minimum_version_id =3D 1, + .version_id =3D 1, + .pre_save_errp =3D nvme_ctrl_pre_save, + .post_load_errp =3D nvme_ctrl_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_PCI_DEVICE(parent_obj, NvmeCtrl), + VMSTATE_MSIX(parent_obj, NvmeCtrl), + VMSTATE_STRUCT(bar, NvmeCtrl, 0, nvme_vmstate_bar, NvmeBar), + + VMSTATE_BOOL(qs_created, NvmeCtrl), + VMSTATE_UINT32(page_size, NvmeCtrl), + VMSTATE_UINT16(page_bits, NvmeCtrl), + VMSTATE_UINT16(max_prp_ents, NvmeCtrl), + VMSTATE_UINT32(max_q_ents, NvmeCtrl), + VMSTATE_UINT8(outstanding_aers, NvmeCtrl), + VMSTATE_UINT32(irq_status, NvmeCtrl), + VMSTATE_INT32(cq_pending, NvmeCtrl), + + VMSTATE_UINT64(host_timestamp, NvmeCtrl), + VMSTATE_UINT64(timestamp_set_qemu_clock_ms, NvmeCtrl), + VMSTATE_UINT64(starttime_ms, NvmeCtrl), + VMSTATE_UINT16(temperature, NvmeCtrl), + VMSTATE_UINT8(smart_critical_warning, NvmeCtrl), + + VMSTATE_UINT32(conf_msix_qsize, NvmeCtrl), + VMSTATE_UINT32(conf_ioqpairs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_dbs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_eis, NvmeCtrl), + VMSTATE_BOOL(dbbuf_enabled, NvmeCtrl), + + VMSTATE_UINT8(aer_mask, NvmeCtrl), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC( + aer_reqs, NvmeCtrl, outstanding_aers, 0, + nvme_vmstate_request, NvmeRequest), + VMSTATE_QTAILQ_V(aer_queue, NvmeCtrl, 1, nvme_vmstate_async_event, + NvmeAsyncEvent, entry), + VMSTATE_INT32(aer_queued, NvmeCtrl), + + VMSTATE_STRUCT(namespace, NvmeCtrl, 0, nvme_vmstate_ns, NvmeNamesp= ace), + + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + sq, NvmeCtrl, num_queues, 0, nvme_vmstate_squeue, NvmeSQueue), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + cq, NvmeCtrl, num_queues, 0, nvme_vmstate_cqueue, NvmeCQueue), + + VMSTATE_UINT16(features.temp_thresh_hi, NvmeCtrl), + VMSTATE_UINT16(features.temp_thresh_low, NvmeCtrl), + VMSTATE_UINT32(features.async_config, NvmeCtrl), + VMSTATE_STRUCT(features.hbs, NvmeCtrl, 0, + nvme_vmstate_hbs, NvmeHostBehaviorSupport), + + VMSTATE_UINT32(dn, NvmeCtrl), + VMSTATE_STRUCT(atomic, NvmeCtrl, 0, nvme_vmstate_atomic, NvmeAtomi= c), + + VMSTATE_END_OF_LIST() + }, }; =20 static void nvme_class_init(ObjectClass *oc, const void *data) diff --git a/hw/nvme/ns.c b/hw/nvme/ns.c index b0106eaa5c8..4caab590977 100644 --- a/hw/nvme/ns.c +++ b/hw/nvme/ns.c @@ -20,6 +20,7 @@ #include "qemu/bitops.h" #include "system/system.h" #include "system/block-backend.h" +#include "migration/vmstate.h" =20 #include "nvme.h" #include "trace.h" @@ -886,6 +887,168 @@ static void nvme_ns_realize(DeviceState *dev, Error *= *errp) } } =20 +static const VMStateDescription nvme_vmstate_lbaf =3D { + .name =3D "nvme_lbaf", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(ms, NvmeLBAF), + VMSTATE_UINT8(ds, NvmeLBAF), + VMSTATE_UINT8(rp, NvmeLBAF), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns =3D { + .name =3D "nvme_id_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(nsze, NvmeIdNs), + VMSTATE_UINT64(ncap, NvmeIdNs), + VMSTATE_UINT64(nuse, NvmeIdNs), + VMSTATE_UINT8(nsfeat, NvmeIdNs), + VMSTATE_UINT8(nlbaf, NvmeIdNs), + VMSTATE_UINT8(flbas, NvmeIdNs), + VMSTATE_UINT8(mc, NvmeIdNs), + VMSTATE_UINT8(dpc, NvmeIdNs), + VMSTATE_UINT8(dps, NvmeIdNs), + VMSTATE_UINT8(nmic, NvmeIdNs), + VMSTATE_UINT8(rescap, NvmeIdNs), + VMSTATE_UINT8(fpi, NvmeIdNs), + VMSTATE_UINT8(dlfeat, NvmeIdNs), + VMSTATE_UINT16(nawun, NvmeIdNs), + VMSTATE_UINT16(nawupf, NvmeIdNs), + VMSTATE_UINT16(nacwu, NvmeIdNs), + VMSTATE_UINT16(nabsn, NvmeIdNs), + VMSTATE_UINT16(nabo, NvmeIdNs), + VMSTATE_UINT16(nabspf, NvmeIdNs), + VMSTATE_UINT16(noiob, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nvmcap, NvmeIdNs, 16), + VMSTATE_UINT16(npwg, NvmeIdNs), + VMSTATE_UINT16(npwa, NvmeIdNs), + VMSTATE_UINT16(npdg, NvmeIdNs), + VMSTATE_UINT16(npda, NvmeIdNs), + VMSTATE_UINT16(nows, NvmeIdNs), + VMSTATE_UINT16(mssrl, NvmeIdNs), + VMSTATE_UINT32(mcl, NvmeIdNs), + VMSTATE_UINT8(msrc, NvmeIdNs), + VMSTATE_UINT8_ARRAY(rsvd81, NvmeIdNs, 18), + VMSTATE_UINT8(nsattr, NvmeIdNs), + VMSTATE_UINT16(nvmsetid, NvmeIdNs), + VMSTATE_UINT16(endgid, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nguid, NvmeIdNs, 16), + VMSTATE_UINT64(eui64, NvmeIdNs), + VMSTATE_STRUCT_ARRAY(lbaf, NvmeIdNs, NVME_MAX_NLBAF, 1, + nvme_vmstate_lbaf, NvmeLBAF), + VMSTATE_UINT8_ARRAY(vs, NvmeIdNs, 3712), + + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_nvm =3D { + .name =3D "nvme_id_ns_nvm", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(lbstm, NvmeIdNsNvm), + VMSTATE_UINT8(pic, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd9, NvmeIdNsNvm, 3), + VMSTATE_UINT32_ARRAY(elbaf, NvmeIdNsNvm, NVME_MAX_NLBAF), + VMSTATE_UINT32(npdgl, NvmeIdNsNvm), + VMSTATE_UINT32(nprg, NvmeIdNsNvm), + VMSTATE_UINT32(npra, NvmeIdNsNvm), + VMSTATE_UINT32(nors, NvmeIdNsNvm), + VMSTATE_UINT32(npdal, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd288, NvmeIdNsNvm, 3808), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_ind =3D { + .name =3D "nvme_id_ns_ind", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(nsfeat, NvmeIdNsInd), + VMSTATE_UINT8(nmic, NvmeIdNsInd), + VMSTATE_UINT8(rescap, NvmeIdNsInd), + VMSTATE_UINT8(fpi, NvmeIdNsInd), + VMSTATE_UINT32(anagrpid, NvmeIdNsInd), + VMSTATE_UINT8(nsattr, NvmeIdNsInd), + VMSTATE_UINT8(rsvd9, NvmeIdNsInd), + VMSTATE_UINT16(nvmsetid, NvmeIdNsInd), + VMSTATE_UINT16(endgrpid, NvmeIdNsInd), + VMSTATE_UINT8(nstat, NvmeIdNsInd), + VMSTATE_UINT8_ARRAY(rsvd15, NvmeIdNsInd, 4081), + VMSTATE_END_OF_LIST() + } +}; + +typedef struct TmpNvmeNamespace { + NvmeNamespace *parent; + bool enable_write_cache; +} TmpNvmeNamespace; + +static bool nvme_ns_tmp_pre_save(void *opaque, Error **errp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + tns->enable_write_cache =3D blk_enable_write_cache(tns->parent->blkcon= f.blk); + + return true; +} + +static bool nvme_ns_tmp_post_load(void *opaque, int version_id, Error **er= rp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + blk_set_enable_write_cache(tns->parent->blkconf.blk, + tns->enable_write_cache); + + return true; +} + +static const VMStateDescription nvme_vmstate_ns_tmp =3D { + .name =3D "nvme_ns_tmp", + .pre_save_errp =3D nvme_ns_tmp_pre_save, + .post_load_errp =3D nvme_ns_tmp_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_BOOL(enable_write_cache, TmpNvmeNamespace), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_ns =3D { + .name =3D "nvme_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_WITH_TMP(NvmeNamespace, TmpNvmeNamespace, nvme_vmstate_ns_= tmp), + + VMSTATE_STRUCT(id_ns, NvmeNamespace, 0, nvme_vmstate_id_ns, NvmeId= Ns), + VMSTATE_STRUCT(id_ns_nvm, NvmeNamespace, 0, + nvme_vmstate_id_ns_nvm, NvmeIdNsNvm), + VMSTATE_STRUCT(id_ns_ind, NvmeNamespace, 0, + nvme_vmstate_id_ns_ind, NvmeIdNsInd), + VMSTATE_STRUCT(lbaf, NvmeNamespace, 0, nvme_vmstate_lbaf, NvmeLBAF= ), + VMSTATE_UINT32(nlbaf, NvmeNamespace), + VMSTATE_UINT8(csi, NvmeNamespace), + VMSTATE_UINT16(status, NvmeNamespace), + VMSTATE_UINT8(pif, NvmeNamespace), + + VMSTATE_UINT16(zns.zrwas, NvmeNamespace), + VMSTATE_UINT16(zns.zrwafg, NvmeNamespace), + VMSTATE_UINT32(zns.numzrwa, NvmeNamespace), + + VMSTATE_UINT32(features.err_rec, NvmeNamespace), + VMSTATE_STRUCT(atomic, NvmeNamespace, 0, + nvme_vmstate_atomic, NvmeAtomic), + VMSTATE_END_OF_LIST() + } +}; + static const Property nvme_ns_props[] =3D { DEFINE_BLOCK_PROPERTIES(NvmeNamespace, blkconf), DEFINE_PROP_BOOL("detached", NvmeNamespace, params.detached, false), @@ -937,6 +1100,7 @@ static void nvme_ns_class_init(ObjectClass *oc, const = void *data) dc->bus_type =3D TYPE_NVME_BUS; dc->realize =3D nvme_ns_realize; dc->unrealize =3D nvme_ns_unrealize; + dc->vmsd =3D &nvme_vmstate_ns; device_class_set_props(dc, nvme_ns_props); dc->desc =3D "Virtual NVMe namespace"; } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 05aee24a15c..78a6eaa1774 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -444,6 +444,11 @@ typedef struct NvmeRequest { NvmeSg sg; bool atomic_write; QTAILQ_ENTRY(NvmeRequest)entry; + /* + * If you add a new field here, please make sure to update + * nvme_vmstate_request, pre_save_validate_aer_req() and + * pre_save_validate_cq_req(). + */ } NvmeRequest; =20 typedef struct NvmeBounceContext { @@ -640,6 +645,7 @@ typedef struct NvmeCtrl { =20 NvmeNamespace namespace; NvmeNamespace *namespaces[NVME_MAX_NAMESPACES + 1]; + uint32_t num_queues; NvmeSQueue **sq; NvmeCQueue **cq; NvmeSQueue admin_sq; @@ -751,4 +757,7 @@ void nvme_atomic_configure_max_write_size(bool dn, uint= 16_t awun, void nvme_ns_atomic_configure_boundary(bool dn, uint16_t nabsn, uint16_t nabspf, NvmeAtomic *atomic= ); =20 +extern const VMStateDescription nvme_vmstate_atomic; +extern const VMStateDescription nvme_vmstate_ns; + #endif /* HW_NVME_NVME_H */ diff --git a/hw/nvme/trace-events b/hw/nvme/trace-events index 6be0bfa1c1f..f97a6a11f36 100644 --- a/hw/nvme/trace-events +++ b/hw/nvme/trace-events @@ -7,6 +7,16 @@ pci_nvme_dbbuf_config(uint64_t dbs_addr, uint64_t eis_addr= ) "dbs_addr=3D0x%"PRIx64 pci_nvme_map_addr(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %"PRI= u64"" pci_nvme_map_addr_cmb(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %= "PRIu64"" pci_nvme_map_prp(uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t= prp2, int num_prps) "trans_len %"PRIu64" len %"PRIu32" prp1 0x%"PRIx64" pr= p2 0x%"PRIx64" num_prps %d" +pci_nvme_pre_save_enter(void *n) "n=3D%p" +pci_nvme_pre_save_ns_drain(void *n, int i) "n=3D%p i=3D%d" +pci_nvme_pre_save_sq_out_req_check(void *n, int i, uint32_t head, uint32_t= tail, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" = size=3D0x%"PRIx32"" +pci_nvme_pre_save_cq_req_check(void *n, int i, uint32_t head, uint32_t tai= l, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_pre_save_cq_unposted_cqe(void *n, int i, uint16_t cid, uint32_t n= sid, uint32_t dw0, uint32_t dw1, uint16_t status, uint8_t opc) "n=3D%p i=3D= %d cid %"PRIu16" nsid %"PRIu32" dw0 0x%"PRIx32" dw1 0x%"PRIx32" status 0x%"= PRIx16" opc 0x%"PRIx8"" +pci_nvme_pre_save_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type 0= x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" +pci_nvme_post_load_enter(void *n) "n=3D%p" +pci_nvme_post_load_restore_cq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_restore_sq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type = 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" pci_nvme_map_sgl(uint8_t typ, uint64_t len) "type 0x%"PRIx8" len %"PRIu64"" pci_nvme_io_cmd(uint16_t cid, uint32_t nsid, uint16_t sqid, uint8_t opcode= , const char *opname) "cid %"PRIu16" nsid 0x%"PRIx32" sqid %"PRIu16" opc 0x= %"PRIx8" opname '%s'" pci_nvme_admin_cmd(uint16_t cid, uint16_t sqid, uint8_t opcode, const char= *opname) "cid %"PRIu16" sqid %"PRIu16" opc 0x%"PRIx8" opname '%s'" --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201427; cv=none; d=zohomail.com; s=zohoarc; b=T0gbMSXhNtVuAT613jfSt37PCa6l+6ArsUU1UVtw4u69iCkkZSOVkMOnKZ5XqvZfXfj+bIZHNbVExFKZBo3egUD1ClIDyzpGVfCHwpWVrNs1PRTNTW8tQjZ+K9LMew0HtEU/9pL88/tHAu1GilE2GURYMTa2XZ/UmTaSTtVqJns= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201427; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=mS6Jgnf1q6xm35E2vee5rv8aGYu78NsPfN/jQ6mxQhg=; b=nqwogSx82ig6oeVYrVaQfgajEPWPD2Ha5p6OmckpBGAe6jB5vHMsXU2mwMd9Q4be9IwzljCIMaSXDb6N13b7/xxDx3t+2XpWg5uJx+nxBrQM+GnvL8URndTW8n9MbPiXSGFrGDZtjt9vR8SO2+D5kaHkGEFszHzkPOHMJS0ng5k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201427579910.1548350824842; Thu, 11 Jun 2026 11:10:27 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpp-00073l-Cr; Thu, 11 Jun 2026 14:08:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpn-00072e-Si for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:55 -0400 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpl-0002JN-DD for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:55 -0400 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-490b3e03939so1603195e9.1 for ; Thu, 11 Jun 2026 11:08:53 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201332; x=1781806132; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mS6Jgnf1q6xm35E2vee5rv8aGYu78NsPfN/jQ6mxQhg=; b=CrEZStPFebC1H0b3oWetiBW8paBIcERIjj/tzMOwZ46ZQAGbY+WLJgI1gIiV3of7ud TuF4L3Z4x4yTqGQYkbmGgtQ3cnqp8nrbaS28L8OvLDw/O+8yBMJH6uvxqV7nl0bLg0R6 dchTKPTFf0QIvxU8AKboE82z5j/edzhyGnrpg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201332; x=1781806132; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=mS6Jgnf1q6xm35E2vee5rv8aGYu78NsPfN/jQ6mxQhg=; b=LpeAxZ85VMoH2HDAbF42g6z51pV1nd3pnkBUiQFqwV306osW53Gwjp+CgM7l9edeDZ mNHLNsztFGK57zxFO7dJMdtQavIhVY6LfLcURXjmwtfvbu4t/4bzbGFo2tAEq2DoaCZ6 sfS9lfu2pLCBUbd8LTeufu8ORD6dF1pGNpbaIwraZoCm2/rAs2JdRrvzsmOnf5zXfYAz pMpb4UcUlynlYCxwES1okaRoYn7u3lQ/j/U0N1oYakjLE8Z0tMUqVyDoj/R9ZJ0tUj8F d+QVew+gaYcClTSwOXaYww7D+b+/doGwUFqD+AaUpZK/7kp1TWg9LQndlBDUZahlMzm7 LlcA== X-Gm-Message-State: AOJu0Yxs0ndbotYC10yAwZgu9m5tzDx/cjcYkaeBswdKVYtd3NOzjsg7 Ekrp9ZheftxQSFFYuWs2XZIChRA83snhKoKhoeBSal4/k2pER6hlV/hy5KTP/O7N7MMj5c58A2d nhx1PIqg= X-Gm-Gg: Acq92OGArcBLbZ1sAft0GY9HMykPNAl+JJH8XPlg/CBMvtqA5m7PFj14J3Z4YTOCAZL 874sK7jJWVP0TMLcBAWLY6uD6ZzpHMAF7yp36iOq8BsU1boezqK5KtewxuWK8VltsRYIcgchNG5 NJ9wtGBZh9CZOc/fBb1RlgkPTnkZyNHCoA01hsaUaeO0k/xlfFWmc28q7gyJDFlUd7fTu5XRPYB tANE94Knrdlh9zxS7VUyEruRQnKL8PXEcJSpE/TEA71nHFsecGtLX6xJs92Ch77DX9wEIVRL5HD 88bN0b9p9WigkB+n1g9s2nz4mr2kQgm2G70tpEFZGksE5kAJVoVIrxxi04uAM3ghcmi2ucDcQs8 +pxjebTjAoYmGODiTS4srC3hcGMnW/mKNihw99D4rGcav0b/1A8QdL0IGeCOm675GA5+xKDLgwZ Dy13/NJZVvCd3DdLsAYhQoq5NWCV8mFI0AZElQ5WZNPIwZn4X6A93AOakB3cg31pI2vscKtRpvw oqKrxT8qzYh2jG3YvYByxs= X-Received: by 2002:a05:600c:4fc8:b0:490:af63:2cb1 with SMTP id 5b1f17b1804b1-490e52df887mr49239555e9.7.1781201331842; Thu, 11 Jun 2026 11:08:51 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn Subject: [PATCH v10 7/8] tests/functional/x86_64: add migration test for NVMe device Date: Thu, 11 Jun 2026 20:08:40 +0200 Message-ID: <20260611180842.6390-8-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::331; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x331.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201428741158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce a very simple test to ensure that NVMe device migration works fine. Test plan is simple: 1. prepare VM with NVMe device 2. run workload that produces relatively heavy IO on the device 3. migrate VM 4. ensure that workload is alive and finishes without errors Test can be run as simple as: $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build In the future we can extend this approach, and introduce some fio-based tests. And probably, it makes sense to make this test to apply not only to NVMe device, but also virtio-{blk,scsi}, ide, sata and other migratable devices. Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn Acked-by: Fabiano Rosas --- v9: - check-patch fixes --- MAINTAINERS | 1 + tests/functional/x86_64/meson.build | 1 + .../functional/x86_64/test_nvme_migration.py | 172 ++++++++++++++++++ 3 files changed, 174 insertions(+) create mode 100755 tests/functional/x86_64/test_nvme_migration.py diff --git a/MAINTAINERS b/MAINTAINERS index 2b5b581e173..d705f5c8e0a 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2622,6 +2622,7 @@ S: Supported F: hw/nvme/* F: include/block/nvme.h F: tests/qtest/nvme-test.c +F: tests/functional/x86_64/test_nvme_migration.py F: docs/system/devices/nvme.rst T: git git://git.infradead.org/qemu-nvme.git nvme-next =20 diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/= meson.build index 1ed10ad6c29..fd77f19d726 100644 --- a/tests/functional/x86_64/meson.build +++ b/tests/functional/x86_64/meson.build @@ -37,6 +37,7 @@ tests_x86_64_system_thorough =3D [ 'linux_initrd', 'multiprocess', 'netdev_ethtool', + 'nvme_migration', 'replay', 'reverse_debug', 'tuxrun', diff --git a/tests/functional/x86_64/test_nvme_migration.py b/tests/functio= nal/x86_64/test_nvme_migration.py new file mode 100755 index 00000000000..890f0aab6d6 --- /dev/null +++ b/tests/functional/x86_64/test_nvme_migration.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# x86_64 NVMe migration test + +from migration import MigrationTest +from qemu_test import QemuSystemTest, Asset +from qemu_test import wait_for_console_pattern +from qemu_test import exec_command, exec_command_and_wait_for_pattern + + +class X8664NVMeMigrationTest(MigrationTest): + ASSET_KERNEL =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/vmlinuz'), + 'd4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129') + + ASSET_INITRD =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/initrd.img'), + '277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b') + + ASSET_DISKIMAGE =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Cloud/x86_64/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2'), + 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0') + + DEFAULT_KERNEL_PARAMS =3D ('root=3D/dev/nvme0n1p1 console=3DttyS0 net.= ifnames=3D0 ' + 'rd.rescue quiet') + + def wait_for_console_pattern(self, success_message, vm): + wait_for_console_pattern( + self, + success_message, + failure_message=3D"Kernel panic - not syncing", + vm=3Dvm, + ) + + def exec_command_and_check(self, command, vm): + prompt =3D '# ' + exec_command_and_wait_for_pattern(self, + f"{command} && echo OK || echo FAI= L", + 'FAIL', vm=3Dvm) + # Note, that commands we send to the console are echo-ed back, + # so if we have a word "FAIL" in the command itself, we should + # expect to see it once. + wait_for_console_pattern(self, 'OK', failure_message=3D"FAIL", vm= =3Dvm) + self.wait_for_console_pattern(prompt, vm) + + def configure_machine(self, vm): + kernel_path =3D self.ASSET_KERNEL.fetch() + initrd_path =3D self.ASSET_INITRD.fetch() + diskimage_path =3D self.ASSET_DISKIMAGE.fetch() + + vm.set_console() + vm.add_args("-cpu", "max") + vm.add_args("-m", "2G") + vm.add_args("-accel", "kvm") + + vm.add_args('-drive', + f'file=3D{diskimage_path},if=3Dnone,id=3Ddrv0,sna= pshot=3Don') + vm.add_args('-device', 'nvme,bus=3Dpcie.0,' + + 'drive=3Ddrv0,id=3Dnvme-disk0,serial=3Dnvmemigtest,boo= tindex=3D1') + + vm.add_args( + "-kernel", + kernel_path, + "-initrd", + initrd_path, + "-append", + self.DEFAULT_KERNEL_PARAMS + ) + + def launch_source_vm(self, vm): + vm.launch() + + self.wait_for_console_pattern('Entering emergency mode.', vm) + prompt =3D '# ' + self.wait_for_console_pattern(prompt, vm) + + # Synchronize on NVMe driver creating the root device + exec_command_and_wait_for_pattern(self, + "while ! (dmesg -c | grep nvme0n1:) ; do sleep 1 ;= done", + "nvme0n1", vm=3Dvm) + self.wait_for_console_pattern(prompt, vm) + + # prepare system + exec_command_and_wait_for_pattern(self, 'mount /dev/nvme0n1p1 /sys= root', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'chroot /sysroot', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t proc proc /proc', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs /sys= ', + prompt, vm=3Dvm) + + # Run workload before migration to check if it continues + # to run properly after migration. + # + # Workload is simple: it continuously calculates checksums of + # all files in /usr/bin to generate some I/O load on + # the NVMe disk and at the same time it drops caches to + # make sure that we have some read I/O on the disk as well. + # If there are any issues with the migration of the NVMe device, + # we should see errors in dmesg and consequently in the workload l= og. + exec_command_and_wait_for_pattern(self, + "(while [ ! -f /tmp/test_nvme_mig_workload.stop ]; do \ + rm -f /tmp/test_nvme_mig_workload.iter_finished; \ + echo 3 > /proc/sys/vm/drop_caches; \ + find /usr/bin -type f -exec cksum {} \\;; \ + touch /tmp/test_nvme_mig_workload.iter_finished; \ + done) > /dev/null 2> /tmp/test_nvme_mig_workload.errors &", + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, + 'echo $! > /tmp/test_nvme_mig_workload.pid', + prompt, vm=3Dvm) + + # check if process is alive and running + self.exec_command_and_check( + "kill -0 $(cat /tmp/test_nvme_mig_workload.pid)", vm) + + def assert_dest_vm(self, vm): + prompt =3D '# ' + + # check if process is alive and running after migration, + # if not - fail the test + self.exec_command_and_check( + "kill -0 $(cat /tmp/test_nvme_mig_workload.pid)", vm) + + # signal workload to stop + exec_command_and_wait_for_pattern(self, + 'touch /tmp/test_nvme_mig_workload.stop', + prompt, vm=3Dvm) + + # wait workload to finish, because we want to examine log + # to see if there are any errors + exec_command_and_wait_for_pattern(self, + "while [ ! -f /tmp/test_nvme_mig_workload.iter_finished ]; do \ + sleep 1; \ + done;", + prompt, vm=3Dvm) + + exec_command_and_wait_for_pattern(self, + 'cat /tmp/test_nvme_mig_workload.errors', + prompt, vm=3Dvm) + + # fail the test if non-empty + self.exec_command_and_check( + "[ ! -s /tmp/test_nvme_mig_workload.errors ]", vm) + + def test_migration_with_tcp_localhost(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_tcp_localhost() + + def test_migration_with_unix(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_unix() + + def test_migration_with_exec(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_exec() + + +if __name__ =3D=3D '__main__': + MigrationTest.main() --=20 2.47.3 From nobody Fri Jun 12 01:36:32 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1781201389; cv=none; d=zohomail.com; s=zohoarc; b=LAGOXS+gZVzqWBlDpzExBhZX83WCorbXhSjZOTRkrqTzEiBhXynQ2DZpZj7RaKBaJItkSD7hVAatTnVfD/p+rjd1wSAa33C2pzY4OJgvjyQNEx5eokeGgNWhdfbLRff+nbJytFyaEyxEeUr+lWAKxU7isgTUgygaL/tCgfmgRhQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1781201389; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=oqpifdS8v655f8ACfjXuhW+OTaqj54gbCvMyV9BvtmQ=; b=bx+CqG1/gQ6qgdu950z52AbEQjeOKNj5sPhHQa6zFXlF/KwqtA/mF1XpEwIjXX20qgGM0trekQ6WQ96xRSxHRludE7uuQGU6al+fMoyRAzJfNgQslW/w2224a79RrO1MUiM7nrYKSxwz+Wlkm43Zw3Yp6uQtW7RTY5pUVExIbVU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1781201389009173.4595537849948; Thu, 11 Jun 2026 11:09:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wXjpq-00074m-K4; Thu, 11 Jun 2026 14:08:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wXjpp-00073c-58 for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:57 -0400 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wXjpm-0002KE-Ek for qemu-devel@nongnu.org; Thu, 11 Jun 2026 14:08:56 -0400 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-490b211ee6aso593745e9.3 for ; Thu, 11 Jun 2026 11:08:54 -0700 (PDT) Received: from alex-laptop.lan (p200300cf570ffe006f778db3bc22ef81.dip0.t-ipconnect.de. [2003:cf:570f:fe00:6f77:8db3:bc22:ef81]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea4a128csm6563345e9.0.2026.06.11.11.08.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 11:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1781201333; x=1781806133; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oqpifdS8v655f8ACfjXuhW+OTaqj54gbCvMyV9BvtmQ=; b=GUPaALM+zPemSLXV6uqZPQfLYokJcWVcAuuO9oTgWy2LMmIOVSweL2sWl28EEyyBay uXMyWFrS+1S1pmstmPajnQES6PGlKh9iZ7rUmeHLf2qlqXd7W3DM48ELj/5xf4IIZ3JQ 43YH3QkefPjoJ8zIxk6ypajOOYFUcMZ9sdbLk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781201333; x=1781806133; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=oqpifdS8v655f8ACfjXuhW+OTaqj54gbCvMyV9BvtmQ=; b=Eqay0Fi4s+K8nmW2I2zMBUthHNz3aOtqz/R68moHWgolEGFJxh9U+V7Y1SfQogW3DH mLwjznB7h7sFcRo44crL73WGPWQUIYPr3nP2SUta3UN1GdJ4wjjxuLmz7ih6YseByQ4b C518K73lQAmSjXlq69+FijuX9iuMARF6NYlNMBxeE+0lCf3FedNDfHFvfEjruvcEMpxV Uv8k/kAYvJBl4ETUmPxThLP3PRACdDr29smwy/5UgbS5M9mJAAuax1FXeo2tpMOuDk3v g26pP21TIrHRSGJClOJqMxJLU42nBHHVtU1JOtE+4jjtMoQ/9TCokkplPmOzymFmZVnd aOYQ== X-Gm-Message-State: AOJu0YyM7zecWFZW3/3yE1SxttffxDyXKttf2/fDFzmph9cpZil7AmcG dExn1iWy4tBNz/cdyyIhz205bxYXs4EAobo4n1jqsju6LccvJL2CxwTHM5unLsJ+ZWTDRkNmbCy oE5r4oZY= X-Gm-Gg: Acq92OFQtEjCqRPIJxhReHxo4XdjKh0TSnou3XexluIcK6ieGxbedYT5hX0WSgNRWpZ hT9xSzq9UnGQv55WfHCqtUoIMHmznFvlWL8Eg5M7w0Cb2NITJuBFBbyIla2QsRmIqVtfUEWGSaG ahTTysWu4mGdX4VltW8B2UQADJwsOCSU/XuITrcTLRnM/AmI3mBfpiOQ4sJbBlgHUBS1HSC2xK9 mW/e09dRvqOWDJIQ1K4xQcmBogDqf3+OHBgO2BZIEkev1YF5lrCDFSsX9m9Jbmpp6eb9fwhQa38 x6z8h8yQiehvTIQpSs8Vl56k+dYbQDSE/t1PSMPkOR4lW/sLoNRqdna787tKLRt3Kv3ihTJt/1o Mn8ii4QB+8rOyKLcE3z7W4YeyJOth7A/DzFJkHBb+fvqHfv4QVUB5/K9UeooDHNWnE1xgqj+63H EY9PCEMXdbHkue9DwWyQHX6QcNcvLSY8S/gFrCD7frP/7ArotLnC0sCeKZxJTldBGjFE4o83yc9 nHcojTMDqyuO6q2IbhbXJY= X-Received: by 2002:a05:600c:5297:b0:490:e5c1:b88d with SMTP id 5b1f17b1804b1-490e5d0fd2bmr62201105e9.1.1781201332951; Thu, 11 Jun 2026 11:08:52 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Klaus Jensen , Peter Xu , Kevin Wolf , Fabiano Rosas , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Keith Busch , =?UTF-8?q?St=C3=A9phane=20Graber?= , Zhao Liu , Alexander Mikhalitsyn , qemu-block@nongnu.org, Laurent Vivier , Stefan Hajnoczi , Paolo Bonzini , Hanna Reitz , Jesper Devantier , Fam Zheng , Alexander Mikhalitsyn Subject: [PATCH v10 8/8] tests/qtest/nvme-test: add migration test with full CQ Date: Thu, 11 Jun 2026 20:08:41 +0200 Message-ID: <20260611180842.6390-9-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260611180842.6390-1-alexander@mihalicyn.com> References: <20260611180842.6390-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x330.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1781201390428158501 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn As suggested by Stefan [1], let's add a migration test to cover rare scenario when CQ is full of non-processed CQEs and migration happens. To run this test: $ meson test -C build 'qtest-x86_64/qos-test' Link: https://lore.kernel.org/qemu-devel/20260408183529.GB319710@fedora/ [1] Suggested-by: Stefan Hajnoczi Acked-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn Acked-by: Fabiano Rosas --- v10: - replace %lx with 0x%" PRIx64 for q->doorbell [ fix cross-win64-system test ] - get rid of PAGE_SIZE macro and explicitly request proper amount of mem [ build-system-alpine test was failing on PAGE_SIZE macro redefinitio= n ] v9: - check-patch fixes - added qpci_check_buggy_msi() check to skip test on ppc64 / spapr pci = bus v7: - fixed endianness bugs (and tested on s390x machine) - code style changes (don't use ptr type for physical addresses) v6: - test added --- tests/qtest/nvme-test.c | 419 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 419 insertions(+) diff --git a/tests/qtest/nvme-test.c b/tests/qtest/nvme-test.c index 4aec1651e6e..f9395cc252b 100644 --- a/tests/qtest/nvme-test.c +++ b/tests/qtest/nvme-test.c @@ -8,9 +8,12 @@ */ =20 #include "qemu/osdep.h" +#include +#include "qemu/bswap.h" #include "qemu/module.h" #include "qemu/units.h" #include "libqtest.h" +#include "libqtest-single.h" #include "libqos/qgraph.h" #include "libqos/pci.h" #include "block/nvme.h" @@ -142,6 +145,420 @@ static void nvmetest_pmr_reg_test(void *obj, void *da= ta, QGuestAllocator *alloc) qpci_iounmap(pdev, pmr_bar); } =20 +typedef struct nvme_ctrl nvme_ctrl; + +typedef struct nvme_queue { + nvme_ctrl *ctrl; + uint64_t doorbell; + uint32_t size; +} nvme_queue; + +typedef struct nvme_cq { + nvme_queue common; + uint64_t phys_cqe; /* NvmeCqe* */ + uint16_t head; + uint8_t phase; +} nvme_cq; + +typedef struct nvme_sq { + nvme_queue common; + uint64_t phys_sqe; /* NvmeCmd* */ + nvme_cq *cq; + uint16_t head; + uint16_t tail; +} nvme_sq; + +struct nvme_ctrl { + QGuestAllocator *alloc; + QPCIDevice *pdev; + QPCIBar bar; + + uint32_t db_stride; + + nvme_sq admin_sq; + nvme_cq admin_cq; +}; + +#define PHYS_ADDR_OF_FIELD(T, base_phys_addr, field) \ + ((uint64_t)&((T *)(base_phys_addr))->field) + +#define PHYS_ADDR_OF(T, base_phys_addr, accessor) \ + ((uint64_t)&((T *)(base_phys_addr))accessor) + +static void nvme_init_queue_common(nvme_ctrl *ctrl, nvme_queue *q, + uint16_t db_idx, uint32_t size) +{ + q->ctrl =3D ctrl; + q->doorbell =3D (sizeof(NvmeBar) + db_idx * ctrl->db_stride); + g_test_message(" q %p db_idx %u doorbell 0x%" PRIx64, q, db_idx, q->do= orbell); + q->size =3D size; +} + +static void nvme_init_sq(nvme_ctrl *ctrl, nvme_sq *sq, uint16_t db_idx, + uint32_t size, nvme_cq *cq) +{ + nvme_init_queue_common(ctrl, &sq->common, db_idx, size); + + sq->phys_sqe =3D guest_alloc(ctrl->alloc, sizeof(NvmeCmd) * size); + g_assert(sq->phys_sqe); + + g_test_message("sq %p db_idx %u sqe 0x%" PRIx64, sq, db_idx, sq->phys_= sqe); + sq->cq =3D cq; + sq->head =3D 0; + sq->tail =3D 0; +} + +static void nvme_init_cq(nvme_ctrl *ctrl, nvme_cq *cq, uint16_t db_idx, + uint32_t size) +{ + nvme_init_queue_common(ctrl, &cq->common, db_idx, size); + + cq->phys_cqe =3D guest_alloc(ctrl->alloc, sizeof(NvmeCqe) * size); + g_assert(cq->phys_cqe); + + g_test_message("cq %p db_idx %u cqe 0x%" PRIx64, cq, db_idx, cq->phys_= cqe); + cq->head =3D 0; + cq->phase =3D 1; +} + +static int nvme_cqe_pending(nvme_cq *cq) +{ + uint16_t status =3D qtest_readw( + cq->common.ctrl->pdev->bus->qts, + PHYS_ADDR_OF(NvmeCqe, cq->phys_cqe, [cq->head].status)); + return (status & 1) =3D=3D cq->phase; +} + +static int nvme_is_cqe_success(NvmeCqe *cqe) +{ + return (le16_to_cpu(cqe->status) >> 1) =3D=3D NVME_SUCCESS; +} + +static NvmeCqe nvme_handle_cqe(nvme_sq *sq) +{ + nvme_cq *cq =3D sq->cq; + uint64_t phys_cqe =3D PHYS_ADDR_OF( + NvmeCqe, cq->phys_cqe, [cq->head]); /* NvmeCqe= * */ + NvmeCqe cqe; + uint16_t cq_next_head; + + g_assert(nvme_cqe_pending(cq)); + + qtest_memread(sq->common.ctrl->pdev->bus->qts, phys_cqe, &cqe, sizeof(= cqe)); + + cq_next_head =3D (cq->head + 1) % cq->common.size; + g_test_message("cq %p head %u -> %u", cq, cq->head, cq_next_head); + if (cq_next_head < cq->head) { + cq->phase ^=3D 1; + } + cq->head =3D cq_next_head; + + if (cqe.sq_head !=3D sq->head) { + sq->head =3D cqe.sq_head; + g_test_message("sq %p head =3D %u", sq, sq->head); + } + + qpci_io_writel(cq->common.ctrl->pdev, cq->common.ctrl->bar, + cq->common.doorbell, cq->head); + + return cqe; +} + +static NvmeCqe nvme_wait(nvme_sq *sq) +{ + int i; + bool ready =3D false; + + for (i =3D 0; i < 10; i++) { + if (nvme_cqe_pending(sq->cq)) { + ready =3D true; + break; + } + + g_usleep(1000); + } + + g_assert(ready); + + return nvme_handle_cqe(sq); +} + +static uint64_t nvme_get_next_sqe(nvme_sq *sq, uint8_t opcode, + uint16_t cid, uint64_t prp1) +{ + uint64_t phys_sqe =3D PHYS_ADDR_OF(NvmeCmd, sq->phys_sqe, [sq->tail]); + + if (((sq->tail + 1) % sq->common.size) =3D=3D sq->head) { + /* no space in SQ */ + g_test_message("%s head %d tail %d", __func__, sq->head, sq->tail); + g_assert_not_reached(); + return 0; + } + + qtest_memset(sq->common.ctrl->pdev->bus->qts, + phys_sqe, 0, sizeof(NvmeCmd)); + + #define GUEST_MEM_WRITE(fn, phys_addr, val) \ + fn(sq->common.ctrl->pdev->bus->qts, phys_addr, (val)) + + GUEST_MEM_WRITE(qtest_writeb, + PHYS_ADDR_OF_FIELD(NvmeCmd, phys_sqe, opcode), opcode); + GUEST_MEM_WRITE(qtest_writew, + PHYS_ADDR_OF_FIELD(NvmeCmd, phys_sqe, cid), cid); + GUEST_MEM_WRITE(qtest_writeq, + PHYS_ADDR_OF_FIELD(NvmeCmd, phys_sqe, dptr.prp1), prp1= ); + + #undef GUEST_MEM_WRITE + + g_test_message("sq %p next_sqe %u sqe 0x%" PRIx64, sq, sq->tail, phys_= sqe); + return phys_sqe; +} + +static void nvme_commit_sqe(nvme_sq *sq) +{ + g_test_message("sq %p commit sqe tail %u", sq, sq->tail); + sq->tail =3D (sq->tail + 1) % sq->common.size; + qpci_io_writel(sq->common.ctrl->pdev, sq->common.ctrl->bar, + sq->common.doorbell, sq->tail); +} + +static uint64_t nvme_admin_identify_ctrl(nvme_ctrl *ctrl, + uint16_t cid, bool no_wait) +{ + uint64_t phys_cmd_identify; /* NvmeCmd* */ + uint64_t phys_identify; /* NvmeIdCtrl* */ + NvmeCqe cqe; + + g_test_message("sending req cid %u no_wait %d", cid, no_wait); + + phys_identify =3D guest_alloc(ctrl->alloc, sizeof(NvmeIdCtrl)); + g_assert(phys_identify); + + phys_cmd_identify =3D nvme_get_next_sqe(&ctrl->admin_sq, + NVME_ADM_CMD_IDENTIFY, cid, + phys_identify); + g_assert(phys_cmd_identify); + + #define GUEST_MEM_WRITE(fn, phys_addr, val) \ + fn(ctrl->pdev->bus->qts, phys_addr, (val)) + + GUEST_MEM_WRITE(qtest_writel, + PHYS_ADDR_OF_FIELD(NvmeCmd, phys_cmd_identify, nsid), = 0); + GUEST_MEM_WRITE(qtest_writel, + PHYS_ADDR_OF_FIELD(NvmeIdentify, phys_cmd_identify, cn= s), + NVME_ID_CNS_CTRL); + + #undef GUEST_MEM_WRITE + + nvme_commit_sqe(&ctrl->admin_sq); + + if (no_wait) { + return phys_identify; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + g_assert(le16_to_cpu(cqe.cid) =3D=3D cid); + + return phys_identify; +} + +static void nvme_wait_ready(nvme_ctrl *ctrl, int val) +{ + int i; + + for (i =3D 0; i < 10; i++) { + uint32_t csts =3D qpci_io_readl(ctrl->pdev, ctrl->bar, NVME_REG_CS= TS); + g_test_message("%s: csts %x", __func__, csts); + + if (NVME_CSTS_RDY(csts) =3D=3D val) { + return; + } + + g_usleep(1000); + } + + g_assert_not_reached(); +} + +static void test_migrate_setup_nvme_ctrl(nvme_ctrl *ctrl) +{ + uint64_t cap; + + /* disable controller */ + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, 0); + nvme_wait_ready(ctrl, 0); + + cap =3D qpci_io_readq(ctrl->pdev, ctrl->bar, NVME_REG_CAP); + ctrl->db_stride =3D 4 << NVME_CAP_DSTRD(cap); + + nvme_init_cq(ctrl, &ctrl->admin_cq, 1, 2 /* CQEs num */); + nvme_init_sq(ctrl, &ctrl->admin_sq, 0, 4 /* SQEs num */, &ctrl->admin_= cq); + + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_AQA, + ((ctrl->admin_cq.common.size - 1) << AQA_ACQS_SHIFT) | + ((ctrl->admin_sq.common.size - 1) << AQA_ASQS_SHIFT) + ); + + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ASQ, (uint64_t)ctrl->admin_sq.phys_sqe); + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ACQ, (uint64_t)ctrl->admin_cq.phys_cqe); + + /* enable controller */ + { + uint32_t cc =3D 0; + NVME_SET_CC_EN(cc, 1); + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, cc); + } + + nvme_wait_ready(ctrl, 1); +} + +typedef struct test_migrate_req { + uint16_t cid; + bool handle_cqe; + uint64_t phys_identify; /* NvmeIdCtrl* */ +} test_migrate_req; + +static void test_migrate_send_nvme_reqs(nvme_ctrl *ctrl, test_migrate_req = *reqs, + int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + reqs[i].phys_identify =3D nvme_admin_identify_ctrl(ctrl, reqs[i].c= id, + !reqs[i].handle_c= qe); + g_assert(reqs[i].phys_identify); + + if (reqs[i].handle_cqe) { + guest_free(ctrl->alloc, reqs[i].phys_identify); + } + } +} + +static void test_migrate_check_nvme(nvme_ctrl *ctrl, + test_migrate_req *reqs, int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + NvmeCqe cqe; + + if (reqs[i].handle_cqe) { + continue; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + + g_assert_cmpint(le16_to_cpu(cqe.cid), =3D=3D, reqs[i].cid); + + #define GUEST_MEM_READB(phys_addr) \ + qtest_readb(ctrl->pdev->bus->qts, (phys_addr)) + + g_assert_cmpint(GUEST_MEM_READB( + PHYS_ADDR_OF_FIELD(NvmeIdCtrl, reqs[i].phys_identify, ieee[0])= ), + =3D=3D, 0x0); + g_assert_cmpint(GUEST_MEM_READB( + PHYS_ADDR_OF_FIELD(NvmeIdCtrl, reqs[i].phys_identify, ieee[1])= ), + =3D=3D, 0x54); + g_assert_cmpint(GUEST_MEM_READB( + PHYS_ADDR_OF_FIELD(NvmeIdCtrl, reqs[i].phys_identify, ieee[2])= ), + =3D=3D, 0x52); + + #undef GUEST_MEM_READB + + guest_free(ctrl->alloc, reqs[i].phys_identify); + } +} + +static void test_migrate(void *obj, void *data, QGuestAllocator *alloc) +{ + g_autofree gchar *tmpfs =3D NULL; + GError *err =3D NULL; + g_autofree gchar *mig_path =3D NULL; + g_autofree gchar *uri =3D NULL; + GString *dest_cmdline; + QTestState *to; + QDict *rsp; + QNvme *nvme =3D obj; + QPCIDevice *pdev =3D &nvme->dev; + g_autofree nvme_ctrl *ctrl =3D NULL; + test_migrate_req test_reqs[] =3D { + { 123, true }, + { 456, false }, + { 300, false }, + { 333, false } + }; + + if (qpci_check_buggy_msi(pdev)) { + return; + } + + /* create temporary dir and prepare unix socket path for migration */ + tmpfs =3D g_dir_make_tmp("nvme-test-XXXXXX", &err); + if (!tmpfs) { + g_test_message("Can't create temporary directory in %s: %s", + g_get_tmp_dir(), err->message); + g_error_free(err); + } + g_assert(tmpfs); + + mig_path =3D g_strdup_printf("%s/socket.mig", tmpfs); + uri =3D g_strdup_printf("unix:%s", mig_path); + + /* enable NVMe PCI device */ + qpci_device_enable(pdev); + + ctrl =3D g_malloc0(sizeof(*ctrl)); + ctrl->alloc =3D alloc; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D global_qtest); + + test_migrate_setup_nvme_ctrl(ctrl); + test_migrate_send_nvme_reqs(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + dest_cmdline =3D g_string_new(qos_get_current_command_line()); + g_string_append_printf(dest_cmdline, " -incoming %s", uri); + + /* Create destination VM */ + to =3D qtest_init(dest_cmdline->str); + + /* Get access to PCI device from destination VM */ + nvme =3D qos_allocate_objects(to, &ctrl->alloc); + pdev =3D &nvme->dev; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D to); + + /* Migrate VM */ + rsp =3D qmp("{ 'execute': 'migrate', 'arguments': { 'uri': %s } }", ur= i); + g_assert(qdict_haskey(rsp, "return")); + qobject_unref(rsp); + + /* Wait when source VM is stopped */ + qmp_eventwait("STOP"); + + /* Copy guest physical memory allocator state */ + migrate_allocator(alloc, ctrl->alloc); + + /* Wait for destination VM to become alive */ + qtest_qmp_eventwait(to, "RESUME"); + + test_migrate_check_nvme(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + qtest_quit(to); + g_unlink(mig_path); + g_rmdir(tmpfs); + g_string_free(dest_cmdline, true); +} + static void nvme_register_nodes(void) { QOSGraphEdgeOptions opts =3D { @@ -168,6 +585,8 @@ static void nvme_register_nodes(void) }); =20 qos_add_test("reg-read", "nvme", nvmetest_reg_read_test, NULL); + + qos_add_test("migrate", "nvme", test_migrate, NULL); } =20 libqos_init(nvme_register_nodes); --=20 2.47.3