From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603779; cv=none; d=zohomail.com; s=zohoarc; b=H8+YqWoTTwWuII4kQJn0KVaLFQY6SgP7iC/4fc8ga8s8SRx4IzQpv/8CuBKNqxjQW8ejcHK86SVtF67gaYo7yMj/DwucBRwbEcHVvBuDAzf/Ap3hTHx5eiii7rSlcfZjXDZAO5ez9SEvwgYG0gvuLkXczctCcxItSXGYzwMcgxU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603779; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=9OztW63tvsDLU5EHuMGMvPzYUtmXl7GocMNqQvsnffQ=; b=juO6sRY5NkoaJwDu8RJI10yyOqILF6cuhFIvNd6or9tV2twGVgaDhB8mO7rR+yGhrOxzrYey4zRqaWsfKRlOHFZRM2Qy/rP67NyrmVrsDujRdEgcha79Rxhzj0KnFUqJyCYTK0BMxpPFEaKcyFLOi7Y6gFspaPWGrma7fpzOT80= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603778925978.0491407525426; Sun, 19 Apr 2026 06:02:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERmz-0000xD-8z; Sun, 19 Apr 2026 09:02:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERme-0000tm-Uc for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:00 -0400 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERmX-0008IU-RU for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:01:54 -0400 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-43d734223e4so1419024f8f.0 for ; Sun, 19 Apr 2026 06:01:44 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603703; x=1777208503; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9OztW63tvsDLU5EHuMGMvPzYUtmXl7GocMNqQvsnffQ=; b=Wz5mhKN5kiETJuPc4AHKD8m/D1YkXPnp3OBO1+96VTJ5Kc0zHNSFuZUDMHe6mIKl3T fkKd5Dqq0CT4KOWrNzIATEfSinAJNTlaojLcQd0bPkDnM272Y01EemNam0HsIMbA9X5l 40bacxg5Yt4H9KmfYAsLdA7gc1nY3BAYQ0/+Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603703; x=1777208503; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9OztW63tvsDLU5EHuMGMvPzYUtmXl7GocMNqQvsnffQ=; b=Em7/hcpNh13wFXn4ElSfSqCTvwSuDP7CLacsqXhbYIaTieoxVAeyKKy9yv6/UeBbw0 6zog2N1Ak51do4uaadV7hL62fSjJ3kPzCOX3Ik9B0WbFTKXzbXf3gAAyh6ew1iN6Etq5 bh4RAMeLwsBtZNG7xHpkWXyP1KRgM6PCnM+CoMDHpUv4CY5LGO1B72dl+3m6VX4uZUIv 6SOr2Zx23i5j9LvMLEqLjq00XoA21xe2FjXpD0ueMm9gVn2+YWEBrKWIKvsF0z2R1swF dgrBWQ550jg0pmlijcMQhQN/GMYt3Qub6L5Krp4ILAgSTlJr9W/2pSvhiICo9abSWeZZ 8CZQ== X-Gm-Message-State: AOJu0YyaqU7GbGuobZbCEhw8/uITdeObJqMrjF6Z7UNiQtqFOA8C8lsY Fc3O2VF4O5CHAoJbO21uSMYYs7J5gxQW9y0dh6SvW85iZUcI41u6+CyRdBA/TICCKJBdiV4cuAU DKgsL2Os= X-Gm-Gg: AeBDieuP61Bx1wRJv4E/cxi+BBOuBZVkb+2X84FThCTGupxGzjxppHzHzrxq0oTQgIt kK3CtwxBOq1XnItqGZliCXonhrL6WJsZxxdvLVD0QNI46D9c+iVgi/Pr6k3vrgr06J9XzViYUom bB2rJs68xvUACajWe1FYGxAGDEfFYRHEFdYV/O1t0yXEs0DFTUHydK43Mk7OK8BZKH+K10ExREt 6HXS/OhcXK5eOEqz7Pm8B/4KpCulpK9DYpUun35KrRBhuxXMID9+lob1hwmUtiadbs1J6guNun7 QaGDi+tifYbjBKhlGbYqME8zfkC0IPK8AnumdWH7dhqDUs/h7g/ycW8kqBOCmzky0yHVd/7cXFe Jy26st/jz3+mv3d5KQRl/X6TZv2hCE9vpRfjS3hTLLxoeDZjPuPtY3DAFfUzyTIKVmL0kCZ9M6m YsMMxmIg3V5Ru6h65m1MQ/K/hjClZpTNPxQUiYM3h6Mtzf0A3SlDFsDtLkMOILsnI533KFvrdM6 bJuhHQ5SOuDvMPSw51UA2/x952X4FYTSA== X-Received: by 2002:a5d:64e5:0:b0:43d:7606:5a47 with SMTP id ffacd0b85a97d-43fe3f358a5mr14523615f8f.2.1776603703174; Sun, 19 Apr 2026 06:01:43 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 1/8] tests/functional/migration: add VM launch/configure hooks Date: Sun, 19 Apr 2026 15:01:32 +0200 Message-ID: <20260419130139.15554-2-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42e; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x42e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603782501158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce configure_machine, launch_source_vm and assert_dest_vm methods to allow child classes to override some pieces of source/dest VMs creation, start and check logic. Signed-off-by: Alexander Mikhalitsyn --- tests/functional/migration.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/functional/migration.py b/tests/functional/migration.py index 2395119d6c6..9b5b31efe3e 100644 --- a/tests/functional/migration.py +++ b/tests/functional/migration.py @@ -40,19 +40,35 @@ def assert_migration(self, src_vm, dst_vm): self.assertEqual(dst_vm.cmd('query-status')['status'], 'running') self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate= ') =20 + # Can be overridden by subclasses to configure both source/dest VMs. + def configure_machine(self, vm): + vm.add_args('-nodefaults') + + # Can be overridden by subclasses to prepare the source VM before migr= ation, + # e.g. by running some workload inside the source VM to see if it cont= inues + # to run properly after migration. + def launch_source_vm(self, vm): + vm.launch() + + # Can be overridden by subclasses to check the destination VM after mi= gration, + # e.g. by checking if the workload is still running after migration. + def assert_dest_vm(self, vm): + pass + def migrate_vms(self, dst_uri, src_uri, dst_vm, src_vm): dst_vm.qmp('migrate-incoming', uri=3Ddst_uri) src_vm.qmp('migrate', uri=3Dsrc_uri) self.assert_migration(src_vm, dst_vm) + self.assert_dest_vm(dst_vm) =20 def migrate(self, dst_uri, src_uri=3DNone): dst_vm =3D self.get_vm('-incoming', 'defer', name=3D"dst-qemu") - dst_vm.add_args('-nodefaults') + self.configure_machine(dst_vm) dst_vm.launch() =20 src_vm =3D self.get_vm(name=3D"src-qemu") - src_vm.add_args('-nodefaults') - src_vm.launch() + self.configure_machine(src_vm) + self.launch_source_vm(src_vm) =20 if src_uri is None: src_uri =3D dst_uri --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603841; cv=none; d=zohomail.com; s=zohoarc; b=Z6BIGv/QY9vWikQ2Xl7fbMDYhZcdyIxRBLrGXVraXOiaDu6QFaVPOKHHimmuobbEAAnJoTMZtXOoJ0VIYpgVUT8LqbQL1B3xItL/w8+zgitqCJAfK4tC0mr4bQcC1h3bYHf7yk32zHDpnn4bJAHtwjTZOo298m7tbEEmp7DzE/I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603841; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SEO6/lrYdm4dOs+1aF7CdwIzHMn2tXQZYpY2v/1+t2k=; b=AuzTadqyMT+RNA6+jgrzOAT+ujp5FLlG5iZGBZbR5rbO43t1O3croEpaW986Tu5IDN06kiGk6z+31yFsS6V52fdG4ezrsYxhCgxKCbEYfnGCRI/aQJiEn35dIdb8sVkqhbB6dD77hC6xtLKxT0BQjvZx6xlWVizJOlv4996mayQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603841352585.0097362161433; Sun, 19 Apr 2026 06:04:01 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERn1-00011D-6o; Sun, 19 Apr 2026 09:02:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERme-0000tj-UI for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:00 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERmX-0008If-Ss for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:01:54 -0400 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-488b00ed86fso22799655e9.3 for ; Sun, 19 Apr 2026 06:01:45 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603704; x=1777208504; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEO6/lrYdm4dOs+1aF7CdwIzHMn2tXQZYpY2v/1+t2k=; b=M3NbKoxdfi7Uy4BiVTE439mbvHPvGCiRr9H3p/m5Ku2eYtfcAFDkUtubz+f82sGkdD 13R3atZdMCf9wBgVX3SmrSrlutPUdDbyj2ROHL3Zbdi9z6y1qiApJBShq7Hu+NMdpqFt xw8Zdd014efQ4h00RzuZVqb+SlOHSHWFFeDp4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603704; x=1777208504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=SEO6/lrYdm4dOs+1aF7CdwIzHMn2tXQZYpY2v/1+t2k=; b=JsoYMn1+7A/goQSeT+HbO+qdfS19Pz7jGiZlqLYf59kQA6p8VPUL6BzlJznG6YXd4D QCi20/z7lO2xiZQ2Tx0mI1x2T30M7ZgB1cYocsW5O0tF5tWsecgyRXTfgJbSV00qXVa2 n6i6mpeCIN0xVCrr6P458PJUMDNOfUoE8OJzHFJemZkxU17grJ32YtvJ6t73aR9pIxOR JusT0lqCQtyOPlUyDLtCZE+DITTIBwnoQWQiJVbYFnvXS//z3/46GKOUbmkeROTbVcIl CEl63/mkhchg/axSBLC0eRgzRapK+Q7JmHtpRt+7cCA4d/fzKg/yXQSAB+hOtRAR0kmv b4aA== X-Gm-Message-State: AOJu0Yw3ZjmPZCYdcqcci1kU78L7rfUqg+pZytuvj1CIZ7ESNNyQ6svB Ahx9g7ZYQNTVEnIeZzrFILPsOYJ95qx7/0layeDrSMqi9yulPGCW0JJQeZjiywLMKIYh38EF1tc WlfliLus= X-Gm-Gg: AeBDiesPH7jyRrlrZPhAYTnlrZjg9jGSF53xkp/hSdW2iJjGMj4Efj3xeo8l0XcNv8E +9ZjF+zYdgr/7XoN5KxYlBxqBWLPPXsdjIVRXbHuW2m2aTKIOZ7jL2T4bIQn9rC5y86ntx581Qz m79RPnhPSbXvqGhX/5oEbC66UXhmADR03X8fz24yRCsG2ILol2ZjBG/y2BFpu61g6ALuPyknm18 w4dNTgvqT/UIGBeR+LeCdzqh09H3yx0AFLz4mE0I/IV2cBOT6qtfl7foS/PeqipC2z5wKJ5QD3K 3i570YGWaecXaDy37Zw1XtvbChEHlxsaDOE7F3BCg/X2nHiWo4yQlr6XApcorccs96rta+BJ+mS UCGa9XXAO8GyRLOLH+S6oXL3ELfLL7DymneZht4wASxcegad5V2al7dhd83kxZom5BpSrg69eXb Zv6n3IFBS3r+4heFUxTOPMNWtCaBOxN8NH1rBQLXQMZYshnCV2L2/8YQpLNKfz+QO1WWTi/gIoF q62K02Ia9urn9udm99xqVv6USUckrh67w== X-Received: by 2002:a05:600c:5295:b0:485:30d4:6b9e with SMTP id 5b1f17b1804b1-488fb77facemr148012285e9.21.1776603704434; Sun, 19 Apr 2026 06:01:44 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 2/8] hw/nvme: add migration blockers for non-supported cases Date: Sun, 19 Apr 2026 15:01:33 +0200 Message-ID: <20260419130139.15554-3-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603842670154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Let's block migration for cases we don't support: - SR-IOV - CMB - PMR - SPDM No functional changes here, because NVMe migration is not supported at all as of this commit. Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 206 +++++++++++++++++++++++++++++++++++++++++++ hw/nvme/nvme.h | 3 + include/block/nvme.h | 12 +++ 3 files changed, 221 insertions(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index cc4593cd427..9f9c9bcbead 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -207,6 +207,7 @@ #include "hw/pci/msix.h" #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" +#include "migration/blocker.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -250,6 +251,7 @@ static const bool nvme_feature_support[NVME_FID_MAX] = =3D { [NVME_COMMAND_SET_PROFILE] =3D true, [NVME_FDP_MODE] =3D true, [NVME_FDP_EVENTS] =3D true, + /* if you add something here, please update nvme_set_migration_blocker= s() */ }; =20 static const uint32_t nvme_feature_cap[NVME_FID_MAX] =3D { @@ -4601,6 +4603,7 @@ static uint16_t nvme_io_mgmt_send(NvmeCtrl *n, NvmeRe= quest *req) return 0; case NVME_IOMS_MO_RUH_UPDATE: return nvme_io_mgmt_send_ruh_update(n, req); + /* if you add something here, please update nvme_set_migration_blocker= s() */ default: return NVME_INVALID_FIELD | NVME_DNR; }; @@ -7518,6 +7521,10 @@ static uint16_t nvme_security_receive(NvmeCtrl *n, N= vmeRequest *req) =20 static uint16_t nvme_directive_send(NvmeCtrl *n, NvmeRequest *req) { + /* + * When adding a new dtype handling here, + * please also update nvme_set_migration_blockers(). + */ return NVME_INVALID_FIELD | NVME_DNR; } =20 @@ -9208,6 +9215,199 @@ static void nvme_init_ctrl(NvmeCtrl *n, PCIDevice *= pci_dev) } } =20 +#define BLOCKER_FEATURES_MAX_LEN 256 + +static inline void nvme_add_blocker_feature(char *blocker_features, const = char *feature) +{ + if (strlen(blocker_features) > 0) { + g_strlcat(blocker_features, ", ", BLOCKER_FEATURES_MAX_LEN); + } + g_strlcat(blocker_features, feature, BLOCKER_FEATURES_MAX_LEN); +} + +static bool nvme_set_migration_blockers(NvmeCtrl *n, PCIDevice *pci_dev, E= rror **errp) +{ + uint64_t unsupported_cap, cap =3D ldq_le_p(&n->bar.cap); + char blocker_features[BLOCKER_FEATURES_MAX_LEN] =3D ""; + bool adm_cmd_security_checked =3D false; + bool cmd_io_mgmt_checked =3D false; + bool cmd_zone_checked =3D false; + + /* + * Idea of this function is simple, we iterate over all Command Sets a= nd + * for each supported command we provide a special handling logic to + * determine if we should block migration or not. + * + * For instance, we have NVME_ADM_CMD_NS_ATTACHMENT and it is always + * available to the guest, but if there is only 1 namespace, then it is + * safe to allow migration, but if there are more, then we need to blo= ck + * migration because we don't handle this in migration code yet. + */ + for (int opcode =3D 0; opcode < sizeof(n->cse.acs) / sizeof(n->cse.acs= [0]); opcode++) { + /* Is command supported? */ + if (!n->cse.acs[opcode]) { + continue; + } + + switch (opcode) { + case NVME_ADM_CMD_DELETE_SQ: + case NVME_ADM_CMD_CREATE_SQ: + case NVME_ADM_CMD_GET_LOG_PAGE: + case NVME_ADM_CMD_DELETE_CQ: + case NVME_ADM_CMD_CREATE_CQ: + case NVME_ADM_CMD_IDENTIFY: + case NVME_ADM_CMD_ABORT: + case NVME_ADM_CMD_SET_FEATURES: + case NVME_ADM_CMD_GET_FEATURES: + case NVME_ADM_CMD_ASYNC_EV_REQ: + case NVME_ADM_CMD_DBBUF_CONFIG: + case NVME_ADM_CMD_FORMAT_NVM: + case NVME_ADM_CMD_DIRECTIVE_SEND: + case NVME_ADM_CMD_DIRECTIVE_RECV: + break; + case NVME_ADM_CMD_NS_ATTACHMENT: + int namespaces_num =3D 0; + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + namespaces_num++; + } + + if (namespaces_num > 1) { + nvme_add_blocker_feature(blocker_features, "Namespace Atta= chment"); + } + + break; + case NVME_ADM_CMD_VIRT_MNGMT: + if (n->params.sriov_max_vfs) { + nvme_add_blocker_feature(blocker_features, "SR-IOV"); + } + + break; + case NVME_ADM_CMD_SECURITY_SEND: + case NVME_ADM_CMD_SECURITY_RECV: + if (adm_cmd_security_checked) { + break; + } + + if (pci_dev->spdm_port) { + nvme_add_blocker_feature(blocker_features, "SPDM"); + } + + adm_cmd_security_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < sizeof(n->cse.iocs.nvm) / sizeof(n->cs= e.iocs.nvm[0]); opcode++) { + if (!n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_FLUSH: + case NVME_CMD_WRITE: + case NVME_CMD_READ: + case NVME_CMD_COMPARE: + case NVME_CMD_WRITE_ZEROES: + case NVME_CMD_DSM: + case NVME_CMD_VERIFY: + case NVME_CMD_COPY: + break; + case NVME_CMD_IO_MGMT_RECV: + case NVME_CMD_IO_MGMT_SEND: + if (cmd_io_mgmt_checked) { + break; + } + + /* check for NVME_IOMS_MO_RUH_UPDATE */ + if (n->subsys->params.fdp.enabled) { + nvme_add_blocker_feature(blocker_features, "FDP"); + } + + cmd_io_mgmt_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < sizeof(n->cse.iocs.zoned) / sizeof(n->= cse.iocs.zoned[0]); opcode++) { + /* + * If command isn't supported or we have the same command + * in n->cse.iocs.nvm, then we can skip it here. + */ + if (!n->cse.iocs.zoned[opcode] || n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_ZONE_APPEND: + case NVME_CMD_ZONE_MGMT_SEND: + case NVME_CMD_ZONE_MGMT_RECV: + if (cmd_zone_checked) { + break; + } + + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + if (ns->params.zoned) { + nvme_add_blocker_feature(blocker_features, "Zoned Name= space"); + break; + } + } + + cmd_zone_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + /* + * Try our best to explicitly detect all not supported caps, + * to let users know what features cause migration to be blocked, + * but in case we miss handling here, everything else will be + * covered by unsupported_cap check. + */ + if (NVME_CAP_CMBS(cap)) { + nvme_add_blocker_feature(blocker_features, "CMB"); + cap &=3D ~((uint64_t)CAP_CMBS_MASK << CAP_CMBS_SHIFT); + } + + if (NVME_CAP_PMRS(cap)) { + nvme_add_blocker_feature(blocker_features, "PMR"); + cap &=3D ~((uint64_t)CAP_PMRS_MASK << CAP_PMRS_SHIFT); + } + + unsupported_cap =3D cap & ~NVME_MIGRATION_SUPPORTED_CAP_BITS; + if (unsupported_cap) { + nvme_add_blocker_feature(blocker_features, "unknown capability"); + } + + assert(n->migration_blocker =3D=3D NULL); + if (strlen(blocker_features) > 0) { + error_setg(&n->migration_blocker, "Migration is not supported for = %s", blocker_features); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + + return true; +} + static int nvme_init_subsys(NvmeCtrl *n, Error **errp) { int cntlid; @@ -9313,6 +9513,10 @@ static void nvme_realize(PCIDevice *pci_dev, Error *= *errp) =20 n->subsys->namespaces[ns->params.nsid] =3D ns; } + + if (!nvme_set_migration_blockers(n, pci_dev, errp)) { + return; + } } =20 static void nvme_exit(PCIDevice *pci_dev) @@ -9365,6 +9569,8 @@ static void nvme_exit(PCIDevice *pci_dev) } =20 memory_region_del_subregion(&n->bar0, &n->iomem); + + migrate_del_blocker(&n->migration_blocker); } =20 static const Property nvme_props[] =3D { diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index d66f7dc82d5..457b6637249 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -666,6 +666,9 @@ typedef struct NvmeCtrl { =20 /* Socket mapping to SPDM over NVMe Security In/Out commands */ int spdm_socket; + + /* Migration-related stuff */ + Error *migration_blocker; } NvmeCtrl; =20 typedef enum NvmeResetType { diff --git a/include/block/nvme.h b/include/block/nvme.h index 9d7159ed7a7..a7f586fc801 100644 --- a/include/block/nvme.h +++ b/include/block/nvme.h @@ -141,6 +141,18 @@ enum NvmeCapMask { #define NVME_CAP_SET_CMBS(cap, val) \ ((cap) |=3D (uint64_t)((val) & CAP_CMBS_MASK) << CAP_CMBS_SHIFT) =20 +#define NVME_MIGRATION_SUPPORTED_CAP_BITS ( \ + ((uint64_t)CAP_MQES_MASK << CAP_MQES_SHIFT) \ + | ((uint64_t)CAP_CQR_MASK << CAP_CQR_SHIFT) \ + | ((uint64_t)CAP_AMS_MASK << CAP_AMS_SHIFT) \ + | ((uint64_t)CAP_TO_MASK << CAP_TO_SHIFT) \ + | ((uint64_t)CAP_DSTRD_MASK << CAP_DSTRD_SHIFT) \ + | ((uint64_t)CAP_NSSRS_MASK << CAP_NSSRS_SHIFT) \ + | ((uint64_t)CAP_CSS_MASK << CAP_CSS_SHIFT) \ + | ((uint64_t)CAP_MPSMIN_MASK << CAP_MPSMIN_SHIFT) \ + | ((uint64_t)CAP_MPSMAX_MASK << CAP_MPSMAX_SHIFT) \ +) + enum NvmeCapCss { NVME_CAP_CSS_NCSS =3D 1 << 0, NVME_CAP_CSS_IOCSS =3D 1 << 6, --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603807; cv=none; d=zohomail.com; s=zohoarc; b=YXApQdkx6FsiFrNkSYOLbEuRvNhUGLZelznzTsRM6Fv/83yP9vKe/z5fVg2RjSSzVq9Hqi6YOX1jXBuc+1JD4bnHNpqHZLKKnN7OCdUAjdXgKyycpL+nxx7FDm+/GWc1ZVrPT56syRx2X7PB1QVCTjg2iFp9zuTQrkwcA++V/Og= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603807; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BC7q9hzRbwtGtEYWubxXb/HFj+bPnVoN51st/id0Ug8=; b=jZLQSFtgo7FLaola9JJ5mW7EcNz1REIfIB1LcnB4wQFWwEasBd05QWSilUKmh8aTQZ7jtqZ+roR9obGlCzirN59Wxg01NreCJ4cOLrFQFqMdV32eFa512v53myK3yxlgZq835T8yyzNLupLlFc9FSmRGOWWfPLXkJ3cA+16w4Ss= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603807687945.5023139225804; Sun, 19 Apr 2026 06:03:27 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERn8-00012E-BJ; Sun, 19 Apr 2026 09:02:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERmh-0000tt-IO for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:01 -0400 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERmZ-0008Is-Lu for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:01:58 -0400 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-488ff90d6c7so12686945e9.2 for ; Sun, 19 Apr 2026 06:01:47 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603706; x=1777208506; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BC7q9hzRbwtGtEYWubxXb/HFj+bPnVoN51st/id0Ug8=; b=KOInC/WrTA2FAWYIZ7+8yLDBgHC/q5rybwn/1vwRiKFtr0l8BxY6hCn/1CSfE1RUp0 +yZ8lYsVQIuZPAJfcI+MQRRJj2/UgPKv13NrLlUNC5gqxGuVznOYKZcmWp4fard01KFS oTfawGO8MsPDROQ3JTbW7rHaNvCe08zo3RpX8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603706; x=1777208506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BC7q9hzRbwtGtEYWubxXb/HFj+bPnVoN51st/id0Ug8=; b=E5GKM223HfQvtbGMMVqtw39t5Kej1OOvbw0kDS2WGOYLlFOrVH7oL8jxlu/0XpCaMs XI+aNsX7It2xIDyIW25BKrbnFeoD3Tj+XWzeSfzhWK3oSG2OR4dwHLuzqJEpzh8RCcyw HxQszkC5g+C5mqC421fceZTfkpHULr7hOct2xAYhvxSM5hrezR+d/dHBjVusa+/FcdHq 33RjIKYbyV9hEGnLQZyXBCBVg183bkz5TLDSSxwVJvVjm4wIeHoRDRUqvxEG8whG4BJv iWlKook+sO2uXrhfpHWkOEqfa2i4uIz/kS3clSVfb/hG9b+93X1JSSoVfMMVlFi5GN4A fxHw== X-Gm-Message-State: AOJu0YzXHJtytwa6DyzCc1nwUd6N5cy89Eysn13sN94j2ky7iK2rQXKO F+s1oFD27b7dcsrKbCKLMh9XSagpnnI50/XskW9mKrEaI8N53gk/2hwSoUD8EY73l03I8F3nCwQ QHMUIZgg= X-Gm-Gg: AeBDies3RE9zQnZzTJBW8QBCfAHxjH4Yh8uJn4SShpFLujdQq/cdGzkpAUUGJcw4kAm oxXvcALOsAqzMaMiQLOJNgBR+COqjSA8lNml75OqBdEqp2ae7x86Uhr++fQkRKm8r0mfjadvahS pX5iuNT+Td6wCO872t6AHdTB+d0nAlEgKtrwNYmTCQs/e0GSzzPYUiq5xUyO1tuwBXJqeHOBbSi HK3CeArUPHsHHqlGAHr5caXSrYqBeYpLtu82FP7v2swMdATR89IECRtmjcbbbqQcNdpFJw+vaVt oS5ua9WlgG7Wpxp9BK2VUgoQrFiXKIkknt35dWG8ihRUvEPpMBNQFgkVTb8ZIAhw4GkktHRXWbB 3Esh3kdD8yHv+hvRmZi0qRZDBm5GtfYwsbtg27dnb3qhrD4HUYgcxk+Rg/VR5mgvvaeU5QZO2nT 2ycYNQ8wrTv1uJDoTOzub6bt0tHj1omUQsYsEWLt7aEmT/3IzEc2GnF2QuZpND7rwyMNMoqBTx4 cfZVpJEJecIdTQp7ybT4yxjIv5mSRd7Iw== X-Received: by 2002:a05:600c:8587:b0:485:3f30:6250 with SMTP id 5b1f17b1804b1-488fb7856b9mr97480865e9.20.1776603705515; Sun, 19 Apr 2026 06:01:45 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 3/8] hw/nvme: split nvme_init_sq/nvme_init_cq into helpers Date: Sun, 19 Apr 2026 15:01:34 +0200 Message-ID: <20260419130139.15554-4-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603809693158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 59 +++++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 9f9c9bcbead..191398e700f 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -4854,18 +4854,14 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t sqid, uint16_t cqid, uint16_t size) +static void __nvme_init_sq(NvmeSQueue *sq) { + NvmeCtrl *n =3D sq->ctrl; + uint16_t sqid =3D sq->sqid; + uint16_t cqid =3D sq->cqid; int i; NvmeCQueue *cq; =20 - sq->ctrl =3D n; - sq->dma_addr =3D dma_addr; - sq->sqid =3D sqid; - sq->size =3D size; - sq->cqid =3D cqid; - sq->head =3D sq->tail =3D 0; sq->io_req =3D g_new0(NvmeRequest, sq->size); =20 QTAILQ_INIT(&sq->req_list); @@ -4895,6 +4891,18 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, n->sq[sqid] =3D sq; } =20 +static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t sqid, uint16_t cqid, uint16_t size) +{ + sq->ctrl =3D n; + sq->dma_addr =3D dma_addr; + sq->sqid =3D sqid; + sq->size =3D size; + sq->cqid =3D cqid; + sq->head =3D sq->tail =3D 0; + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5555,25 +5563,16 @@ static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t cqid, uint16_t vector, uint16_t size, - uint16_t irq_enabled) +static void __nvme_init_cq(NvmeCQueue *cq) { + NvmeCtrl *n =3D cq->ctrl; PCIDevice *pci =3D PCI_DEVICE(n); + uint16_t cqid =3D cq->cqid; =20 - if (msix_enabled(pci) && irq_enabled) { - msix_vector_use(pci, vector); + if (msix_enabled(pci) && cq->irq_enabled) { + msix_vector_use(pci, cq->vector); } =20 - cq->ctrl =3D n; - cq->cqid =3D cqid; - cq->size =3D size; - cq->dma_addr =3D dma_addr; - cq->phase =3D 1; - cq->irq_enabled =3D irq_enabled; - cq->vector =3D vector; - cq->head =3D cq->tail =3D 0; - QTAILQ_INIT(&cq->req_list); QTAILQ_INIT(&cq->sq_list); if (n->dbbuf_enabled) { cq->db_addr =3D n->dbbuf_dbs + (cqid << 3) + (1 << 2); @@ -5590,6 +5589,22 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, &DEVICE(cq->ctrl)->mem_reentrancy_guard); } =20 +static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t cqid, uint16_t vector, uint16_t size, + uint16_t irq_enabled) +{ + cq->ctrl =3D n; + cq->cqid =3D cqid; + cq->size =3D size; + cq->dma_addr =3D dma_addr; + cq->phase =3D 1; + cq->irq_enabled =3D irq_enabled; + cq->vector =3D vector; + cq->head =3D cq->tail =3D 0; + QTAILQ_INIT(&cq->req_list); + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603793; cv=none; d=zohomail.com; s=zohoarc; b=aNvhwM9Py+4sR/enqoZuHOgcBCPmEu7ckq1OYUoKQ6r69b42BGk09kVmuOe+0s+zqCuVkKdRYk+DROzDLykWuOYQmtD2/4dL1UBb3aUZUIrDaUqsd5AgF2R5U3JwPsyPGbWItTE++LXp8jvwVM7cVDJQTuS9Qrg3fMASZjQDtNY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603793; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BN9Hs6B25FCaLyK+WKS27HHbG03I3Ib65GOh38oVQqU=; b=GoE5P2QPIhg97MM3hkkvGJAD+IdjRyx1lctx6RuA0NWvicfAq6LIBt82pIFz8111CBJY0PhFTpzxd0zGqaHm/TPXN3ZmBqEwhYJnf5n/NHImW7JXNhNPezmHUXEcYpFbiR+y6Bpojyv8B+rZQx4tOH6d2kydb4PXwnOmWMRmgyA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603793477861.2983202818752; Sun, 19 Apr 2026 06:03:13 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERn9-00013O-Lg; Sun, 19 Apr 2026 09:02:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERmk-0000u7-Gy for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:05 -0400 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERme-0008J5-N0 for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:01:59 -0400 Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-43d77f60944so1657388f8f.3 for ; Sun, 19 Apr 2026 06:01:48 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603707; x=1777208507; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BN9Hs6B25FCaLyK+WKS27HHbG03I3Ib65GOh38oVQqU=; b=OusyDXJBoFiBTbKFb9EDNMBGTwnzREwHNbsCY7sNJYHoCMHFBgsi2XUCKUKgN1cqb1 eW5dXB8+nCFoqryoSX1UvX+mW63WjOxZvv26a06/ESX+w+N4+BXsyVM9EnQTNrIK4o9E ZGEbRUjVOJ1oy+V/UWUl/wOJpOgajRwPajUvs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603707; x=1777208507; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BN9Hs6B25FCaLyK+WKS27HHbG03I3Ib65GOh38oVQqU=; b=X1KSknt9XdRA0QS8h2X4zHLcf8DqPdJf7Q28r/0ZrjIYhU/YPHo+C0T2XB9FJrBcFE No1bjYNqSbmGftvACh1cpMQuCwdPLqE1kjPkrn38ACwl2aUVR6LYhrmGH8Vw9fNYLlS+ Y2j9PQ7S8mph0vdHvoX4lkxyKxkT6wNCKzI+qIC05Z2PEvJHqqPl5L1twho3vlnk75DU o4qK2NPFc1lArPwJhK+eN2VSos9LS7IDyzuUu0388dHgCqD4ucpWzX9wWwiGDDed5kMU P1LvKtb4KHLHsRgqRSowKkutFeGQr1ZoKbIcDraRgE+x0xhFvSaJ6ueKV5H4x0Ouo1VE H8Qg== X-Gm-Message-State: AOJu0YyHUEBEZQwaXeNv6+3Z8JSYc6Vgtn+CG/InGzm+/e/TUpmDn/GB SXcunQG260OuYr7oWsoZPEk8DESMTHMPhotDwwKLDBjV3Mdj1nX4I+vGgpuMT0qeFTj/eclPP5h CU8WBoaY= X-Gm-Gg: AeBDievPzUVkbHzHuAjD7dMi6ahROIYkH+7yh7k5TjsXOJ33DIwq1DLA02G3OEnW8d7 52Y1tZnwNEcZvNym9yh2G1zu1iXvGssp5GN7h4srMVRa5NrWyXWn+eskdEpX0ToB+DGr/3cVUKQ rBmTGzbNLkEsfocJ13zgLcnUr7QUxwrG3omcwHJ7RtEzLPtK25TNTllQ+FlkF+wiWSlklnCEHXp r7wdMADYuJX00hoMp3A9DuAzrAVYKQbkCzmShkGLZdU6zGfLyblPqwLEcr63840WwJCyaYd0nFL YGjv4Jp/5SDwhJ3ALTyAW+CXE1ORBfTIHWv/hsj3LnQrdkjZ5VYd8O+jtDWdBqQ7gZcQBJ+FaiO hgB7fP4t0lh2YPzNXidBY5vNHyUdoUKJFwp9joBPsvEZJqCPyzTV3NASJhnrUDGqUrCPYdzCrq2 B3iDIAb8bVapOQS386fql8gu4MIzxlgZFKvkNHpch2W1zsPlPik0OA/s3eo7AnK3wegwsPjarJe OfwtyaX/G07rNVi1HtkLNmCG24x6xwJnJJNa/mFnZTs X-Received: by 2002:a05:6000:2511:b0:43d:76d8:5798 with SMTP id ffacd0b85a97d-43fe3e0db09mr14977430f8f.27.1776603706568; Sun, 19 Apr 2026 06:01:46 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 4/8] hw/nvme: set CQE.sq_id earlier in nvme_process_sq Date: Sun, 19 Apr 2026 15:01:35 +0200 Message-ID: <20260419130139.15554-5-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::435; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x435.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603795591158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of filling req->cqe.sq_id in nvme_post_cqes, let's set it earlier in nvme_process_sq. This shouldn't cause any issues, because req->cqe.sq_id never changes during lifetime of req. This will help us for migration support. Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 191398e700f..d9bf32bff2c 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1520,7 +1520,6 @@ static void nvme_post_cqes(void *opaque) =20 sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); - req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.sq_head =3D cpu_to_le16(sq->head); addr =3D cq->dma_addr + (cq->tail << NVME_CQES); ret =3D pci_dma_write(PCI_DEVICE(n), addr, (void *)&req->cqe, @@ -7848,6 +7847,7 @@ static void nvme_process_sq(void *opaque) QTAILQ_REMOVE(&sq->req_list, req, entry); QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); nvme_req_clear(req); + req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.cid =3D cmd.cid; memcpy(&req->cmd, &cmd, sizeof(NvmeCmd)); =20 --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603835; cv=none; d=zohomail.com; s=zohoarc; b=Osa6kMHZZeKxAJpE1LOjng9pRcuJ71oRCOAiN7qaiAwhHEYfU3Tj8PKHaqcZOiH61URrzDM/zEYhiRTj4kYzgf85SulDARJqhdvvCJVa6YoyGsAHQ6Ui9NN5L6nPr2ZVZ75UkWon/JrChfTVE7r4EvWJ5mo2xapiN7w2nIIP/jY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603835; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=HQw+kuVjtbO6TkwAkF7eczBsBb+4xiQrsikCv0Jap6c=; b=I/Y4D+YZJvDAR/G6MfkVt87Wrj0Ip+XUkfDxSn3r0OOW3Mqk4ci/mT6FN8e/VHqBrXYGw/FKSWk1MWS2YtL9gLZ66H9Ccm8hScmFz0yMo8jQlrc5dbeCMXKYOQgt2bk74Lvj3sJw/PrGB8oX7XhRXK08BscMADrj6cKfR2wMS7k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603835464259.71318145773546; Sun, 19 Apr 2026 06:03:55 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERnD-00013w-MW; Sun, 19 Apr 2026 09:02:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERmk-0000u8-He for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:05 -0400 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERmc-0008Jr-Cn for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:01:59 -0400 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-43d64313c39so1680202f8f.3 for ; Sun, 19 Apr 2026 06:01:48 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603708; x=1777208508; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HQw+kuVjtbO6TkwAkF7eczBsBb+4xiQrsikCv0Jap6c=; b=idPypAc9pa6wpOoAO+u1Ftp5iX0eqYyNLBzIzCCDYsMqIIwz7891i116pX+4bPmZPs ynRbyMsKUorTt8En7GpI8m7tuCn6lXBHdhK9ZPZdeR6omr2eB2Rbq7NFYoMIgV8tw9CE 1BBC7NWGVWf838tAPJzd7V0pddXz+1Z7aNjXo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603708; x=1777208508; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HQw+kuVjtbO6TkwAkF7eczBsBb+4xiQrsikCv0Jap6c=; b=fp/CJ0PSquclpZ2oEsYqoQhCqEhlWoTn+g+0jMlyIVsddVzBS5N59hU5X3ed6vk0OB ai/i8JjKcrSZ6JJundeUgteg/nI7NrK5HgIr7WEnfowIkVmCkJvGT9Xh1Vch/mbtlcB4 2grMhGSsED11LHHSwQZlVnFEOma/OQEXpESfUNDiSw/hd3DEUJ0HbGfuuRNIL0M5Rduv U3OL/FO25R6QpXakwxs/0LPLNB16BjGbTJkrTM1bxFC64HnLrtwMhglvWy8IYhFzThqh T4SUfLm5ivpB5P0QszZvqwCgg7zrVdg6BFubq79fXWIhXMIPB+4ffRMGDFOZmh0AJ1dl soQQ== X-Gm-Message-State: AOJu0YxWo72yA7479kVgQ2EyaosciAsqSdJiPnqN8fixsBIR2CHtOe52 MX4Izyj+cFuw6J8UpIsQv4u4GN6VJjrjcPJ1LPuwuawK6EYvy7O83wYvLrqu2CFtBXP10aMT4bl c0P7YbU0= X-Gm-Gg: AeBDiev8FVd6fNeDLtnRuT8JrFHMYdLm9shE7RrxD3L/Coih6Mv9Wy6gzrpgnnCVJWv 7EC5kRAWDXEDV77GyI+kRmN/nK/PcauQ/f4/DmGyzyhq7rPMzvBjdWa7V07nVUAoQmtv46Pl4A8 1zokkYWRUFVqgCZeGbcEPd3VXrVUZMlmhSCxY/iFbCeXDBTR5J1YMt6iXUMAKkWsjOe6HQyHJaP NOasl9c7udezMZySPiihsiPEJsz63Q2g+OazsZMjqeaYVmIF/rnCZgKTqz6kguXCBM8klrWznje n2xiuhERWKZMbnw+m4gWlVigtn2d6kFPMXVPiahM18oOaeD5aEoGYDmyP7DYSr/wAMxlcImLMmD 1Vemy0tZ0C0VuplEedVL+1RPwfp3Slz8w1eNNNh4FSiKeUVVK5igxdNSbZ55UrrCW8ALHELntQc GoPZmb3i4SSVtXEsbL7ISv9SUnOLTi3yweddldEGeWlBv629ci8cY0J0Yy91R3zDhJBMHWgNr0w AiaSTTv2EMc1pFhy46hCSCvjg+y6LT95Q== X-Received: by 2002:a05:6000:25c4:b0:43b:8f56:1e1d with SMTP id ffacd0b85a97d-43fe3dc10demr14095668f8f.13.1776603707596; Sun, 19 Apr 2026 06:01:47 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 5/8] hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completion Date: Sun, 19 Apr 2026 15:01:36 +0200 Message-ID: <20260419130139.15554-6-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::429; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603836359154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of unmapping req->sg in nvme_post_cqes(), we can do it earlier in nvme_enqueue_req_completion(). When req completion is enqueued we don't need to access req->sg anymore. We only care about req->sq, req->cqe and req->status. Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index d9bf32bff2c..1ff91493b60 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1534,7 +1534,6 @@ static void nvme_post_cqes(void *opaque) QTAILQ_REMOVE(&cq->req_list, req, entry); =20 nvme_inc_cq_tail(cq); - nvme_sg_unmap(&req->sg); =20 if (QTAILQ_EMPTY(&sq->req_list) && !nvme_sq_empty(sq)) { qemu_bh_schedule(sq->bh); @@ -1564,6 +1563,8 @@ static void nvme_enqueue_req_completion(NvmeCQueue *c= q, NvmeRequest *req) req->status, req->cmd.opcode); } =20 + nvme_sg_unmap(&req->sg); + QTAILQ_REMOVE(&req->sq->out_req_list, req, entry); QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); =20 --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603834; cv=none; d=zohomail.com; s=zohoarc; b=kEuYwP24J9GwfPzmZZKV6NpzJGOxu37e7uQKlKVLfX+p14h6fGhTxWjmegCiz6yvtKMG1qXCLocmZQRF0mjbPJXi07ib/T2uKhUFjPVZh8dKCYdDWyJcXPhTeFgDcXvXsQZofpx6wVZdo5YkX+36+QLYzePuC/l98zrp9NIQnWE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603834; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=rk0IgYjAfH9Pdowv6rsdSE3Aypa1t/enhNxNQIimAQc=; b=lQ/UgHF74mJ9Vi9mYndwU9UFFkLAztW48gduD2Mn+v4e30wL/4SlHb8Rg/Zey2OvId1mxG7FQminWbwFYG0kfH8eTzI1AasBksoMmpidba7mkeRKAef8JuIOkO4xgbgokh8QSDa5dOA8lf6fbb06tIUhj/IRe5BJB0vx/zvw16M= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603834204328.24035132577353; Sun, 19 Apr 2026 06:03:54 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERnE-00014v-P3; Sun, 19 Apr 2026 09:02:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERmm-0000uJ-Gd for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:05 -0400 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERme-0008K1-N7 for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:04 -0400 Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-43d7605ec91so1874561f8f.3 for ; Sun, 19 Apr 2026 06:01:50 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603709; x=1777208509; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rk0IgYjAfH9Pdowv6rsdSE3Aypa1t/enhNxNQIimAQc=; b=DdiLzh1xiVSEVAsjuW6bKo7yW58hBzVBAyLCP6yJDcZ0mVv6pdyOlw+v6b+sdiOEGo OlsgBDGTDRzKhh+xJO3enIbOWwh1oh1Z/mNQ8USdDMcKi86yXrBfVTh9r758CvafmP3m X6ePF+N8lphtzgAg65NA/RIHpB21SYZ1rPfio= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603709; x=1777208509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=rk0IgYjAfH9Pdowv6rsdSE3Aypa1t/enhNxNQIimAQc=; b=QitwEh6rPVZwrHoI4fzomLbkKZ5G+1YjWJ7TEC7lYuNN9EKIKsk7HsrmwcnDwaMwS+ cIYuY+lXJEgqBoV8DaCW9VBF+WVshJyWTdATDMA2Q2fNjE7SfHq9CRlcpQlCFey0HBXd x667BpnByKVqrraEeMAdLv1S7R+EksAGx3OnwVQB/z8Q5xxOHHKRGFkc4iEF4lKXqjj0 v/BgtNTTjOJEa6BsELYHjcnRcKmsdJdWsJ1sZSUUNqIsm7K9WlVpgMErYvbz6ceciWsy DPiJjPSTRsjhXr9wjhzPlyvQ8jIq/Ne5EriDvylmRz2EebvZr2R2bkt05BrYZRVkFPlU sadQ== X-Gm-Message-State: AOJu0YyXW/hNWAD6+jo8nQZEVuui/F889M1hae+Oitn8MyAPT4MkfJH0 XRJlCktyojFmJCE8x7oNE0TMyDD7efdYrWRb2dR+4/ylu8JiHAJpVwdDGh/PYWREtqGHScyB3id lCMxQvBE= X-Gm-Gg: AeBDietlo01IGXki8amoSgSal6fdbZBBjJdwIKcjuPIV32ddmIH3HMUG7IK2qVk6NpN dQGK4S0WGJVvLuw3xm6kXJOykcyMm6BL3F+o+4bIJK7YzxcAWO+0ydacaK9irLG6N7/maE8z4IY 1OCwofHKLMs/4G2+6UPbokuxNopCb0kmnvGP00ouIlDTyUTZTvvkKHpueWVqkFM8kdrSXbb/nyj i7A2lxa+z9XmdlrHOX7CHjSelzvsaev6WUPQk4be+LrO6eUDdBU+fAFD7Kgx5S3XUbT6FqJGP+y cEMbCFO5VP2Plm72zh98NX2QEkm9pDSHwyf619zHNY0JJgdDuM9KFp66SAN1N2mLr8v3rm5cazG MyhyxtI4t6iBrDWwu9bymip2bXwoasYEu1c7rSbVo7k0oSGLICU818Kc0JxuxjD3nstkyuEatgM LrWvvbFoTSJbw2t7GFC+FsRs5sJa/ktTAs2UVHSpUgDTrEQwjs60nts7ZQmQRXHjVDv3dE7MUkf zt9tPc866TnkvK5g/9LLMV3/Upgj8RefA== X-Received: by 2002:a05:6000:230b:b0:43d:773d:7908 with SMTP id ffacd0b85a97d-43fe3e13d14mr15424776f8f.32.1776603708893; Sun, 19 Apr 2026 06:01:48 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 6/8] hw/nvme: add basic live migration support Date: Sun, 19 Apr 2026 15:01:37 +0200 Message-ID: <20260419130139.15554-7-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::435; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x435.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603835753158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn It has some limitations: - only one NVMe namespace is supported - SMART counters are not preserved - CMB is not supported - PMR is not supported - SPDM is not supported - SR-IOV is not supported Signed-off-by: Alexander Mikhalitsyn v2: - AERs are now fully supported v6: - handle full CQ case --- hw/nvme/ctrl.c | 722 ++++++++++++++++++++++++++++++++++++++++++- hw/nvme/ns.c | 160 ++++++++++ hw/nvme/nvme.h | 9 + hw/nvme/trace-events | 10 + 4 files changed, 892 insertions(+), 9 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 1ff91493b60..5157c7fd5a4 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -208,6 +208,7 @@ #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" #include "migration/blocker.h" +#include "migration/qemu-file-types.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -1518,6 +1519,18 @@ static void nvme_post_cqes(void *opaque) break; } =20 + /* + * Here we take the following fields from NvmeRequest structure + * and write cqe to the guest RAM based on them: + * - req->sq + * - req->status + * - req->cqe + * + * If you change this code and more fields from NvmeRequest are + * used, please make sure that you have handled this in: + * nvme_vmstate_request and nvme_ctrl_pre_save(). + */ + sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); req->cqe.sq_head =3D cpu_to_le16(sq->head); @@ -4903,6 +4916,25 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_sq(sq); } =20 +static void nvme_restore_sq(NvmeSQueue *sq_from) +{ + NvmeCtrl *n =3D sq_from->ctrl; + NvmeSQueue *sq =3D sq_from; + + if (sq_from->sqid =3D=3D 0) { + sq =3D &n->admin_sq; + sq->ctrl =3D n; + sq->dma_addr =3D sq_from->dma_addr; + sq->sqid =3D sq_from->sqid; + sq->size =3D sq_from->size; + sq->cqid =3D sq_from->cqid; + sq->head =3D sq_from->head; + sq->tail =3D sq_from->tail; + } + + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5605,6 +5637,39 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_cq(cq); } =20 +static void copy_cq_req_list(NvmeCQueue *cq_to, NvmeCQueue *cq_from) +{ + NvmeRequest *req, *next; + + QTAILQ_FOREACH_SAFE(req, &cq_from->req_list, entry, next) { + QTAILQ_REMOVE(&cq_from->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq_to->req_list, req, entry); + } +} + +static void nvme_restore_cq(NvmeCQueue *cq_from) +{ + NvmeCtrl *n =3D cq_from->ctrl; + NvmeCQueue *cq =3D cq_from; + + if (cq_from->cqid =3D=3D 0) { + cq =3D &n->admin_cq; + cq->ctrl =3D n; + cq->cqid =3D cq_from->cqid; + cq->size =3D cq_from->size; + cq->dma_addr =3D cq_from->dma_addr; + cq->phase =3D cq_from->phase; + cq->irq_enabled =3D cq_from->irq_enabled; + cq->vector =3D cq_from->vector; + cq->head =3D cq_from->head; + cq->tail =3D cq_from->tail; + QTAILQ_INIT(&cq->req_list); + copy_cq_req_list(cq, cq_from); + } + + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; @@ -7293,7 +7358,7 @@ static uint16_t nvme_dbbuf_config(NvmeCtrl *n, const = NvmeRequest *req) n->dbbuf_eis =3D eis_addr; n->dbbuf_enabled =3D true; =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { NvmeSQueue *sq =3D n->sq[i]; NvmeCQueue *cq =3D n->cq[i]; =20 @@ -7737,7 +7802,7 @@ static int nvme_atomic_write_check(NvmeCtrl *n, NvmeC= md *cmd, /* * Walk the queues to see if there are any atomic conflicts. */ - for (i =3D 1; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 1; i < n->num_queues; i++) { NvmeSQueue *sq; NvmeRequest *req; NvmeRwCmd *req_rw; @@ -7807,6 +7872,12 @@ static void nvme_process_sq(void *opaque) NvmeCmd cmd; NvmeRequest *req; =20 + /* + * We don't want to have a race with nvme_ctrl_pre_save(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + if (n->dbbuf_enabled) { nvme_update_sq_tail(sq); } @@ -7924,12 +7995,12 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) nvme_ns_drain(ns); } =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->sq[i] !=3D NULL) { nvme_free_sq(n->sq[i], n); } } - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->cq[i] !=3D NULL) { nvme_free_cq(n->cq[i], n); } @@ -8599,6 +8670,8 @@ static bool nvme_check_params(NvmeCtrl *n, Error **er= rp) params->max_ioqpairs =3D params->num_queues - 1; } =20 + n->num_queues =3D params->max_ioqpairs + 1; + if (n->namespace.blkconf.blk && n->subsys) { error_setg(errp, "subsystem support is unavailable with legacy " "namespace ('drive' property)"); @@ -8753,8 +8826,8 @@ static void nvme_init_state(NvmeCtrl *n) n->conf_msix_qsize =3D n->params.msix_qsize; } =20 - n->sq =3D g_new0(NvmeSQueue *, n->params.max_ioqpairs + 1); - n->cq =3D g_new0(NvmeCQueue *, n->params.max_ioqpairs + 1); + n->sq =3D g_new0(NvmeSQueue *, n->num_queues); + n->cq =3D g_new0(NvmeCQueue *, n->num_queues); n->temperature =3D NVME_TEMPERATURE; n->features.temp_thresh_hi =3D NVME_TEMPERATURE_WARNING; n->starttime_ms =3D qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); @@ -8989,7 +9062,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) } =20 if (n->params.msix_exclusive_bar && !pci_is_vf(pci_dev)) { - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, 0, NULL, N= ULL); + bar_size =3D nvme_mbar_size(n->num_queues, 0, NULL, NULL); memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n, "nv= me", bar_size); pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | @@ -9001,7 +9074,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) /* add one to max_ioqpairs to account for the admin queue pair */ if (!pci_is_vf(pci_dev)) { nr_vectors =3D n->params.msix_qsize; - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, + bar_size =3D nvme_mbar_size(n->num_queues, nr_vectors, &msix_table_offset, &msix_pba_offset); } else { @@ -9724,9 +9797,640 @@ static uint32_t nvme_pci_read_config(PCIDevice *dev= , uint32_t address, int len) return pci_default_read_config(dev, address, len); } =20 +static const VMStateDescription nvme_vmstate_cqe =3D { + .name =3D "nvme-cqe", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(result, NvmeCqe), + VMSTATE_UINT32(dw1, NvmeCqe), + VMSTATE_UINT16(sq_head, NvmeCqe), + VMSTATE_UINT16(sq_id, NvmeCqe), + VMSTATE_UINT16(cid, NvmeCqe), + VMSTATE_UINT16(status, NvmeCqe), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr_sgl =3D { + .name =3D "nvme-request-cmd-dptr-sgl", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(addr, NvmeSglDescriptor), + VMSTATE_UINT32(len, NvmeSglDescriptor), + VMSTATE_UINT8_ARRAY(rsvd, NvmeSglDescriptor, 3), + VMSTATE_UINT8(type, NvmeSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr =3D { + .name =3D "nvme-request-cmd-dptr", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(prp1, NvmeCmdDptr), + VMSTATE_UINT64(prp2, NvmeCmdDptr), + VMSTATE_STRUCT(sgl, NvmeCmdDptr, 0, nvme_vmstate_cmd_dptr_sgl, Nvm= eSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd =3D { + .name =3D "nvme-request-cmd", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(opcode, NvmeCmd), + VMSTATE_UINT8(flags, NvmeCmd), + VMSTATE_UINT16(cid, NvmeCmd), + VMSTATE_UINT32(nsid, NvmeCmd), + VMSTATE_UINT64(res1, NvmeCmd), + VMSTATE_UINT64(mptr, NvmeCmd), + VMSTATE_STRUCT(dptr, NvmeCmd, 0, nvme_vmstate_cmd_dptr, NvmeCmdDpt= r), + VMSTATE_UINT32(cdw10, NvmeCmd), + VMSTATE_UINT32(cdw11, NvmeCmd), + VMSTATE_UINT32(cdw12, NvmeCmd), + VMSTATE_UINT32(cdw13, NvmeCmd), + VMSTATE_UINT32(cdw14, NvmeCmd), + VMSTATE_UINT32(cdw15, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static bool nvme_req_pre_load(void *opaque, Error **errp) +{ + memset(opaque, 0x0, sizeof(NvmeRequest)); + return true; +} + +static const VMStateDescription nvme_vmstate_request =3D { + .name =3D "nvme-request", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_req_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(status, NvmeRequest), + VMSTATE_STRUCT(cqe, NvmeRequest, 0, nvme_vmstate_cqe, NvmeCqe), + VMSTATE_STRUCT(cmd, NvmeRequest, 0, nvme_vmstate_cmd, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_bar =3D { + .name =3D "nvme-bar", + .minimum_version_id =3D 1, + .version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(cap, NvmeBar), + VMSTATE_UINT32(vs, NvmeBar), + VMSTATE_UINT32(intms, NvmeBar), + VMSTATE_UINT32(intmc, NvmeBar), + VMSTATE_UINT32(cc, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd24, NvmeBar, 4), + VMSTATE_UINT32(csts, NvmeBar), + VMSTATE_UINT32(nssr, NvmeBar), + VMSTATE_UINT32(aqa, NvmeBar), + VMSTATE_UINT64(asq, NvmeBar), + VMSTATE_UINT64(acq, NvmeBar), + VMSTATE_UINT32(cmbloc, NvmeBar), + VMSTATE_UINT32(cmbsz, NvmeBar), + VMSTATE_UINT32(bpinfo, NvmeBar), + VMSTATE_UINT32(bprsel, NvmeBar), + VMSTATE_UINT64(bpmbl, NvmeBar), + VMSTATE_UINT64(cmbmsc, NvmeBar), + VMSTATE_UINT32(cmbsts, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd92, NvmeBar, 3492), + VMSTATE_UINT32(pmrcap, NvmeBar), + VMSTATE_UINT32(pmrctl, NvmeBar), + VMSTATE_UINT32(pmrsts, NvmeBar), + VMSTATE_UINT32(pmrebs, NvmeBar), + VMSTATE_UINT32(pmrswtp, NvmeBar), + VMSTATE_UINT32(pmrmscl, NvmeBar), + VMSTATE_UINT32(pmrmscu, NvmeBar), + VMSTATE_UINT8_ARRAY(css, NvmeBar, 484), + VMSTATE_END_OF_LIST() + }, +}; + +static bool nvme_cqueue_pre_load(void *opaque, Error **errp) +{ + NvmeCQueue *cq =3D opaque; + + QTAILQ_INIT(&cq->req_list); + return true; +} + +static const VMStateDescription nvme_vmstate_cqueue =3D { + .name =3D "nvme-cq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_cqueue_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(phase, NvmeCQueue), + VMSTATE_UINT16(cqid, NvmeCQueue), + VMSTATE_UINT16(irq_enabled, NvmeCQueue), + VMSTATE_UINT32(head, NvmeCQueue), + VMSTATE_UINT32(tail, NvmeCQueue), + VMSTATE_UINT32(vector, NvmeCQueue), + VMSTATE_UINT32(size, NvmeCQueue), + VMSTATE_UINT64(dma_addr, NvmeCQueue), + + VMSTATE_QTAILQ_V(req_list, NvmeCQueue, 1, nvme_vmstate_request, + NvmeRequest, entry), + + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_squeue =3D { + .name =3D "nvme-sq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(sqid, NvmeSQueue), + VMSTATE_UINT16(cqid, NvmeSQueue), + VMSTATE_UINT32(head, NvmeSQueue), + VMSTATE_UINT32(tail, NvmeSQueue), + VMSTATE_UINT32(size, NvmeSQueue), + VMSTATE_UINT64(dma_addr, NvmeSQueue), + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event_result =3D { + .name =3D "nvme-async-event-result", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(event_type, NvmeAerResult), + VMSTATE_UINT8(event_info, NvmeAerResult), + VMSTATE_UINT8(log_page, NvmeAerResult), + VMSTATE_UINT8(resv, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event =3D { + .name =3D "nvme-async-event", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_STRUCT(result, NvmeAsyncEvent, 0, nvme_vmstate_async_event= _result, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_hbs =3D { + .name =3D "nvme-hbs", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(acre, NvmeHostBehaviorSupport), + VMSTATE_UINT8(etdas, NvmeHostBehaviorSupport), + VMSTATE_UINT8(lbafee, NvmeHostBehaviorSupport), + VMSTATE_UINT8(rsvd3, NvmeHostBehaviorSupport), + VMSTATE_UINT16(cdfe, NvmeHostBehaviorSupport), + VMSTATE_UINT8_ARRAY(rsvd6, NvmeHostBehaviorSupport, 506), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_atomic =3D { + .name =3D "nvme-atomic", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(atomic_max_write_size, NvmeAtomic), + VMSTATE_UINT64(atomic_boundary, NvmeAtomic), + VMSTATE_UINT64(atomic_nabo, NvmeAtomic), + VMSTATE_BOOL(atomic_writes, NvmeAtomic), + VMSTATE_END_OF_LIST() + } +}; + +static bool pre_save_validate_aer_req(NvmeRequest *req, Error **errp) +{ + /* + * Can't use assert() here, because we don't want + * to just crash QEMU when user requests a migration. + */ + if (!(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ)) { + error_setg(errp, "req->cmd.opcode (%u) !=3D NVME_ADM_CMD_ASYNC_EV_= REQ", req->cmd.opcode); + return false; + } + + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->sq =3D=3D &req->sq->ctrl->admin_sq)) { + error_setg(errp, "req->sq !=3D &req->sq->ctrl->admin_sq"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool pre_save_validate_cq_req(NvmeRequest *req, Error **errp) +{ + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool nvme_ctrl_pre_save(void *opaque, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_pre_save_enter(n); + + /* + * We don't want to have a race with nvme_process_sq(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + + /* cancel all SQ processing BHs */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_cancel(sq->bh); + } + + /* drain all IO */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns; + + ns =3D nvme_ns(n, i); + if (!ns) { + continue; + } + + trace_pci_nvme_pre_save_ns_drain(n, i); + nvme_ns_drain(ns); + } + + /* + * Now, we should take care of AERs. + * + * 1. Save all queued events (n->aer_queue). + * This is done automatically, see nvme_vmstate VMStateDescription. + * Here we only need to print them for debugging purpose. + * 2. Go over outstanding AER requests (n->aer_reqs) and check they are + * all have expected opcode (NVME_ADM_CMD_ASYNC_EV_REQ) and other f= ields. + * + * We must be really careful here, because in case of further QEMU NVM= e changes, + * we may break migration without noticing it, or worse, introduce sil= ent + * data corruption during migration. + */ + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_pre_save_aer(event->result.event_type, event->r= esult.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeRequest *req =3D n->aer_reqs[i]; + + if (!pre_save_validate_aer_req(req, errp)) { + return false; + } + } + + /* make sure that all in-flight IO requests (except NVME_ADM_CMD_ASYNC= _EV_REQ) are processed */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req; + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + trace_pci_nvme_pre_save_sq_out_req_check(n, i, sq->head, sq->tail,= sq->size); + + QTAILQ_FOREACH(req, &sq->out_req_list, entry) { + assert(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ); + } + } + + /* wait when all IO requests completions are written to guest memory */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + qemu_bh_cancel(cq->bh); + /* this should empty cq->req_list unless CQ is full */ + nvme_post_cqes(cq); + + trace_pci_nvme_pre_save_cq_req_check(n, i, cq->head, cq->tail, cq-= >size); + + if (!QTAILQ_EMPTY(&cq->req_list)) { + NvmeRequest *req; + + assert(nvme_cq_full(cq)); + + QTAILQ_FOREACH(req, &cq->req_list, entry) { + trace_pci_nvme_pre_save_cq_unposted_cqe(n, i, nvme_cid(req= ), + nvme_nsid(req->ns), + le32_to_cpu(req->c= qe.result), + le32_to_cpu(req->c= qe.dw1), + req->status, req->= cmd.opcode); + if (!pre_save_validate_cq_req(req, errp)) { + return false; + } + } + } + } + + for (uint32_t nsid =3D 0; nsid <=3D NVME_MAX_NAMESPACES; nsid++) { + NvmeNamespace *ns =3D n->namespaces[nsid]; + + if (!ns) + continue; + + if (ns !=3D &n->namespace) { + error_setg(errp, "only one NVMe namespace is supported for mig= ration"); + return false; + } + } + + return true; +} + +static bool nvme_ctrl_post_load(void *opaque, int version_id, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_post_load_enter(n); + + /* restore CQs first */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + cq->ctrl =3D n; + nvme_restore_cq(cq); + trace_pci_nvme_post_load_restore_cq(n, i, cq->head, cq->tail, cq->= size); + + if (i =3D=3D 0) { + /* + * Admin CQ lives in n->admin_cq, we don't need + * memory allocated for it in get_ptrs_array_entry() anymore. + * + * nvme_restore_cq() also takes care of: + * n->cq[0] =3D &n->admin_cq; + * so n->cq[0] remains valid. + */ + g_free(cq); + } + } + + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + sq->ctrl =3D n; + nvme_restore_sq(sq); + trace_pci_nvme_post_load_restore_sq(n, i, sq->head, sq->tail, sq->= size); + + if (i =3D=3D 0) { + /* same as for CQ */ + g_free(sq); + } + } + + /* restore cq->req_list-s */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req_from, *next; + typeof_field(NvmeCQueue, req_list) req_list; + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq || QTAILQ_EMPTY(&cq->req_list)) + continue; + + /* + * We use nvme_vmstate_request VMStateDescription to save/restore + * NvmeRequest structures, but tricky thing here is that + * memory for each cq->req_list item is allocated separately + * during restore. It doesn't work for us. We need to take + * an existing NvmeRequest structure from SQ's req_list pool + * and fill it with data from the newly allocated one (req_from). + * Then, we can safely release allocated memory for it. + */ + + /* make a copy of cq->req_list (QTAILQ head) and clean cq->req_lis= t */ + QTAILQ_INIT(&req_list); + QTAILQ_FOREACH_SAFE(req_from, &cq->req_list, entry, next) { + QTAILQ_REMOVE(&cq->req_list, req_from, entry); + QTAILQ_INSERT_TAIL(&req_list, req_from, entry); + } + QTAILQ_INIT(&cq->req_list); + + QTAILQ_FOREACH_SAFE(req_from, &req_list, entry, next) { + uint16_t sqid =3D le16_to_cpu(req_from->cqe.sq_id); + NvmeRequest *req; + NvmeSQueue *sq; + + assert(!nvme_check_sqid(n, sqid)); + sq =3D n->sq[sqid]; + + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + QTAILQ_REMOVE(&req_list, req_from, entry); + g_free(req_from); + } + + qemu_bh_schedule(cq->bh); + } + + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_post_load_aer(event->result.event_type, event->= result.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeSQueue *sq =3D &n->admin_sq; + NvmeRequest *req_from =3D n->aer_reqs[i]; + NvmeRequest *req; + + /* Idea here is the same as for "restore cq->req_list-s" step */ + + /* take an NvmeRequest struct from SQ */ + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + n->aer_reqs[i] =3D req; + g_free(req_from); + } + + /* + * We need to attach namespaces (currently, only one namespace is + * supported for migration). + * This logic comes from nvme_start_ctrl(). + */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + + if (!ns || (!ns->params.shared && ns->ctrl !=3D n)) { + continue; + } + + if (nvme_csi_supported(n, ns->csi) && !ns->params.detached) { + if (!ns->attached || ns->params.shared) { + nvme_attach_ns(n, ns); + } + } + } + + /* schedule SQ processing */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_schedule(sq->bh); + } + + return true; +} + static const VMStateDescription nvme_vmstate =3D { .name =3D "nvme", - .unmigratable =3D 1, + .minimum_version_id =3D 1, + .version_id =3D 1, + .pre_save_errp =3D nvme_ctrl_pre_save, + .post_load_errp =3D nvme_ctrl_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_PCI_DEVICE(parent_obj, NvmeCtrl), + VMSTATE_MSIX(parent_obj, NvmeCtrl), + VMSTATE_STRUCT(bar, NvmeCtrl, 0, nvme_vmstate_bar, NvmeBar), + + VMSTATE_BOOL(qs_created, NvmeCtrl), + VMSTATE_UINT32(page_size, NvmeCtrl), + VMSTATE_UINT16(page_bits, NvmeCtrl), + VMSTATE_UINT16(max_prp_ents, NvmeCtrl), + VMSTATE_UINT32(max_q_ents, NvmeCtrl), + VMSTATE_UINT8(outstanding_aers, NvmeCtrl), + VMSTATE_UINT32(irq_status, NvmeCtrl), + VMSTATE_INT32(cq_pending, NvmeCtrl), + + VMSTATE_UINT64(host_timestamp, NvmeCtrl), + VMSTATE_UINT64(timestamp_set_qemu_clock_ms, NvmeCtrl), + VMSTATE_UINT64(starttime_ms, NvmeCtrl), + VMSTATE_UINT16(temperature, NvmeCtrl), + VMSTATE_UINT8(smart_critical_warning, NvmeCtrl), + + VMSTATE_UINT32(conf_msix_qsize, NvmeCtrl), + VMSTATE_UINT32(conf_ioqpairs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_dbs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_eis, NvmeCtrl), + VMSTATE_BOOL(dbbuf_enabled, NvmeCtrl), + + VMSTATE_UINT8(aer_mask, NvmeCtrl), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC( + aer_reqs, NvmeCtrl, outstanding_aers, 0, nvme_vmstate_request,= NvmeRequest), + VMSTATE_QTAILQ_V(aer_queue, NvmeCtrl, 1, nvme_vmstate_async_event, + NvmeAsyncEvent, entry), + VMSTATE_INT32(aer_queued, NvmeCtrl), + + VMSTATE_STRUCT(namespace, NvmeCtrl, 0, nvme_vmstate_ns, NvmeNamesp= ace), + + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + sq, NvmeCtrl, num_queues, 0, nvme_vmstate_squeue, NvmeSQueue), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + cq, NvmeCtrl, num_queues, 0, nvme_vmstate_cqueue, NvmeCQueue), + + VMSTATE_UINT16(features.temp_thresh_hi, NvmeCtrl), + VMSTATE_UINT16(features.temp_thresh_low, NvmeCtrl), + VMSTATE_UINT32(features.async_config, NvmeCtrl), + VMSTATE_STRUCT(features.hbs, NvmeCtrl, 0, nvme_vmstate_hbs, NvmeHo= stBehaviorSupport), + + VMSTATE_UINT32(dn, NvmeCtrl), + VMSTATE_STRUCT(atomic, NvmeCtrl, 0, nvme_vmstate_atomic, NvmeAtomi= c), + + VMSTATE_END_OF_LIST() + }, }; =20 static void nvme_class_init(ObjectClass *oc, const void *data) diff --git a/hw/nvme/ns.c b/hw/nvme/ns.c index 38f86a17268..dd374677078 100644 --- a/hw/nvme/ns.c +++ b/hw/nvme/ns.c @@ -20,6 +20,7 @@ #include "qemu/bitops.h" #include "system/system.h" #include "system/block-backend.h" +#include "migration/vmstate.h" =20 #include "nvme.h" #include "trace.h" @@ -886,6 +887,164 @@ static void nvme_ns_realize(DeviceState *dev, Error *= *errp) } } =20 +static const VMStateDescription nvme_vmstate_lbaf =3D { + .name =3D "nvme_lbaf", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(ms, NvmeLBAF), + VMSTATE_UINT8(ds, NvmeLBAF), + VMSTATE_UINT8(rp, NvmeLBAF), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns =3D { + .name =3D "nvme_id_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(nsze, NvmeIdNs), + VMSTATE_UINT64(ncap, NvmeIdNs), + VMSTATE_UINT64(nuse, NvmeIdNs), + VMSTATE_UINT8(nsfeat, NvmeIdNs), + VMSTATE_UINT8(nlbaf, NvmeIdNs), + VMSTATE_UINT8(flbas, NvmeIdNs), + VMSTATE_UINT8(mc, NvmeIdNs), + VMSTATE_UINT8(dpc, NvmeIdNs), + VMSTATE_UINT8(dps, NvmeIdNs), + VMSTATE_UINT8(nmic, NvmeIdNs), + VMSTATE_UINT8(rescap, NvmeIdNs), + VMSTATE_UINT8(fpi, NvmeIdNs), + VMSTATE_UINT8(dlfeat, NvmeIdNs), + VMSTATE_UINT16(nawun, NvmeIdNs), + VMSTATE_UINT16(nawupf, NvmeIdNs), + VMSTATE_UINT16(nacwu, NvmeIdNs), + VMSTATE_UINT16(nabsn, NvmeIdNs), + VMSTATE_UINT16(nabo, NvmeIdNs), + VMSTATE_UINT16(nabspf, NvmeIdNs), + VMSTATE_UINT16(noiob, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nvmcap, NvmeIdNs, 16), + VMSTATE_UINT16(npwg, NvmeIdNs), + VMSTATE_UINT16(npwa, NvmeIdNs), + VMSTATE_UINT16(npdg, NvmeIdNs), + VMSTATE_UINT16(npda, NvmeIdNs), + VMSTATE_UINT16(nows, NvmeIdNs), + VMSTATE_UINT16(mssrl, NvmeIdNs), + VMSTATE_UINT32(mcl, NvmeIdNs), + VMSTATE_UINT8(msrc, NvmeIdNs), + VMSTATE_UINT8_ARRAY(rsvd81, NvmeIdNs, 18), + VMSTATE_UINT8(nsattr, NvmeIdNs), + VMSTATE_UINT16(nvmsetid, NvmeIdNs), + VMSTATE_UINT16(endgid, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nguid, NvmeIdNs, 16), + VMSTATE_UINT64(eui64, NvmeIdNs), + VMSTATE_STRUCT_ARRAY(lbaf, NvmeIdNs, NVME_MAX_NLBAF, 1, + nvme_vmstate_lbaf, NvmeLBAF), + VMSTATE_UINT8_ARRAY(vs, NvmeIdNs, 3712), + + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_nvm =3D { + .name =3D "nvme_id_ns_nvm", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(lbstm, NvmeIdNsNvm), + VMSTATE_UINT8(pic, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd9, NvmeIdNsNvm, 3), + VMSTATE_UINT32_ARRAY(elbaf, NvmeIdNsNvm, NVME_MAX_NLBAF), + VMSTATE_UINT32(npdgl, NvmeIdNsNvm), + VMSTATE_UINT32(nprg, NvmeIdNsNvm), + VMSTATE_UINT32(npra, NvmeIdNsNvm), + VMSTATE_UINT32(nors, NvmeIdNsNvm), + VMSTATE_UINT32(npdal, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd288, NvmeIdNsNvm, 3808), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_ind =3D { + .name =3D "nvme_id_ns_ind", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(nsfeat, NvmeIdNsInd), + VMSTATE_UINT8(nmic, NvmeIdNsInd), + VMSTATE_UINT8(rescap, NvmeIdNsInd), + VMSTATE_UINT8(fpi, NvmeIdNsInd), + VMSTATE_UINT32(anagrpid, NvmeIdNsInd), + VMSTATE_UINT8(nsattr, NvmeIdNsInd), + VMSTATE_UINT8(rsvd9, NvmeIdNsInd), + VMSTATE_UINT16(nvmsetid, NvmeIdNsInd), + VMSTATE_UINT16(endgrpid, NvmeIdNsInd), + VMSTATE_UINT8(nstat, NvmeIdNsInd), + VMSTATE_UINT8_ARRAY(rsvd15, NvmeIdNsInd, 4081), + VMSTATE_END_OF_LIST() + } +}; + +typedef struct TmpNvmeNamespace { + NvmeNamespace *parent; + bool enable_write_cache; +} TmpNvmeNamespace; + +static bool nvme_ns_tmp_pre_save(void *opaque, Error **errp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + tns->enable_write_cache =3D blk_enable_write_cache(tns->parent->blkcon= f.blk); + + return true; +} + +static bool nvme_ns_tmp_post_load(void *opaque, int version_id, Error **er= rp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + blk_set_enable_write_cache(tns->parent->blkconf.blk, tns->enable_write= _cache); + + return true; +} + +static const VMStateDescription nvme_vmstate_ns_tmp =3D { + .name =3D "nvme_ns_tmp", + .pre_save_errp =3D nvme_ns_tmp_pre_save, + .post_load_errp =3D nvme_ns_tmp_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_BOOL(enable_write_cache, TmpNvmeNamespace), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_ns =3D { + .name =3D "nvme_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_WITH_TMP(NvmeNamespace, TmpNvmeNamespace, nvme_vmstate_ns_= tmp), + + VMSTATE_STRUCT(id_ns, NvmeNamespace, 0, nvme_vmstate_id_ns, NvmeId= Ns), + VMSTATE_STRUCT(id_ns_nvm, NvmeNamespace, 0, nvme_vmstate_id_ns_nvm= , NvmeIdNsNvm), + VMSTATE_STRUCT(id_ns_ind, NvmeNamespace, 0, nvme_vmstate_id_ns_ind= , NvmeIdNsInd), + VMSTATE_STRUCT(lbaf, NvmeNamespace, 0, nvme_vmstate_lbaf, NvmeLBAF= ), + VMSTATE_UINT32(nlbaf, NvmeNamespace), + VMSTATE_UINT8(csi, NvmeNamespace), + VMSTATE_UINT16(status, NvmeNamespace), + VMSTATE_UINT8(pif, NvmeNamespace), + + VMSTATE_UINT16(zns.zrwas, NvmeNamespace), + VMSTATE_UINT16(zns.zrwafg, NvmeNamespace), + VMSTATE_UINT32(zns.numzrwa, NvmeNamespace), + + VMSTATE_UINT32(features.err_rec, NvmeNamespace), + VMSTATE_STRUCT(atomic, NvmeNamespace, 0, nvme_vmstate_atomic, Nvme= Atomic), + VMSTATE_END_OF_LIST() + } +}; + static const Property nvme_ns_props[] =3D { DEFINE_BLOCK_PROPERTIES(NvmeNamespace, blkconf), DEFINE_PROP_BOOL("detached", NvmeNamespace, params.detached, false), @@ -937,6 +1096,7 @@ static void nvme_ns_class_init(ObjectClass *oc, const = void *data) dc->bus_type =3D TYPE_NVME_BUS; dc->realize =3D nvme_ns_realize; dc->unrealize =3D nvme_ns_unrealize; + dc->vmsd =3D &nvme_vmstate_ns; device_class_set_props(dc, nvme_ns_props); dc->desc =3D "Virtual NVMe namespace"; } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 457b6637249..2e7597cded3 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -444,6 +444,11 @@ typedef struct NvmeRequest { NvmeSg sg; bool atomic_write; QTAILQ_ENTRY(NvmeRequest)entry; + /* + * If you add a new field here, please make sure to update + * nvme_vmstate_request, pre_save_validate_aer_req() and + * pre_save_validate_cq_req(). + */ } NvmeRequest; =20 typedef struct NvmeBounceContext { @@ -638,6 +643,7 @@ typedef struct NvmeCtrl { =20 NvmeNamespace namespace; NvmeNamespace *namespaces[NVME_MAX_NAMESPACES + 1]; + uint32_t num_queues; NvmeSQueue **sq; NvmeCQueue **cq; NvmeSQueue admin_sq; @@ -749,4 +755,7 @@ void nvme_atomic_configure_max_write_size(bool dn, uint= 16_t awun, void nvme_ns_atomic_configure_boundary(bool dn, uint16_t nabsn, uint16_t nabspf, NvmeAtomic *atomic= ); =20 +extern const VMStateDescription nvme_vmstate_atomic; +extern const VMStateDescription nvme_vmstate_ns; + #endif /* HW_NVME_NVME_H */ diff --git a/hw/nvme/trace-events b/hw/nvme/trace-events index 6be0bfa1c1f..f97a6a11f36 100644 --- a/hw/nvme/trace-events +++ b/hw/nvme/trace-events @@ -7,6 +7,16 @@ pci_nvme_dbbuf_config(uint64_t dbs_addr, uint64_t eis_addr= ) "dbs_addr=3D0x%"PRIx64 pci_nvme_map_addr(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %"PRI= u64"" pci_nvme_map_addr_cmb(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %= "PRIu64"" pci_nvme_map_prp(uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t= prp2, int num_prps) "trans_len %"PRIu64" len %"PRIu32" prp1 0x%"PRIx64" pr= p2 0x%"PRIx64" num_prps %d" +pci_nvme_pre_save_enter(void *n) "n=3D%p" +pci_nvme_pre_save_ns_drain(void *n, int i) "n=3D%p i=3D%d" +pci_nvme_pre_save_sq_out_req_check(void *n, int i, uint32_t head, uint32_t= tail, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" = size=3D0x%"PRIx32"" +pci_nvme_pre_save_cq_req_check(void *n, int i, uint32_t head, uint32_t tai= l, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_pre_save_cq_unposted_cqe(void *n, int i, uint16_t cid, uint32_t n= sid, uint32_t dw0, uint32_t dw1, uint16_t status, uint8_t opc) "n=3D%p i=3D= %d cid %"PRIu16" nsid %"PRIu32" dw0 0x%"PRIx32" dw1 0x%"PRIx32" status 0x%"= PRIx16" opc 0x%"PRIx8"" +pci_nvme_pre_save_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type 0= x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" +pci_nvme_post_load_enter(void *n) "n=3D%p" +pci_nvme_post_load_restore_cq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_restore_sq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type = 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" pci_nvme_map_sgl(uint8_t typ, uint64_t len) "type 0x%"PRIx8" len %"PRIu64"" pci_nvme_io_cmd(uint16_t cid, uint32_t nsid, uint16_t sqid, uint8_t opcode= , const char *opname) "cid %"PRIu16" nsid 0x%"PRIx32" sqid %"PRIu16" opc 0x= %"PRIx8" opname '%s'" pci_nvme_admin_cmd(uint16_t cid, uint16_t sqid, uint8_t opcode, const char= *opname) "cid %"PRIu16" sqid %"PRIu16" opc 0x%"PRIx8" opname '%s'" --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603846; cv=none; d=zohomail.com; s=zohoarc; b=jdSCIwlaAAvk0BT5d8xyqMdvfhGMxEFDit1BxtQ9XmU93vzjPade27wKa7/WwlsqlMSk3DrzeBQwkHTpsUlxClt5d4pL3LySsXal8CEJ+rscbPcJoPhr97Lc40+Jp4O4eJ5+XekCL7m3zPG4ThpgNw8AQwOPQNbO+XaxmPCAM/E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603846; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=Iu2UmIopVWTkma58KK2yAyVD39lnIdSEpg6zKsuqYk+BZnR13GUQD3L8bN3WxnlB+ZO2bViF2niU3J0HD2w4I25lBn2VXyma449FJlJqzmbbT8oOTcP7/LOtR67lcugyviQkx+CU+hsLtKIDx+zRUl+gx0G4+KWwnGP49X5hcOs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603846236929.6359406279653; Sun, 19 Apr 2026 06:04:06 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERn7-00011m-ML; Sun, 19 Apr 2026 09:02:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERml-0000uF-IJ for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:05 -0400 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERme-0008K8-NI for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:03 -0400 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-4887ca8e529so15664995e9.0 for ; Sun, 19 Apr 2026 06:01:51 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603710; x=1777208510; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=YPN9TbsP303pdJpJiA0fJA/xhd75FAhT+fN/FV/EDepmIvcW/j26hlxq1OGg49Njk0 xNs5kH3iu4KSPrgfaO04ThZiKMwSancvTMVH/k7OV5mbmwQXDKcek/0G80WDj48YBKTa ZPtu+bPaVoAUQvu+gJyyN/Afr3lB/1CZ+iBrQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603710; x=1777208510; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=ZLM6HKvM2BLAjuYukMqRy8m3sYex7wefWMu6XmcQYwawn/NjiBQxapzqAmoK/mpV1I VJ4yNmilBMA8HkEfP2OcHSZFhu4QEuSWXRrw0H+mRhxgN05c2ua8fm96xWgOvLHTMl9r DVGA8fGdWnCd+dtoymG7BXR+HUOyGFreYBs7Uxq2p0lrkbrBYplVHIu3J77fNzariz68 nCX6NjrOI5zYMZvWF6X2f5KOXNIKBgWcf/G3cB8KZYintzwdECdd1o9bqgp8hTFts/ll 9Ns47TRXnJaFAUwizgQR/0G2j1VRO1MGxpMa9iYEmEDEoNuAoPRr+lzdVCvXN9ZMU+J8 nT0A== X-Gm-Message-State: AOJu0YzA+4wweA/olWZkYZPLBAOWzEuAHYsTtX8CseFoLmDsD//RnlKJ 41Gd7ne0xvSqrs5wklYnsumQQyDxraPU+t6rdYMy8uhReSrGufHFNJTn6/faHZQhMu2Z/689xC0 rTxPnj+o= X-Gm-Gg: AeBDiev8RpZB6UyGkJiDUin2WcPeGBdVPRJFWJAhuprK/73h7ywkoQfQfDOKZZ9y77S uCDapj/I1JatmSqD+uTpvv+fpoCVV0Z0t5/QJ8awuDmioa2FSlcADn2HxFX0TTwMKuMxlgmcMpk FkVoSB9/7PVv9liRUkQqlD/Q4FruWF9bE1VR9ko2Sv4HELSmtVxyeHBydeJWo872vHoK+Mt2qMW IvTprq/pSbyYjYaodhlHvD+JoK64o2UH3EAFJdhDDZUZH33UcmTVkh89X0/IEdkFV2w3nO9TeKa Zrq0T3NCyUBOGIc6wAHpq2ZydX1DF3tnL+IQvhwxN6Uy2q7tcuw7nhMSX+YnO9aDLK/ghQ9Vbpn FXCapYDCfs65HZvA0IUVLpzHnn4YBOtkFSt8l3OeOhX8J1FIFYbb2hYcXNgJ45QVyCIBvPHBdQu YcRyA6kYB2aPEReHZrVOcUO60+er7mKPrafFjPnFywiSUgQAel2aXe9mXeShs555ElMjA9E5mYd ZQ6Qtjrs12lRJULy2jim5KZpiZrLVk8kg== X-Received: by 2002:a05:600c:8a1a:20b0:485:3038:72c2 with SMTP id 5b1f17b1804b1-488fb8bcdf6mr98009815e9.17.1776603710387; Sun, 19 Apr 2026 06:01:50 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 7/8] tests/functional/x86_64: add migration test for NVMe device Date: Sun, 19 Apr 2026 15:01:38 +0200 Message-ID: <20260419130139.15554-8-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603848546154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce a very simple test to ensure that NVMe device migration works fine. Test plan is simple: 1. prepare VM with NVMe device 2. run workload that produces relatively heavy IO on the device 3. migrate VM 4. ensure that workload is alive and finishes without errors Test can be run as simple as: $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build In the future we can extend this approach, and introduce some fio-based tests. And probably, it makes sense to make this test to apply not only to NVMe device, but also virtio-{blk,scsi}, ide, sata and other migratable devices. Signed-off-by: Alexander Mikhalitsyn --- tests/functional/x86_64/meson.build | 1 + .../functional/x86_64/test_nvme_migration.py | 159 ++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100755 tests/functional/x86_64/test_nvme_migration.py diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/= meson.build index 1ed10ad6c29..fd77f19d726 100644 --- a/tests/functional/x86_64/meson.build +++ b/tests/functional/x86_64/meson.build @@ -37,6 +37,7 @@ tests_x86_64_system_thorough =3D [ 'linux_initrd', 'multiprocess', 'netdev_ethtool', + 'nvme_migration', 'replay', 'reverse_debug', 'tuxrun', diff --git a/tests/functional/x86_64/test_nvme_migration.py b/tests/functio= nal/x86_64/test_nvme_migration.py new file mode 100755 index 00000000000..3788a8e3473 --- /dev/null +++ b/tests/functional/x86_64/test_nvme_migration.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# x86_64 NVMe migration test + +from migration import MigrationTest +from qemu_test import QemuSystemTest, Asset +from qemu_test import wait_for_console_pattern +from qemu_test import exec_command, exec_command_and_wait_for_pattern + + +class X8664NVMeMigrationTest(MigrationTest): + ASSET_KERNEL =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/vmlinuz'), + 'd4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129') + + ASSET_INITRD =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/initrd.img'), + '277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b') + + ASSET_DISKIMAGE =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Cloud/x86_64/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2'), + 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0') + + DEFAULT_KERNEL_PARAMS =3D ('root=3D/dev/nvme0n1p1 console=3DttyS0 net.= ifnames=3D0 ' + 'rd.rescue quiet') + + def wait_for_console_pattern(self, success_message, vm): + wait_for_console_pattern( + self, + success_message, + failure_message=3D"Kernel panic - not syncing", + vm=3Dvm, + ) + + def exec_command_and_check(self, command, vm): + prompt =3D '# ' + exec_command_and_wait_for_pattern(self, + f"{command} && echo OK || echo FAI= L", + 'FAIL', vm=3Dvm) + # Note, that commands we send to the console are echo-ed back, so = if we have a word "FAIL" + # in the command itself, we should expect to see it once. + wait_for_console_pattern(self, 'OK', failure_message=3D"FAIL", vm= =3Dvm) + self.wait_for_console_pattern(prompt, vm) + + def configure_machine(self, vm): + kernel_path =3D self.ASSET_KERNEL.fetch() + initrd_path =3D self.ASSET_INITRD.fetch() + diskimage_path =3D self.ASSET_DISKIMAGE.fetch() + + vm.set_console() + vm.add_args("-cpu", "max") + vm.add_args("-m", "2G") + vm.add_args("-accel", "kvm") + + vm.add_args('-drive', + f'file=3D{diskimage_path},if=3Dnone,id=3Ddrv0,sna= pshot=3Don') + vm.add_args('-device', 'nvme,bus=3Dpcie.0,' + + 'drive=3Ddrv0,id=3Dnvme-disk0,serial=3Dnvmemigrat= etest,bootindex=3D1') + + vm.add_args( + "-kernel", + kernel_path, + "-initrd", + initrd_path, + "-append", + self.DEFAULT_KERNEL_PARAMS + ) + + def launch_source_vm(self, vm): + vm.launch() + + self.wait_for_console_pattern('Entering emergency mode.', vm) + prompt =3D '# ' + self.wait_for_console_pattern(prompt, vm) + + # Synchronize on NVMe driver creating the root device + exec_command_and_wait_for_pattern(self, + "while ! (dmesg -c | grep nvme0n1:) ; do sleep 1 ;= done", + "nvme0n1", vm=3Dvm) + self.wait_for_console_pattern(prompt, vm) + + # prepare system + exec_command_and_wait_for_pattern(self, 'mount /dev/nvme0n1p1 /sys= root', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'chroot /sysroot', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t proc proc /proc', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs /sys= ', + prompt, vm=3Dvm) + + # Run workload before migration to check if it continues to run pr= operly after migration + # + # Workload is simple: it continuously calculates checksums of all = files in /usr/bin + # to generate some I/O load on the NVMe disk and at the same time = it drops caches to + # make sure that we have some read I/O on the disk as well. + # If there are any issues with the migration of the NVMe device, w= e should see errors + # in dmesg and consequently in the workload log. + exec_command_and_wait_for_pattern(self, + "(while [ ! -f /tmp/test_nvme_migr= ation_workload.stop ]; do \ + rm -f /tmp/test_nvme_migration= _workload.iteration_finished; \ + echo 3 > /proc/sys/vm/drop_cac= hes; \ + find /usr/bin -type f -exec ck= sum {} \\;; \ + touch /tmp/test_nvme_migration= _workload.iteration_finished; \ + done) > /dev/null 2> /tmp/test_nvm= e_migration_workload.errors &", + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'echo $! > /tmp/test_nvme_= migration_workload.pid', + prompt, vm=3Dvm) + + # check if process is alive and running + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + def assert_dest_vm(self, vm): + prompt =3D '# ' + + # check if process is alive and running after migration, if not - = fail the test + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + # signal workload to stop + exec_command_and_wait_for_pattern(self, 'touch /tmp/test_nvme_migr= ation_workload.stop', + prompt, vm=3Dvm) + + # wait workload to finish, because we want to examine log to see i= f there are any errors + exec_command_and_wait_for_pattern(self, + "while [ ! -f /tmp/test_nvme_migra= tion_workload.iteration_finished ]; do sleep 1; done;", + prompt, vm=3Dvm) + + exec_command_and_wait_for_pattern(self, 'cat /tmp/test_nvme_migrat= ion_workload.errors', + prompt, vm=3Dvm) + + # fail the test if non-empty + self.exec_command_and_check("[ ! -s /tmp/test_nvme_migration_workl= oad.errors ]", vm) + + def test_migration_with_tcp_localhost(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_tcp_localhost() + + def test_migration_with_unix(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_unix() + + def test_migration_with_exec(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_exec() + + +if __name__ =3D=3D '__main__': + MigrationTest.main() --=20 2.47.3 From nobody Tue Apr 21 14:38:20 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1776603818; cv=none; d=zohomail.com; s=zohoarc; b=fywL4Y11+LoCgn5ZLfejcgNEsRe2j1LJGPZHcCIJKhqiO/s4b2djAlqle9xU55UQok93GacU8HO6qdx8dkocFsCNbippmlzjX371Zh+utHwPYIXfTRcQjySvr2emYOJnt+MKgTG8IZHAW5UAjq2KOOBGxpJCVM2KOmQkAtR81D8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1776603818; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Y091UvS4a88ptSlV8x9WKnLH+imaiJafi8HLwtDBHMs=; b=fV323h9tKrh4HK/WaIeEZznnvfyxCp6EqAX4gUsBybPAGot1gsAyBfV8P+RT+Wtxj11oaZwGz3Poi2lOywfKqgCMRSeklppaszzd4kpc9eeTNzNXtaTPgbRMquIVVKAqggE3OnMNaHI3PflzVJOJNx2JQuCyE5HnnxCI3DGxwjg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1776603818212903.3273125038115; Sun, 19 Apr 2026 06:03:38 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wERnH-00016n-2u; Sun, 19 Apr 2026 09:02:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wERmk-0000uD-Jl for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:05 -0400 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wERme-0008KL-Lq for qemu-devel@nongnu.org; Sun, 19 Apr 2026 09:02:01 -0400 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-488c2690057so20460645e9.0 for ; Sun, 19 Apr 2026 06:01:53 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00995e4e0d3496e07b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:995e:4e0d:3496:e07b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43fe4cc2cacsm20734304f8f.13.2026.04.19.06.01.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 06:01:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1776603711; x=1777208511; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y091UvS4a88ptSlV8x9WKnLH+imaiJafi8HLwtDBHMs=; b=O/CKtncC+CnAe2OsAd1ilrpVHsBEy4GXNsEeoMgp330tWoctjtw+Chd5JtxtHDts4A c7BysGfldoxL0o/P1fBXlaEB0DoMdniVvGadN1jyi9/7R45gaV9NFoQqYi/zh5j1l60s QZFm0QNWTujFKBWnWgAAGAaijxiU6MLpyUjbI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776603711; x=1777208511; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Y091UvS4a88ptSlV8x9WKnLH+imaiJafi8HLwtDBHMs=; b=L/OXiZFiQTxwCmUMmgZwXp0e/y7e9iOA7x9WULFueRIeWZuLdEgYqNvJDuVbhtLbfl q9HIsy9pP65GhAGfMWwJXo6Wu6C6sthu7shb86V28fIVpaBBoSeoGHR+w7UtqkAdMYax cSS0T7bOYTuP32zD7XoJTliL7ROD0JWc/NAn/X4Y+7rnw0DXOId+S8iqk9DKb5KXscQX Xrd+VQc+nT468Oxh4JvAMZXB16v+x9rqIr1Whjhc+yfG9ksoMH1CGJQ3WpHKt5ArSTrL f9Hq7mVsT76U/6TG7Ff3yfJEQdan0DopPq6aY4TWUlHeckTt4wT0s0ltj4jMy5LSkal9 LgsQ== X-Gm-Message-State: AOJu0YwlDiC/pE7Uffj+/FnzcpE9gSEjISkn31h1nhwl21ptkRkPVJR8 ir5b9+6FDYVORrBgd6B2NJGoV37Wg/7npaora9KIRmeiSOBrev8pwPd5KUHronFGd2RfIiJ+AfI iNIL1P9I= X-Gm-Gg: AeBDieudcakPK0u/v5dqY1maypaD7Qch6Ok2wtiWO7AG0l+hNSiZ7XyYMjd4WroguBa YS91wJf1oI5hWrqpXWw9wGelveHKtxXOLrtkPmXffv/XvqAo31yJo4LYEaahVdW8UsoC1l/fHQd Rz0c4yw/Tsq766c7kk/BceXtl6jZ0GjkOgaw9KW/3D5ryaTs2XPfjc979nD6f0rj1d23ie2MRk4 Y7n+83b02OlaAZRFJ1HdzVH191FqWaG8hh9QpmA7KxWeeh/nLR4HIGc315WeUlLwDsxdDQU3Bw/ 2thTQ77Bw/bz1VucwJebMRBS0pHSH261AC9P0FldS/T4vvhhoJCp5Iq6HN8nCxEWSG9EsGhaqsb 3xks2iADyw4ih2tK51nqA32WjJGB8rJEy1urPRDmhehp3uIGZmbDUsYdM2uOxxjLQmRLaOJNNhY iij1zzZ5dXi5gJW1XZxfLvUq4uk5WHUiNz7HfTFvuAFmbj2av4Der8xJSJqFSnipMc6ke7lSeh4 w5JKRNADdg0RcsFlAajCg/y8b9R95Ta9Q== X-Received: by 2002:a05:600c:888b:b0:488:c80c:c236 with SMTP id 5b1f17b1804b1-488fb73d525mr115373365e9.5.1776603711438; Sun, 19 Apr 2026 06:01:51 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Alexander Mikhalitsyn , Kevin Wolf , qemu-block@nongnu.org, Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Paolo Bonzini , Stefan Hajnoczi , Laurent Vivier , Jesper Devantier , Klaus Jensen , Fabiano Rosas , Zhao Liu , Keith Busch , Peter Xu , Hanna Reitz , Alexander Mikhalitsyn Subject: [PATCH v6 8/8] tests/qtest/nvme-test: add migration test with full CQ Date: Sun, 19 Apr 2026 15:01:39 +0200 Message-ID: <20260419130139.15554-9-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260419130139.15554-1-alexander@mihalicyn.com> References: <20260419130139.15554-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32c; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1776603819813158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn As suggested by Stefan [1], let's add a migration test to cover rare scenario when CQ is full of non-processed CQEs and migration happens. To run this test: $ meson test -C build 'qtest-x86_64/qos-test' Link: https://lore.kernel.org/qemu-devel/20260408183529.GB319710@fedora/ [1] Suggested-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- tests/qtest/nvme-test.c | 393 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 393 insertions(+) diff --git a/tests/qtest/nvme-test.c b/tests/qtest/nvme-test.c index 4aec1651e6e..1ba2fa6943f 100644 --- a/tests/qtest/nvme-test.c +++ b/tests/qtest/nvme-test.c @@ -8,9 +8,11 @@ */ =20 #include "qemu/osdep.h" +#include "qemu/bswap.h" #include "qemu/module.h" #include "qemu/units.h" #include "libqtest.h" +#include "libqtest-single.h" #include "libqos/qgraph.h" #include "libqos/pci.h" #include "block/nvme.h" @@ -142,6 +144,395 @@ static void nvmetest_pmr_reg_test(void *obj, void *da= ta, QGuestAllocator *alloc) qpci_iounmap(pdev, pmr_bar); } =20 +#define PAGE_SIZE 4096 + +typedef struct nvme_ctrl nvme_ctrl; + +typedef struct nvme_queue { + nvme_ctrl *ctrl; + uint64_t doorbell; + uint32_t size; +} nvme_queue; + +typedef struct nvme_cq { + nvme_queue common; + NvmeCqe *phys_cqe; + uint16_t head; + uint8_t phase; +} nvme_cq; + +typedef struct nvme_sq { + nvme_queue common; + NvmeCmd *phys_sqe; + nvme_cq *cq; + uint16_t head; + uint16_t tail; +} nvme_sq; + +struct nvme_ctrl { + QGuestAllocator *alloc; + QPCIDevice *pdev; + QPCIBar bar; + + uint32_t db_stride; + + nvme_sq admin_sq; + nvme_cq admin_cq; +}; + +static void nvme_init_queue_common(nvme_ctrl *ctrl, nvme_queue *q, + uint16_t db_idx, uint32_t size) +{ + q->ctrl =3D ctrl; + q->doorbell =3D (sizeof(NvmeBar) + db_idx * ctrl->db_stride); + g_test_message(" q %p db_idx %u doorbell %lx", q, db_idx, q->doorbell); + q->size =3D size; +} + +static void nvme_init_sq(nvme_ctrl *ctrl, nvme_sq *sq, uint16_t db_idx, + uint32_t size, nvme_cq *cq) +{ + nvme_init_queue_common(ctrl, &sq->common, db_idx, size); + + sq->phys_sqe =3D (typeof(sq->phys_sqe))guest_alloc(ctrl->alloc, + PAGE_SIZE); + g_assert(sq->phys_sqe); + + g_test_message("sq %p db_idx %u sqe %p", sq, db_idx, sq->phys_sqe); + sq->cq =3D cq; + sq->head =3D 0; + sq->tail =3D 0; +} + +static void nvme_init_cq(nvme_ctrl *ctrl, nvme_cq *cq, uint16_t db_idx, + uint32_t size) +{ + nvme_init_queue_common(ctrl, &cq->common, db_idx, size); + + cq->phys_cqe =3D (typeof(cq->phys_cqe))guest_alloc(ctrl->alloc, + PAGE_SIZE); + g_assert(cq->phys_cqe); + + g_test_message("cq %p db_idx %u cqe %p", cq, db_idx, cq->phys_cqe); + cq->head =3D 0; + cq->phase =3D 1; +} + +static int nvme_cqe_pending(nvme_cq *cq) +{ + uint16_t status =3D qtest_readw(cq->common.ctrl->pdev->bus->qts, + (uint64_t)&cq->phys_cqe[cq->head].status= ); + return (le16_to_cpu(status) & 1) =3D=3D cq->phase; +} + +static int nvme_is_cqe_success(NvmeCqe *cqe) +{ + return (le16_to_cpu(cqe->status) >> 1) =3D=3D NVME_SUCCESS; +} + +static NvmeCqe nvme_handle_cqe(nvme_sq *sq) +{ + nvme_cq *cq =3D sq->cq; + NvmeCqe *phys_cqe =3D &cq->phys_cqe[cq->head]; + NvmeCqe cqe; + uint16_t cq_next_head; + + g_assert(nvme_cqe_pending(cq)); + + qtest_memread(sq->common.ctrl->pdev->bus->qts, (uint64_t)phys_cqe, &cq= e, sizeof(cqe)); + + cq_next_head =3D (cq->head + 1) % cq->common.size; + g_test_message("cq %p head %u -> %u", cq, cq->head, cq_next_head); + if (cq_next_head < cq->head) { + cq->phase ^=3D 1; + } + cq->head =3D cq_next_head; + + if (cqe.sq_head !=3D sq->head) { + sq->head =3D cqe.sq_head; + g_test_message("sq %p head =3D %u", sq, sq->head); + } + + qpci_io_writel(cq->common.ctrl->pdev, cq->common.ctrl->bar, cq->common= .doorbell, cq->head); + + return cqe; +} + +static NvmeCqe nvme_wait(nvme_sq *sq) +{ + int i; + bool ready =3D false; + + for (i =3D 0; i < 10; i++) { + if (nvme_cqe_pending(sq->cq)) { + ready =3D true; + break; + } + + g_usleep(1000); + } + + g_assert(ready); + + return nvme_handle_cqe(sq); +} + +static NvmeCmd *nvme_get_next_sqe(nvme_sq *sq, uint8_t opcode, uint16_t ci= d, void *prp1) +{ + NvmeCmd *phys_sqe =3D &sq->phys_sqe[sq->tail]; + + if (((sq->tail + 1) % sq->common.size) =3D=3D sq->head) { + /* no space in SQ */ + g_test_message("%s head %d tail %d", __func__, sq->head, sq->tail); + g_assert(false); + return NULL; + } + + qtest_memset(sq->common.ctrl->pdev->bus->qts, + (uint64_t)phys_sqe, 0, sizeof(*phys_sqe)); + + #define GUEST_MEM_WRITE(fn, field, val) \ + fn(sq->common.ctrl->pdev->bus->qts, (uint64_t)&(field), (val)) + + GUEST_MEM_WRITE(qtest_writeb, phys_sqe->opcode, opcode); + GUEST_MEM_WRITE(qtest_writew, phys_sqe->cid, cid); + GUEST_MEM_WRITE(qtest_writeq, phys_sqe->dptr.prp1, (uint32_t)(uint64_t= )prp1); + + #undef GUEST_MEM_WRITE + + g_test_message("sq %p next_sqe %u sqe %p", sq, sq->tail, phys_sqe); + return phys_sqe; +} + +static void nvme_commit_sqe(nvme_sq *sq) +{ + g_test_message("sq %p commit sqe tail %u", sq, sq->tail); + sq->tail =3D (sq->tail + 1) % sq->common.size; + qpci_io_writel(sq->common.ctrl->pdev, sq->common.ctrl->bar, sq->common= .doorbell, sq->tail); +} + +static NvmeIdCtrl *nvme_admin_identify_ctrl(nvme_ctrl *ctrl, uint16_t cid,= bool no_wait) +{ + NvmeCmd *phys_cmd_identify; + NvmeIdCtrl *phys_identify; + NvmeCqe cqe; + + g_test_message("sending req cid %u no_wait %d", cid, no_wait); + + phys_identify =3D (typeof(phys_identify))guest_alloc(ctrl->alloc, PAGE= _SIZE); + g_assert(phys_identify); + + phys_cmd_identify =3D nvme_get_next_sqe(&ctrl->admin_sq, + NVME_ADM_CMD_IDENTIFY, cid, + phys_identify); + g_assert(phys_cmd_identify); + + #define GUEST_MEM_WRITE(fn, field, val) \ + fn(ctrl->pdev->bus->qts, (uint64_t)&(field), (val)) + + GUEST_MEM_WRITE(qtest_writel, phys_cmd_identify->nsid, 0); + GUEST_MEM_WRITE(qtest_writel, ((NvmeIdentify *)phys_cmd_identify)->cns= , NVME_ID_CNS_CTRL); + + #undef GUEST_MEM_WRITE + + nvme_commit_sqe(&ctrl->admin_sq); + + if (no_wait) { + return phys_identify; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + g_assert(cqe.cid =3D=3D cid); + + return phys_identify; +} + +static void nvme_wait_ready(nvme_ctrl *ctrl, int val) +{ + int i; + + for (i =3D 0; i < 10; i++) { + uint32_t csts =3D qpci_io_readl(ctrl->pdev, ctrl->bar, NVME_REG_CS= TS); + g_test_message("%s: csts %x", __func__, csts); + + if (NVME_CSTS_RDY(csts) =3D=3D val) { + return; + } + + g_usleep(1000); + } + + g_assert(false); +} + +static void test_migrate_setup_nvme_ctrl(nvme_ctrl *ctrl) +{ + uint64_t cap; + + /* disable controller */ + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, 0); + nvme_wait_ready(ctrl, 0); + + cap =3D qpci_io_readq(ctrl->pdev, ctrl->bar, NVME_REG_CAP); + ctrl->db_stride =3D 4 << NVME_CAP_DSTRD(cap); + + nvme_init_cq(ctrl, &ctrl->admin_cq, 1, 2 /* CQEs num */); + nvme_init_sq(ctrl, &ctrl->admin_sq, 0, 4 /* SQEs num */, &ctrl->admin_= cq); + + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_AQA, + ((ctrl->admin_cq.common.size - 1) << AQA_ACQS_SHIFT) | + ((ctrl->admin_sq.common.size - 1) << AQA_ASQS_SHIFT) + ); + + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ASQ, (uint64_t)ctrl->admin_sq.phys_sqe); + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ACQ, (uint64_t)ctrl->admin_cq.phys_cqe); + + /* enable controller */ + { + uint32_t cc =3D 0; + NVME_SET_CC_EN(cc, 1); + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, cc); + } + + nvme_wait_ready(ctrl, 1); +} + +typedef struct test_migrate_req { + uint16_t cid; + bool handle_cqe; + NvmeIdCtrl *phys_identify; +} test_migrate_req; + +static void test_migrate_send_nvme_reqs(nvme_ctrl *ctrl, test_migrate_req = *reqs, + int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + reqs[i].phys_identify =3D nvme_admin_identify_ctrl(ctrl, reqs[i].c= id, + !reqs[i].handle_c= qe); + g_assert(reqs[i].phys_identify); + + if (reqs[i].handle_cqe) { + guest_free(ctrl->alloc, (uint64_t)reqs[i].phys_identify); + } + } +} + +static void test_migrate_check_nvme(nvme_ctrl *ctrl, test_migrate_req *req= s, int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + NvmeCqe cqe; + + if (reqs[i].handle_cqe) { + continue; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + + g_assert_cmpint(cqe.cid, =3D=3D, reqs[i].cid); + + #define GUEST_MEM_READB(field) \ + qtest_readb(ctrl->pdev->bus->qts, (uint64_t)&(= field)) + + g_assert_cmpint(GUEST_MEM_READB(reqs[i].phys_identify->ieee[0]), = =3D=3D, 0x0); + g_assert_cmpint(GUEST_MEM_READB(reqs[i].phys_identify->ieee[1]), = =3D=3D, 0x54); + g_assert_cmpint(GUEST_MEM_READB(reqs[i].phys_identify->ieee[2]), = =3D=3D, 0x52); + + #undef GUEST_MEM_READB + + guest_free(ctrl->alloc, (uint64_t)reqs[i].phys_identify); + } +} + +static void test_migrate(void *obj, void *data, QGuestAllocator *alloc) +{ + g_autofree gchar *tmpfs =3D NULL; + GError *err =3D NULL; + g_autofree gchar *mig_path; + g_autofree gchar *uri; + GString *dest_cmdline; + QTestState *to; + QDict *rsp; + QNvme *nvme =3D obj; + QPCIDevice *pdev =3D &nvme->dev; + nvme_ctrl *ctrl; + test_migrate_req test_reqs[] =3D { + { 123, true }, + { 456, false }, + { 300, false }, + { 333, false } + }; + + /* create temporary dir and prepare unix socket path for migration */ + tmpfs =3D g_dir_make_tmp("nvme-test-XXXXXX", &err); + if (!tmpfs) { + g_test_message("Can't create temporary directory in %s: %s", + g_get_tmp_dir(), err->message); + g_error_free(err); + } + g_assert(tmpfs); + + mig_path =3D g_strdup_printf("%s/socket.mig", tmpfs); + uri =3D g_strdup_printf("unix:%s", mig_path); + + /* enable NVMe PCI device */ + qpci_device_enable(pdev); + + ctrl =3D g_malloc0(sizeof(*ctrl)); + ctrl->alloc =3D alloc; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D global_qtest); + + test_migrate_setup_nvme_ctrl(ctrl); + test_migrate_send_nvme_reqs(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + dest_cmdline =3D g_string_new(qos_get_current_command_line()); + g_string_append_printf(dest_cmdline, " -incoming %s", uri); + + /* Create destination VM */ + to =3D qtest_init(dest_cmdline->str); + + /* Get access to PCI device from destination VM */ + nvme =3D qos_allocate_objects(to, &ctrl->alloc); + pdev =3D &nvme->dev; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D to); + + /* Migrate VM */ + rsp =3D qmp("{ 'execute': 'migrate', 'arguments': { 'uri': %s } }", ur= i); + g_assert(qdict_haskey(rsp, "return")); + qobject_unref(rsp); + + /* Wait when source VM is stopped */ + qmp_eventwait("STOP"); + + /* Copy guest physical memory allocator state */ + migrate_allocator(alloc, ctrl->alloc); + + /* Wait for destination VM to become alive */ + qtest_qmp_eventwait(to, "RESUME"); + + test_migrate_check_nvme(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + qtest_quit(to); + g_unlink(mig_path); + g_rmdir(tmpfs); + g_string_free(dest_cmdline, true); +} + static void nvme_register_nodes(void) { QOSGraphEdgeOptions opts =3D { @@ -168,6 +559,8 @@ static void nvme_register_nodes(void) }); =20 qos_add_test("reg-read", "nvme", nvmetest_reg_read_test, NULL); + + qos_add_test("migrate", "nvme", test_migrate, NULL); } =20 libqos_init(nvme_register_nodes); --=20 2.47.3