From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512348; cv=none; d=zohomail.com; s=zohoarc; b=Ot4TPi0p+ejX823OQNISmU38jU4Hxnv+u7kEovrFrl5MVfpGFCcKRBUsKsvB3pmTRTa59mpzOuOdanv1LNUp8H7GazrVACUjuPB3FhJnN9/cUpRv6Shjjyb6LqVKvCd0FFPxhyaCd7qbrGV+3CYn4dO14KeQcoA5JBTldQJaB84= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512348; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=1CxqHbd3By135psgYqUZT9zQLEQSBWcZUYKQnFu3P5w=; b=Gsabu/YEWHtdQXne0oBvKCq34XURkUNmqyBHRPTg5E3KCkwdcaVAtzyYNQdaC9baa6Y3GSlV1+rGc4nXT/9sm1k0eOrKSmDpH3T7g2AO/aJUcTAgw1XqilxcBLmaE1BvSOD3IUwJQnBPe/uUxFqy/YakQgLsMUxtIU1afyq6jiI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512348445801.1233808866024; Mon, 11 May 2026 08:12:28 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSIR-0005pR-BQ; Mon, 11 May 2026 11:11:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIJ-0005om-Dv for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:43 -0400 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIG-0001or-V2 for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:42 -0400 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-488a9033b2cso41324565e9.2 for ; Mon, 11 May 2026 08:11:40 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512299; x=1779117099; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1CxqHbd3By135psgYqUZT9zQLEQSBWcZUYKQnFu3P5w=; b=DFywGeQhk3UMFO/GLvuhBs3ao6TnqzB5tjIDZEMt1DW5XiBaHthOS1iJVkrsrthwhb I9/GWcuFqYbuvkauvD7Rr6cDyuoFqhwuC0D9Kh+d3QF4709Ur6G7aPUgITULdU7OXt4B CYlnJH8XFgi0HrnPD0NBypAnGDIndRuj2Cxy4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512299; x=1779117099; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=1CxqHbd3By135psgYqUZT9zQLEQSBWcZUYKQnFu3P5w=; b=PyhBUf3dsleXOu99azwi9mAxQTy5biQk90rW0fBs5RTFHOwzAppVe5PbbWE4YIqIKq W5FJPe1+RkHOP0FykfJWUXFDAMUCCDkvz1Geb5xO/bjiPqgvppaDPti9IOxfbKLHcPST devKuXdaodr6MH72MLHg1GIXV5q7hYCbf8YQjpeD2Xu7g/1uGyn36vr0pDLDSoXDCv6h lYEet+ZgCmpwf4A0fdnBPNoiTpPYlm8rgn4cXR6jLrz2N0sZFKVqGvmzq6qjcGj8imFG L/ZxROwx01t2Nm2nRe4LmPQdqzKmZxHbFO23d2csQ7+PLrJuZM02FQHuHyYBkqZdcFoi Qvug== X-Gm-Message-State: AOJu0Yw8jmmg0ioe2GaVUd922m/BEkrC9zvtXJe3mF+6aYZWAPTNbVAN 15Az/PICXSZhtrmJ7FOVMXV3Y7WKj9nDvyWXk1oxYLor7ZENmuI/lLhh0mAvFmi7RyNdlh6mdWH Aef9achU= X-Gm-Gg: Acq92OEubPgKWrO/xM/hvOB+bU7U8aTwcO3dHAjtHbocvgYw2pW+eDglUk7Hm1AiUwp Tg2+tGLY5K7at6Uj6K5ojYeNI1m/X+fttmPE3Wr42AaeN5KGUbybrm+RW+gfa++ACu9YJjyMpHU hdj/HYIDXjxPd73IY95uR2AD3eV0XYIvPwiKcogy98q6zXjfE0DpJq3mm7ACzGaJZeoPAraziKJ ZWHsFx1l+IJXwrGBCyCzzih7u0EgMO/gw+4gjw+UocWpea1vSQ995/ZcCBiXpKvSQzfHrzHKTFL zOmBLg73qr/heMBoVo9KsoQjZVcmXVJRdxHzvwtLrk1qJkhsQhG3MSx1PZziwQMGk4XOSDZBjeY QUsrMFUlxacnFw7r1PJg1lWRKgr11zgxvmf8hXN0o5TBeNxICfLu/Qi7dTVGHU3/g2dlGDUdgVF F1JeXDNwDG9kvrmsvK4gvI1wsNJOwEeXDb0erKMKbHR0QfMNUr0YcRWGEHFwgXHfSKNelNWKx31 4CY17/5V2YNjEXvIrML X-Received: by 2002:a05:600c:19c8:b0:489:1c2d:211e with SMTP id 5b1f17b1804b1-48e70687ed2mr170139945e9.5.1778512294752; Mon, 11 May 2026 08:11:34 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn Subject: [PATCH v7 1/8] tests/functional/migration: add VM launch/configure hooks Date: Mon, 11 May 2026 17:11:24 +0200 Message-ID: <20260511151131.483217-2-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::331; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x331.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512351517154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce configure_machine, launch_source_vm and assert_dest_vm methods to allow child classes to override some pieces of source/dest VMs creation, start and check logic. Signed-off-by: Alexander Mikhalitsyn Reviewed-by: Peter Xu --- tests/functional/migration.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/functional/migration.py b/tests/functional/migration.py index 3b7674af3b6..1932151ab64 100644 --- a/tests/functional/migration.py +++ b/tests/functional/migration.py @@ -40,19 +40,35 @@ def assert_migration(self, src_vm, dst_vm): self.assertEqual(dst_vm.cmd('query-status')['status'], 'running') self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate= ') =20 + # Can be overridden by subclasses to configure both source/dest VMs. + def configure_machine(self, vm): + vm.add_args('-nodefaults') + + # Can be overridden by subclasses to prepare the source VM before migr= ation, + # e.g. by running some workload inside the source VM to see if it cont= inues + # to run properly after migration. + def launch_source_vm(self, vm): + vm.launch() + + # Can be overridden by subclasses to check the destination VM after mi= gration, + # e.g. by checking if the workload is still running after migration. + def assert_dest_vm(self, vm): + pass + def migrate_vms(self, dst_uri, src_uri, dst_vm, src_vm): dst_vm.qmp('migrate-incoming', uri=3Ddst_uri) src_vm.qmp('migrate', uri=3Dsrc_uri) self.assert_migration(src_vm, dst_vm) + self.assert_dest_vm(dst_vm) =20 def migrate(self, dst_uri, src_uri=3DNone): dst_vm =3D self.get_vm('-incoming', 'defer', name=3D"dst-qemu") - dst_vm.add_args('-nodefaults') + self.configure_machine(dst_vm) dst_vm.launch() =20 src_vm =3D self.get_vm(name=3D"src-qemu") - src_vm.add_args('-nodefaults') - src_vm.launch() + self.configure_machine(src_vm) + self.launch_source_vm(src_vm) =20 if src_uri is None: src_uri =3D dst_uri --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512363; cv=none; d=zohomail.com; s=zohoarc; b=HfETNjoyAZwAXYfxJKdNPLTTtQGm4YhyylccGpkKMl5bl8KkxUfEkTHRQeQ9ayzxn6V7C+H+gqEfH4WJAxxus5a2fq443nc3kF3tSROlgYwVm5wysaBLWgObnen1BdRdBbVUVt+XXIBrgrrSbd9x3cc3amKenlv+X0IRsoO1eO0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512363; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=YwItJNDspg4dGH2S+8T7C+pecZvU41/84qZaYtEAlvM=; b=R5azX3pounyqH19qAPoonbhkR1lDHjtzLSvif0vnxQsv37V1/PGiGWyDVSs71lSOqkTHzbEP7SUs1cVxzsI4z9lauLUYQI/BjuuwJSr6Q6i6sdeH1477rC9HVy74/3dw4lnjSljEy8wxMfXBalYBsuyVEeLpXZ/O0HmLGDIamsw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512363559717.845508252892; Mon, 11 May 2026 08:12:43 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSIy-0005yV-Nm; Mon, 11 May 2026 11:12:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIK-0005p3-CM for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:45 -0400 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSII-0001pQ-2y for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:44 -0400 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-488a14c31eeso30921305e9.0 for ; Mon, 11 May 2026 08:11:41 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512301; x=1779117101; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YwItJNDspg4dGH2S+8T7C+pecZvU41/84qZaYtEAlvM=; b=OEIlVpszcGfprixzMXMGPjMzVQnMv9XcV5Km6yK+OwwkuerTv2oNS7LvyiZe+h8hmE KhJY2YuyztTVejuitoVOkcNLzuPQ33A2ZpONTGj7ifGrSfN9OD1TuBg/ACfqxBhR53ZD FcsLvrs8ZaYtAFmEd7+/aVn++4nLySVleBaTo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512301; x=1779117101; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=YwItJNDspg4dGH2S+8T7C+pecZvU41/84qZaYtEAlvM=; b=Ybz5MbFiITYqqtEHf/R/ZSsJ2AiOGDn5JNKcF4+evq1E0rf122J7XWU7IdNhhA0FJM YmPhGBamq1lhlttEJC22qOhL10gSc57npKl6jbwhy8FiTPHdGdrce/Al3KxHrQa4JEED NcQjkKuu3vvpvHQOy4Eg3vKyLgaXncUFBFEi6DY5hN8OpMVtWUDqT0FHo9aI1IjxXA9J aGym3lCDgCkvFvTeM6ANJEnV/n2fczsfJlKDUGizjVWD50D9ZGrf1Ju3SQ7DpR9jQ3QH r6UdZ9bzXYUTZ6kWHSjffz2zhzaloIDni9fJ75xLwPRS3h6lydLy7Dmm1naASCnVHBZI Lv6w== X-Gm-Message-State: AOJu0Yy3BtWRc1r8xv1xh7RsAYqqDsVkb6qujmeX/4ewku9qswovT16R 889578WQ4R12E/S7bIRBJ08/UJLAGAHOrCXSpADpkftwGA3qJKzAEAy8HNlI943AUo7VRNrb8iA qexkNoNU= X-Gm-Gg: Acq92OHGKUTguhARiUxY66VpFuy16SNUuNQNlc1qwo7mQ71OGueQgFt8TjsprWLzgas n2FVTEdmcbDvqRKWRPRUFzTVnJGc6S+IilZxZbUyaDdSysKZpXenzGnl7MmepmyqFQArE8eA+6y qv7DHTvB4Gl4iIS5oWN0w0X8meycS6Pi50+J94l88joS5G9Lds3XElmxe+3UtaC9vGoCLzPF3x9 jisxBXGd+OZebPeL4LIlNUu/CQ7orx0E80V//e3wFjbcMdm5uYcV/ZLm/T4WAvxcxwL841yLBcA 6lnv8/2LLECeQ3xjKAIMoaXiK7cJhbesR9yR6rXD3Pct52RRF/Ev1dNJEjyABh8yemPZF1ChHXj nkWrrCM/KEeJOrGwMOuXn9RADlMd59l9H5/cOfSHBa0ib1cL3Enm4vk50dw1IHXZdsqCdNBga7A tqyvXi2lytBEKn+ZXiATPFOzA01+24ODPvy1ZeUILdGWAJjk2l/6X2El5HJvFJyDEX9LPxEQbp1 CW7M2JE95QfjiuoXQ+A X-Received: by 2002:a05:600c:198b:b0:48a:592c:e63d with SMTP id 5b1f17b1804b1-48e676a5245mr251037455e9.14.1778512295928; Mon, 11 May 2026 08:11:35 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn Subject: [PATCH v7 2/8] hw/nvme: add migration blockers for non-supported cases Date: Mon, 11 May 2026 17:11:25 +0200 Message-ID: <20260511151131.483217-3-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x330.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512365743158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Let's block migration for cases we don't support: - SR-IOV - CMB - PMR - SPDM No functional changes here, because NVMe migration is not supported at all as of this commit. Signed-off-by: Alexander Mikhalitsyn Reviewed-by: Klaus Jensen --- hw/nvme/ctrl.c | 206 +++++++++++++++++++++++++++++++++++++++++++ hw/nvme/nvme.h | 3 + include/block/nvme.h | 12 +++ 3 files changed, 221 insertions(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index be6c7028cb5..167b8da6342 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -207,6 +207,7 @@ #include "hw/pci/msix.h" #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" +#include "migration/blocker.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -250,6 +251,7 @@ static const bool nvme_feature_support[NVME_FID_MAX] = =3D { [NVME_COMMAND_SET_PROFILE] =3D true, [NVME_FDP_MODE] =3D true, [NVME_FDP_EVENTS] =3D true, + /* if you add something here, please update nvme_set_migration_blocker= s() */ }; =20 static const uint32_t nvme_feature_cap[NVME_FID_MAX] =3D { @@ -4601,6 +4603,7 @@ static uint16_t nvme_io_mgmt_send(NvmeCtrl *n, NvmeRe= quest *req) return 0; case NVME_IOMS_MO_RUH_UPDATE: return nvme_io_mgmt_send_ruh_update(n, req); + /* if you add something here, please update nvme_set_migration_blocker= s() */ default: return NVME_INVALID_FIELD | NVME_DNR; }; @@ -7520,6 +7523,10 @@ static uint16_t nvme_security_receive(NvmeCtrl *n, N= vmeRequest *req) =20 static uint16_t nvme_directive_send(NvmeCtrl *n, NvmeRequest *req) { + /* + * When adding a new dtype handling here, + * please also update nvme_set_migration_blockers(). + */ return NVME_INVALID_FIELD | NVME_DNR; } =20 @@ -9210,6 +9217,199 @@ static void nvme_init_ctrl(NvmeCtrl *n, PCIDevice *= pci_dev) } } =20 +#define BLOCKER_FEATURES_MAX_LEN 256 + +static inline void nvme_add_blocker_feature(char *blocker_features, const = char *feature) +{ + if (strlen(blocker_features) > 0) { + g_strlcat(blocker_features, ", ", BLOCKER_FEATURES_MAX_LEN); + } + g_strlcat(blocker_features, feature, BLOCKER_FEATURES_MAX_LEN); +} + +static bool nvme_set_migration_blockers(NvmeCtrl *n, PCIDevice *pci_dev, E= rror **errp) +{ + uint64_t unsupported_cap, cap =3D ldq_le_p(&n->bar.cap); + char blocker_features[BLOCKER_FEATURES_MAX_LEN] =3D ""; + bool adm_cmd_security_checked =3D false; + bool cmd_io_mgmt_checked =3D false; + bool cmd_zone_checked =3D false; + + /* + * Idea of this function is simple, we iterate over all Command Sets a= nd + * for each supported command we provide a special handling logic to + * determine if we should block migration or not. + * + * For instance, we have NVME_ADM_CMD_NS_ATTACHMENT and it is always + * available to the guest, but if there is only 1 namespace, then it is + * safe to allow migration, but if there are more, then we need to blo= ck + * migration because we don't handle this in migration code yet. + */ + for (int opcode =3D 0; opcode < sizeof(n->cse.acs) / sizeof(n->cse.acs= [0]); opcode++) { + /* Is command supported? */ + if (!n->cse.acs[opcode]) { + continue; + } + + switch (opcode) { + case NVME_ADM_CMD_DELETE_SQ: + case NVME_ADM_CMD_CREATE_SQ: + case NVME_ADM_CMD_GET_LOG_PAGE: + case NVME_ADM_CMD_DELETE_CQ: + case NVME_ADM_CMD_CREATE_CQ: + case NVME_ADM_CMD_IDENTIFY: + case NVME_ADM_CMD_ABORT: + case NVME_ADM_CMD_SET_FEATURES: + case NVME_ADM_CMD_GET_FEATURES: + case NVME_ADM_CMD_ASYNC_EV_REQ: + case NVME_ADM_CMD_DBBUF_CONFIG: + case NVME_ADM_CMD_FORMAT_NVM: + case NVME_ADM_CMD_DIRECTIVE_SEND: + case NVME_ADM_CMD_DIRECTIVE_RECV: + break; + case NVME_ADM_CMD_NS_ATTACHMENT: + int namespaces_num =3D 0; + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + namespaces_num++; + } + + if (namespaces_num > 1) { + nvme_add_blocker_feature(blocker_features, "Namespace Atta= chment"); + } + + break; + case NVME_ADM_CMD_VIRT_MNGMT: + if (n->params.sriov_max_vfs) { + nvme_add_blocker_feature(blocker_features, "SR-IOV"); + } + + break; + case NVME_ADM_CMD_SECURITY_SEND: + case NVME_ADM_CMD_SECURITY_RECV: + if (adm_cmd_security_checked) { + break; + } + + if (pci_dev->spdm_port) { + nvme_add_blocker_feature(blocker_features, "SPDM"); + } + + adm_cmd_security_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < sizeof(n->cse.iocs.nvm) / sizeof(n->cs= e.iocs.nvm[0]); opcode++) { + if (!n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_FLUSH: + case NVME_CMD_WRITE: + case NVME_CMD_READ: + case NVME_CMD_COMPARE: + case NVME_CMD_WRITE_ZEROES: + case NVME_CMD_DSM: + case NVME_CMD_VERIFY: + case NVME_CMD_COPY: + break; + case NVME_CMD_IO_MGMT_RECV: + case NVME_CMD_IO_MGMT_SEND: + if (cmd_io_mgmt_checked) { + break; + } + + /* check for NVME_IOMS_MO_RUH_UPDATE */ + if (n->subsys->params.fdp.enabled) { + nvme_add_blocker_feature(blocker_features, "FDP"); + } + + cmd_io_mgmt_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + for (int opcode =3D 0; opcode < sizeof(n->cse.iocs.zoned) / sizeof(n->= cse.iocs.zoned[0]); opcode++) { + /* + * If command isn't supported or we have the same command + * in n->cse.iocs.nvm, then we can skip it here. + */ + if (!n->cse.iocs.zoned[opcode] || n->cse.iocs.nvm[opcode]) { + continue; + } + + switch (opcode) { + case NVME_CMD_ZONE_APPEND: + case NVME_CMD_ZONE_MGMT_SEND: + case NVME_CMD_ZONE_MGMT_RECV: + if (cmd_zone_checked) { + break; + } + + for (int i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + if (!ns) { + continue; + } + + if (ns->params.zoned) { + nvme_add_blocker_feature(blocker_features, "Zoned Name= space"); + break; + } + } + + cmd_zone_checked =3D true; + + break; + default: + g_assert_not_reached(); + } + } + + /* + * Try our best to explicitly detect all not supported caps, + * to let users know what features cause migration to be blocked, + * but in case we miss handling here, everything else will be + * covered by unsupported_cap check. + */ + if (NVME_CAP_CMBS(cap)) { + nvme_add_blocker_feature(blocker_features, "CMB"); + cap &=3D ~((uint64_t)CAP_CMBS_MASK << CAP_CMBS_SHIFT); + } + + if (NVME_CAP_PMRS(cap)) { + nvme_add_blocker_feature(blocker_features, "PMR"); + cap &=3D ~((uint64_t)CAP_PMRS_MASK << CAP_PMRS_SHIFT); + } + + unsupported_cap =3D cap & ~NVME_MIGRATION_SUPPORTED_CAP_BITS; + if (unsupported_cap) { + nvme_add_blocker_feature(blocker_features, "unknown capability"); + } + + assert(n->migration_blocker =3D=3D NULL); + if (strlen(blocker_features) > 0) { + error_setg(&n->migration_blocker, "Migration is not supported for = %s", blocker_features); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + + return true; +} + static int nvme_init_subsys(NvmeCtrl *n, Error **errp) { int cntlid; @@ -9315,6 +9515,10 @@ static void nvme_realize(PCIDevice *pci_dev, Error *= *errp) =20 n->subsys->namespaces[ns->params.nsid] =3D ns; } + + if (!nvme_set_migration_blockers(n, pci_dev, errp)) { + return; + } } =20 static void nvme_exit(PCIDevice *pci_dev) @@ -9367,6 +9571,8 @@ static void nvme_exit(PCIDevice *pci_dev) } =20 memory_region_del_subregion(&n->bar0, &n->iomem); + + migrate_del_blocker(&n->migration_blocker); } =20 static const Property nvme_props[] =3D { diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index d66f7dc82d5..457b6637249 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -666,6 +666,9 @@ typedef struct NvmeCtrl { =20 /* Socket mapping to SPDM over NVMe Security In/Out commands */ int spdm_socket; + + /* Migration-related stuff */ + Error *migration_blocker; } NvmeCtrl; =20 typedef enum NvmeResetType { diff --git a/include/block/nvme.h b/include/block/nvme.h index 9d7159ed7a7..a7f586fc801 100644 --- a/include/block/nvme.h +++ b/include/block/nvme.h @@ -141,6 +141,18 @@ enum NvmeCapMask { #define NVME_CAP_SET_CMBS(cap, val) \ ((cap) |=3D (uint64_t)((val) & CAP_CMBS_MASK) << CAP_CMBS_SHIFT) =20 +#define NVME_MIGRATION_SUPPORTED_CAP_BITS ( \ + ((uint64_t)CAP_MQES_MASK << CAP_MQES_SHIFT) \ + | ((uint64_t)CAP_CQR_MASK << CAP_CQR_SHIFT) \ + | ((uint64_t)CAP_AMS_MASK << CAP_AMS_SHIFT) \ + | ((uint64_t)CAP_TO_MASK << CAP_TO_SHIFT) \ + | ((uint64_t)CAP_DSTRD_MASK << CAP_DSTRD_SHIFT) \ + | ((uint64_t)CAP_NSSRS_MASK << CAP_NSSRS_SHIFT) \ + | ((uint64_t)CAP_CSS_MASK << CAP_CSS_SHIFT) \ + | ((uint64_t)CAP_MPSMIN_MASK << CAP_MPSMIN_SHIFT) \ + | ((uint64_t)CAP_MPSMAX_MASK << CAP_MPSMAX_SHIFT) \ +) + enum NvmeCapCss { NVME_CAP_CSS_NCSS =3D 1 << 0, NVME_CAP_CSS_IOCSS =3D 1 << 6, --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512411; cv=none; d=zohomail.com; s=zohoarc; b=Wcppe9oPxlbcR9zeHsCFEzFzQrHxuWiwEt5/TFPFbouVI1HSiZNNqtnREIc3+j22OPsAhDC5GzmWJINVktd/vVpuoUJD5lHxG0RnmzAuMUJCaBj7wLZVDQNpdbisPs4vtUwQCkPz+eimKMDSU7mDzf9cNoq/PuE4D7hAaw9wKD0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512411; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=KdKjGUvxwDyenN9C+PKFKdrIeRGRcm18kTsUgAEHB4A=; b=IKrishqVXwt0W8zbK09B6uGGFi9A0uLOgGsxq8KY4cb945BCKF9u8HPuDIUatwZ8g+GQo9YTWzqsb2er9nfgAXM1eIBt+Gwy7MhqKy667eafb3loIXQ+cy79I0TGLjp/CkpuCzXP7C83F4DQwvpCSMUp5ZuEv2Q4v5FtgdgoyrE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512411752426.0890329241081; Mon, 11 May 2026 08:13:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJs-0006ol-PO; Mon, 11 May 2026 11:13:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIL-0005p6-19 for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:46 -0400 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIJ-0001q5-Fh for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:44 -0400 Received: by mail-wr1-x42f.google.com with SMTP id ffacd0b85a97d-44e1860558fso2909965f8f.0 for ; Mon, 11 May 2026 08:11:43 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512302; x=1779117102; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KdKjGUvxwDyenN9C+PKFKdrIeRGRcm18kTsUgAEHB4A=; b=VnHdilDO84u3O/iIX9Twdz231Kfp7wZF+3IoY9smi+Fqwtd6yrF8DPJG/vjsW50Q1i anEMp5DkunjVZPKbkXDsJ25aeayX274rIlvowtYPPCJhrSms1lk4j2lVtKzLbo1xSE7G 8F4Zp2zVhntuiUb6lvX8s0nkcJvCOGLRssqZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512302; x=1779117102; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=KdKjGUvxwDyenN9C+PKFKdrIeRGRcm18kTsUgAEHB4A=; b=VqQGSR8JJ40DPV6x6iIFEr7pxBrDT/7wz5qErN8yeoV0rFnQms6I+CZcWYkUYpmmca PFP5dHycNJDCVvgsu0EIY8xqp/M2KgGGFbo7QyjZCr38S3YGDey783kCdzzWoiapKZEe lMKlApJvXJYhhyMw9s3JeSDLIl62VmiZpOi8ap/tf1CRXSlnA4JBxf+Kqne9Q0iYBiy5 UEQVJRZEzLkhbaICQErnnKf55GiEb1GiyL0ePZk8z8v/U6B8JGBP7r1u76qg4pjD7zux CP7bNhXsiHzxzDK8ctOFMyRe1+/6JKJmMTtosXxsjm0/DhX5gx303oekAz1/oSsYdCC0 GOFQ== X-Gm-Message-State: AOJu0YwlUStGNDZTh1q7774RMaNZiBYUtKcE5Ah43M+0rU1QQNnSNoMU GZNTSzuYZFmOAyLj9EbGynaxpProHlEWre7lsIu2AsTZILgMmLU2BkPV/OC3BHlhMrb3q+lFSn7 AZn0yp2k= X-Gm-Gg: Acq92OHUwH34pNSErsSBXTBwC4q9UtARa+ArqSR8LQE7xzgBq9HxA9w3AYSPgicIU1Z YCTei6Zj+wL7GoWw1EbJrNXgrNF3fuktyJZi027rKqCamWybv5SM6O4SQpflCI2HCs4ka5B5Mbo xiM9X8eSD9sNmRtwcpp3DJna/I/tELzy7syS77PotF4ArIoa+OT2Sm/IqjqfHKnMbzEXGbi8lB6 SJwZcTdiLNWNl0vkFugqXGmGc8tqTbABWIDvWMji/lnNfyhl+g96higg4Ar+G8pcDbNCtSQq+XX FLvRftCPew2yGTizz1wMwP7C0cPvAueRu+H6XyKTjha643+Ys3QrFwEyPcKyXCUEpLRVKd3hPiL Jk57eijclWdLGlq3GF8XpWYULeZm0jYBaxNtXGCldwbBBQqc9vVM6FVWBkbr1kuWWn0SUQ5hQOm nH3b4lv7PMS1jhpFNbi4kLhrId0byMHwFybaNpeQghhK/eVbCSufW4gPLaRbRDTlOYZFgus+NJ/ 7nj2eigwBM9Do3zwcPj X-Received: by 2002:a05:600c:8b62:b0:47e:e2eb:bc22 with SMTP id 5b1f17b1804b1-48e6748afe8mr216558655e9.5.1778512297175; Mon, 11 May 2026 08:11:37 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v7 3/8] hw/nvme: split nvme_init_sq/nvme_init_cq into helpers Date: Mon, 11 May 2026 17:11:26 +0200 Message-ID: <20260511151131.483217-4-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42f; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512413831154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn We will make a benefit from this split in later patches. Reviewed-by: Klaus Jensen Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 59 +++++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 167b8da6342..e79ef536ef8 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -4854,18 +4854,14 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t sqid, uint16_t cqid, uint16_t size) +static void __nvme_init_sq(NvmeSQueue *sq) { + NvmeCtrl *n =3D sq->ctrl; + uint16_t sqid =3D sq->sqid; + uint16_t cqid =3D sq->cqid; int i; NvmeCQueue *cq; =20 - sq->ctrl =3D n; - sq->dma_addr =3D dma_addr; - sq->sqid =3D sqid; - sq->size =3D size; - sq->cqid =3D cqid; - sq->head =3D sq->tail =3D 0; sq->io_req =3D g_new0(NvmeRequest, sq->size); =20 QTAILQ_INIT(&sq->req_list); @@ -4895,6 +4891,18 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, n->sq[sqid] =3D sq; } =20 +static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t sqid, uint16_t cqid, uint16_t size) +{ + sq->ctrl =3D n; + sq->dma_addr =3D dma_addr; + sq->sqid =3D sqid; + sq->size =3D size; + sq->cqid =3D cqid; + sq->head =3D sq->tail =3D 0; + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5555,25 +5563,16 @@ static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t cqid, uint16_t vector, uint16_t size, - uint16_t irq_enabled) +static void __nvme_init_cq(NvmeCQueue *cq) { + NvmeCtrl *n =3D cq->ctrl; PCIDevice *pci =3D PCI_DEVICE(n); + uint16_t cqid =3D cq->cqid; =20 - if (msix_enabled(pci) && irq_enabled) { - msix_vector_use(pci, vector); + if (msix_enabled(pci) && cq->irq_enabled) { + msix_vector_use(pci, cq->vector); } =20 - cq->ctrl =3D n; - cq->cqid =3D cqid; - cq->size =3D size; - cq->dma_addr =3D dma_addr; - cq->phase =3D 1; - cq->irq_enabled =3D irq_enabled; - cq->vector =3D vector; - cq->head =3D cq->tail =3D 0; - QTAILQ_INIT(&cq->req_list); QTAILQ_INIT(&cq->sq_list); if (n->dbbuf_enabled) { cq->db_addr =3D n->dbbuf_dbs + (cqid << 3) + (1 << 2); @@ -5590,6 +5589,22 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, &DEVICE(cq->ctrl)->mem_reentrancy_guard); } =20 +static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t cqid, uint16_t vector, uint16_t size, + uint16_t irq_enabled) +{ + cq->ctrl =3D n; + cq->cqid =3D cqid; + cq->size =3D size; + cq->dma_addr =3D dma_addr; + cq->phase =3D 1; + cq->irq_enabled =3D irq_enabled; + cq->vector =3D vector; + cq->head =3D cq->tail =3D 0; + QTAILQ_INIT(&cq->req_list); + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512398; cv=none; d=zohomail.com; s=zohoarc; b=Nsl+7HK+KRc+GdAAiAmnSRqLiCCO3YDTzRdCyTAoyQxT/mGot14wzrNApcJWlvAP8dxp2PTGLFveKD4Ax+p+hUYf4KbN74OUQyaVx4Zt309EVGJMRLOB9sHCWwpCJTbyAB9AQo3T4HVdFIFelgnveoOrniEodPgeGxepyGLNxpo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512398; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=CXtR0f8lswzrNk5sn13Mr7WVfwlNVtnqZBag2y/b9bI=; b=d6hp/qNwJ0OEdu+ywKTb6L3BAFKqps7Wv87mlh2RgVVrmGVO0F4BnwCPQaDk6/3dNiUmwpduj3yr8zd6QWsCLt1zKiFqFBAt2LO4g/6sqg+09dPUvNjQlHuBI6iX3a6bvec1xxV2mxGYPSta9GAfWy+aMRISimgkdriChCpKm/Y= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512398242200.6985268383488; Mon, 11 May 2026 08:13:18 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJ5-00065J-DU; Mon, 11 May 2026 11:12:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIM-0005pF-FG for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:46 -0400 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIK-0001qT-Ol for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:46 -0400 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-444826c16ffso3909793f8f.1 for ; Mon, 11 May 2026 08:11:44 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512303; x=1779117103; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CXtR0f8lswzrNk5sn13Mr7WVfwlNVtnqZBag2y/b9bI=; b=XQoaQtpg80PdIAiSa7NTxkmZJYo6oXZfNBbs3lL30/AglU/6wQmj9WPVt21hZ9ntWM orAZQ8Xdnm4/hrs7XP7zhjkqehthZPhESUeOWysxwRhmP2bCXBPoxqC7jumDesUPhHcO a3tC2igGvJob8/Rma6PnMm8/jb6Dq7AenvOSA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512303; x=1779117103; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=CXtR0f8lswzrNk5sn13Mr7WVfwlNVtnqZBag2y/b9bI=; b=H3KJwBbJyKk1l4fIZxoFyFAeYhCANweq1lofDAv/A+r8RBDd5hIyGmuGqCNr3LZNZ3 2IjZKL+cETHbOndv8tY4xVBc1wev3NsD/uHBWUlOsRZs3jDB+03b0wyEqZwifm0iJQuf TE9/sbhQ2tYrJVoWh/oDqiVoJMHKykcIMbxOpuHaWX7MPEsRZDBOQgoWNPlhjpjgXfOt Tyyjc+8P/Tg5pR1akT3a6Va2NmSHtzWAO11U+nl+FLiDoOBsKey0Xi6LpjReC6wMMbu6 NsFnTokZ6hdC8JZRI8qnl3BiVG5De4uzGrfHXGw8J1uFqJNREFAOSVJz1DnUgVxW+WP7 Hygg== X-Gm-Message-State: AOJu0YygWbADoXXwpvaNkoaXMkMSDHGExHFqVm4ybsUKtrSmc21RyVhe hR9BFFYOx0d5K0k1RY1OJLhWqrXUT2hJuryLT/Jo5O6rmLpnzZFf1GF6Wyfma4mzvFH51FDEhAy MLbPJdag= X-Gm-Gg: Acq92OG5Wui9doHBGEd5vIlGIfblOmhpLGmk5WElcZPxu85LVk1TJdx/wunN58RtvPE iwx30N/B7GsXWy7jFN8eTnY0dlKfvevUqga1HaTODBDGhryzte3PpSYOCAKkxLFO8ABD2dUgEBG UDi4tdrDI8oC/496mmB2bvijFbZHUmxsCCpmoHbuNfStVGK6a38uB6v62fg+/11Rdt9Y25p/Fs/ sTrUoyyw3cT1dWpQUAgcmyOr6RHPbK2HQvbaikZb/3iUptpbtvkolr8wy5AxgDlbuQvlNpFKUgC LuaTqMVuEHyNLmCzCvp7ndyG173839s42WCyyWVkBEr52tup46247gLaRQnj91lEh8oEituaVgR uYP3UHi6vFZSKoI6f3zlWiQ3OM19TrEcUcWzWTUfWUtfUHiXL/g3aaQYuAD3uGgdqXxov4Btwm+ E6j+JGqtz2O4VYv1xRLVHtsXsjp3BbwSqSG115fs3uZrGIb9lhFZirJEGnbERbWUtYPh52aRXAu DD65yFAqus6JZJUfl5WqolG0+FDSnY= X-Received: by 2002:a05:600c:8908:b0:48a:7a10:4f17 with SMTP id 5b1f17b1804b1-48e51f21e0cmr271209205e9.6.1778512298557; Mon, 11 May 2026 08:11:38 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v7 4/8] hw/nvme: set CQE.sq_id earlier in nvme_process_sq Date: Mon, 11 May 2026 17:11:27 +0200 Message-ID: <20260511151131.483217-5-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512399927158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of filling req->cqe.sq_id in nvme_post_cqes, let's set it earlier in nvme_process_sq. This shouldn't cause any issues, because req->cqe.sq_id never changes during lifetime of req. This will help us for migration support. Reviewed-by: Klaus Jensen Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index e79ef536ef8..ffee7ef224c 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1520,7 +1520,6 @@ static void nvme_post_cqes(void *opaque) =20 sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); - req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.sq_head =3D cpu_to_le16(sq->head); addr =3D cq->dma_addr + (cq->tail << NVME_CQES); ret =3D pci_dma_write(PCI_DEVICE(n), addr, (void *)&req->cqe, @@ -7850,6 +7849,7 @@ static void nvme_process_sq(void *opaque) QTAILQ_REMOVE(&sq->req_list, req, entry); QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); nvme_req_clear(req); + req->cqe.sq_id =3D cpu_to_le16(sq->sqid); req->cqe.cid =3D cmd.cid; memcpy(&req->cmd, &cmd, sizeof(NvmeCmd)); =20 --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512414; cv=none; d=zohomail.com; s=zohoarc; b=P31N01ElHRyIwR/44sv+9PNhVzBAwlL9kJqjwUXi9mKZgNoLh5NAHmfzt9Bf0IxG4eQEiqjvjM50ypsoXwNZeo7vDqOPm3x9Upi8QzCiG8ETrfry5AhYhVZn3zakOe656y0ftkbK2GvRU4j3Vp1ITf1hJRohhotYoFlYYdOsPT4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512414; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=yv/dVX0kdD5IT4KjvoobqGU1SwaEsj6wNpgpDbOtol8=; b=PapsLBow6fFn5uBVtPo3X2FmYOtL0vzsHsWkWUKSncfqD5pktao02LCGHxSVGphz07va82c3zrjD7cap69ac0F6FYKGNmQGbDApdm4sBEX9SxAjZRWQCXkve4BtLkk9wLILNKLgV/92G8L08CElsnaNd0SG8LOcdykpbEBF0lHY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512414131628.2924280405082; Mon, 11 May 2026 08:13:34 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJR-00067S-Of; Mon, 11 May 2026 11:12:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIP-0005q9-Fg for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:50 -0400 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIL-0001qv-Vq for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:47 -0400 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-43eb05b1875so2650641f8f.3 for ; Mon, 11 May 2026 08:11:45 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512305; x=1779117105; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yv/dVX0kdD5IT4KjvoobqGU1SwaEsj6wNpgpDbOtol8=; b=FQftJiFmPSpvrhvk9nTBEli3v0D2U1Rsdhigh43LmJOtju7RlC3c8QFX531Y4OyDwx w6ETu5456yvYZEyFp0kdgn8caWH9PFmU6hOLJYNjqd0Z3vUnOLs12/kf5UP64aV3HvzE D5qKz38qMiFedaBaIhyS4KhKCECA7BugziMy8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512305; x=1779117105; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=yv/dVX0kdD5IT4KjvoobqGU1SwaEsj6wNpgpDbOtol8=; b=QUcwBDaXbDfFoAanbl5l2DllYUmk89ShJkHYQaUVh1e3Ujv0Gyr40Q3VPBQq6UdYQ2 sns7wLFCYhppqTYQ5Vlc/ad6Pn7udtY/C+pDoFftcGjB3DM3RHtH++XBMcfRcqDuBk4G +88oD+718fMKul3AIm39VDWQhg3Tvj5G8Hhm8zwwQE5d65maoTmvJtpYqmKQkpYV4wOo Q+UrWIrJBH9GoT2eKNFrvrPYwdILOdXxUKLykWyH9ZTJbPf0QkvXtSN0YJm+4prjDGME sKoH6rRQ7w+dM/Tku9SL4NaQQ+gNipdeV8gIRfn69j9sDKNTs4stnIWo0EyIxCT0crpB Bdvw== X-Gm-Message-State: AOJu0YycK0+hNOB7jIcjoHKFDGxsDW1J9APCbI+PAm6BEySjpAY8Wr+L xUKVDdT237Vqp1+6+ZW3zajEPGC1Wtg79ncQuUi2Q7FHTeHNc6/stOjc+6uxu32KR8zS7h1/A4o 5/fiIUIQ= X-Gm-Gg: Acq92OFRR22q6WzkeBp9UCHtuYNlWiImJHq+C+LK+DcJACnnKkLodad+Evbp1xXtBt5 JxhiJ9mtNNfKugJL+U9UcoCmmjEboB2i9aYWRfaXAltvWiM6hQ4zB5LiTlfu4M+0+RVeAaCNMyT ma/D7UO8TRa8MAYZyyMzb9XM1L/QYNUY0cRS7w1q/WyeFc6KRhA3jbQGFklRmzHMvAyZbVIls6N gzJ6yEAEByyJTPK+WijmjMxre1kjqZ6QD9XHlqmUakpk4C58C2MavcfK0iP2HOfNw0fiD0Czm4b mF7fgwc2gDAdLqMP1J8xFBIibWuocm7YYoOcVZy+cREb2LcM6gxEybJUVfUhP41ZyN887dwcSAf inaTL9w7Nn2uMXIolyKvbe53LX34SnCZ57TyokNInvhLOYuqdhrzKH4REnCTDdk5OtmuLULPhym koC4MjQ5Ogce2M1K8u8kEd7K01Hgzy9yjGqx6CqaxnlGcrvEda+YPfnl73spcrenZ/s7LPQbZA+ NK80yWwwKDuiirjpjNk X-Received: by 2002:a05:600c:5248:b0:48a:906a:9050 with SMTP id 5b1f17b1804b1-48e7069175fmr167350045e9.10.1778512299862; Mon, 11 May 2026 08:11:39 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn , Klaus Jensen Subject: [PATCH v7 5/8] hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completion Date: Mon, 11 May 2026 17:11:28 +0200 Message-ID: <20260511151131.483217-6-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::430; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x430.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512415347154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Instead of unmapping req->sg in nvme_post_cqes(), we can do it earlier in nvme_enqueue_req_completion(). When req completion is enqueued we don't need to access req->sg anymore. We only care about req->sq, req->cqe and req->status. Reviewed-by: Klaus Jensen Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index ffee7ef224c..31c1da4a61e 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -1534,7 +1534,6 @@ static void nvme_post_cqes(void *opaque) QTAILQ_REMOVE(&cq->req_list, req, entry); =20 nvme_inc_cq_tail(cq); - nvme_sg_unmap(&req->sg); =20 if (QTAILQ_EMPTY(&sq->req_list) && !nvme_sq_empty(sq)) { qemu_bh_schedule(sq->bh); @@ -1564,6 +1563,8 @@ static void nvme_enqueue_req_completion(NvmeCQueue *c= q, NvmeRequest *req) req->status, req->cmd.opcode); } =20 + nvme_sg_unmap(&req->sg); + QTAILQ_REMOVE(&req->sq->out_req_list, req, entry); QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); =20 --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512448; cv=none; d=zohomail.com; s=zohoarc; b=hRYtOGXA/vVuFblVxaYGJI9V93jLxHD837+a9cOEVoC+zdhZVS/tDPiiUNYTaBszqgQ3Hx+kdFXRRZXqKMiJwOn5saZbyyG/nj58Qbz932hBrP3zdzmzQ075XksGo9EUOESQheFXExmat+tKIiuTy2THbs4T51qW5DRRGWB5dyc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512448; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=P6c4JQJF7847Wu/pKzcrmxMdVHLI89xuVASc0CK2Ia0=; b=Y3d0PZ5da/v+RJoszKZF4SKiKmbBlkDvnZsUrbLOz3sGP0YK+wV30Y1OR0R0LmcfCy6n+nzZaf8hJnD41ukj+duUj+Ki7eG2A9R0pGy2ybB0nKhLGnzPDUifIhy9Z+DNkcwvkTQkm3VkdYEwB9p/s3dO0N6tlmF6wSANoUWpq9c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512448329211.81477370133086; Mon, 11 May 2026 08:14:08 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJu-0007AH-Vp; Mon, 11 May 2026 11:13:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIh-0005u2-66 for qemu-devel@nongnu.org; Mon, 11 May 2026 11:12:09 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIP-0001rH-8u for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:53 -0400 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-488ad135063so36089765e9.0 for ; Mon, 11 May 2026 08:11:46 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512306; x=1779117106; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P6c4JQJF7847Wu/pKzcrmxMdVHLI89xuVASc0CK2Ia0=; b=hSEa0lfjh4Mppqx3NFvaoCz+K/yfgOIE0OfsfZ4H8xRE+gGG1DJWUci4RFLjQcaH7q COOXvvnodSu6txVUBaQIFslphgwQ359/1jaWlQEXWcdXKGVgxDHt+Wvfi+fjooQkoZfJ /se3IlEfruVD7YisXfHNbASP3bz7LC2mbPHW0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512306; x=1779117106; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=P6c4JQJF7847Wu/pKzcrmxMdVHLI89xuVASc0CK2Ia0=; b=g6+rpO+KvvvHuX0yDs++uQg8WlgkIXeFVGHOVvWrvNDcK/HJGT/4540iW+RrI/F6Np Lvz9hc35G+ZQF/GRen7M4ByqFEgAsxbiNbVxCBcx73eozUQ6OylaGlOGHUeJH/81FoJx 8RtXlqIc5iScsHn79TdvjS84ikVoOvppk4KF71ksUcl4HSZ6EG/et/SLNc73DcayibUC 2+5fyunD/Q6BQG4Sh+CanUgL2l0kHQ4cRkAoMWFgm3ShODqmbIS5ajx2rSMJODPKuBYl Al4PrinWvFWsVRQG3KfHmxTfnPRRr3GEQCjW7LhTcDUQeWccJz+f0Mi6s9Pn4jEoUvgi ffmA== X-Gm-Message-State: AOJu0Yxw+HsbxA+qx9TB5hWK5tFKTVZIM6lbMp4ftPbSKJ9CrEdgQfng cjZuD83PkrxwauDYrP0kI1G8CbH8y76DM1I7iL9HZeSLpOzOgH99p7/dZuMP/RzhC/mRuvpoN2t n5o/gBPM= X-Gm-Gg: Acq92OHZkudLAWqk239lcybd14iiGdZIJc680Yq2TQarTx2qYOYBP/tCpdF+NgZ8wCC 0dvYFmC8s0EQr9GYNu/jxSU2pacKIaBzAb9pidx916Zh2q0W40E10CfmlUHAING3XZE4DWw6A+6 fTRI8tMUrDki2deB2fr9slp4uW0+qGOJdWvq5xHQj1Uq+E2WY8rQn9OGT1EPmywegOVlxO7WW1a X+W4j50cZmBKekzfS6b/Yux/kSPU4bFb9ULB8KaMqdXkJVo4k6gTzob1+WeGRoaPL6amkJ7ZDfW yFytUpWrf+zeaNDY0BbvP1n9Iz1ado9HvQCNEu6ppstUX7em0r7vl9407emVPIbP1x0iPmdxIv2 JtqkjE9vbcv++1oqLK8SEDYJHLxHIP/4H0BHyWPeQnuY9SEQEznZRPVhxwfbyqBwPukiVpsyY6T nXEqAC25hWMCXa2iQSKTLi8x9W3q64Sf/H63a7FKu7oLI1k51BUbGDGO+WbAlgkOZCdmFoQhstV XSE+yxdgNzCPE7cNVfg X-Received: by 2002:a05:600c:6290:b0:485:3f30:6250 with SMTP id 5b1f17b1804b1-48e51f3fd58mr431754515e9.20.1778512301181; Mon, 11 May 2026 08:11:41 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn Subject: [PATCH v7 6/8] hw/nvme: add basic live migration support Date: Mon, 11 May 2026 17:11:29 +0200 Message-ID: <20260511151131.483217-7-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512450039154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn It has some limitations: - only one NVMe namespace is supported - SMART counters are not preserved - CMB is not supported - PMR is not supported - SPDM is not supported - SR-IOV is not supported Signed-off-by: Alexander Mikhalitsyn --- v2: - AERs are now fully supported v6: - handle full CQ case v7: - renamed copy_cq_req_list to move_cq_req_list - validate incoming migration stream better --- hw/nvme/ctrl.c | 737 ++++++++++++++++++++++++++++++++++++++++++- hw/nvme/ns.c | 160 ++++++++++ hw/nvme/nvme.h | 9 + hw/nvme/trace-events | 10 + 4 files changed, 907 insertions(+), 9 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 31c1da4a61e..ea7590da1f0 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -208,6 +208,7 @@ #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" #include "migration/blocker.h" +#include "migration/qemu-file-types.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -1518,6 +1519,18 @@ static void nvme_post_cqes(void *opaque) break; } =20 + /* + * Here we take the following fields from NvmeRequest structure + * and write cqe to the guest RAM based on them: + * - req->sq + * - req->status + * - req->cqe + * + * If you change this code and more fields from NvmeRequest are + * used, please make sure that you have handled this in: + * nvme_vmstate_request and nvme_ctrl_pre_save(). + */ + sq =3D req->sq; req->cqe.status =3D cpu_to_le16((req->status << 1) | cq->phase); req->cqe.sq_head =3D cpu_to_le16(sq->head); @@ -4903,6 +4916,25 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_sq(sq); } =20 +static void nvme_restore_sq(NvmeSQueue *sq_from) +{ + NvmeCtrl *n =3D sq_from->ctrl; + NvmeSQueue *sq =3D sq_from; + + if (sq_from->sqid =3D=3D 0) { + sq =3D &n->admin_sq; + sq->ctrl =3D n; + sq->dma_addr =3D sq_from->dma_addr; + sq->sqid =3D sq_from->sqid; + sq->size =3D sq_from->size; + sq->cqid =3D sq_from->cqid; + sq->head =3D sq_from->head; + sq->tail =3D sq_from->tail; + } + + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5605,6 +5637,39 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_cq(cq); } =20 +static void move_cq_req_list(NvmeCQueue *cq_to, NvmeCQueue *cq_from) +{ + NvmeRequest *req, *next; + + QTAILQ_FOREACH_SAFE(req, &cq_from->req_list, entry, next) { + QTAILQ_REMOVE(&cq_from->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq_to->req_list, req, entry); + } +} + +static void nvme_restore_cq(NvmeCQueue *cq_from) +{ + NvmeCtrl *n =3D cq_from->ctrl; + NvmeCQueue *cq =3D cq_from; + + if (cq_from->cqid =3D=3D 0) { + cq =3D &n->admin_cq; + cq->ctrl =3D n; + cq->cqid =3D cq_from->cqid; + cq->size =3D cq_from->size; + cq->dma_addr =3D cq_from->dma_addr; + cq->phase =3D cq_from->phase; + cq->irq_enabled =3D cq_from->irq_enabled; + cq->vector =3D cq_from->vector; + cq->head =3D cq_from->head; + cq->tail =3D cq_from->tail; + QTAILQ_INIT(&cq->req_list); + move_cq_req_list(cq, cq_from); + } + + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; @@ -7295,7 +7360,7 @@ static uint16_t nvme_dbbuf_config(NvmeCtrl *n, const = NvmeRequest *req) n->dbbuf_eis =3D eis_addr; n->dbbuf_enabled =3D true; =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { NvmeSQueue *sq =3D n->sq[i]; NvmeCQueue *cq =3D n->cq[i]; =20 @@ -7739,7 +7804,7 @@ static int nvme_atomic_write_check(NvmeCtrl *n, NvmeC= md *cmd, /* * Walk the queues to see if there are any atomic conflicts. */ - for (i =3D 1; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 1; i < n->num_queues; i++) { NvmeSQueue *sq; NvmeRequest *req; NvmeRwCmd *req_rw; @@ -7809,6 +7874,12 @@ static void nvme_process_sq(void *opaque) NvmeCmd cmd; NvmeRequest *req; =20 + /* + * We don't want to have a race with nvme_ctrl_pre_save(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + if (n->dbbuf_enabled) { nvme_update_sq_tail(sq); } @@ -7926,12 +7997,12 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) nvme_ns_drain(ns); } =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->sq[i] !=3D NULL) { nvme_free_sq(n->sq[i], n); } } - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->cq[i] !=3D NULL) { nvme_free_cq(n->cq[i], n); } @@ -8601,6 +8672,8 @@ static bool nvme_check_params(NvmeCtrl *n, Error **er= rp) params->max_ioqpairs =3D params->num_queues - 1; } =20 + n->num_queues =3D params->max_ioqpairs + 1; + if (n->namespace.blkconf.blk && n->subsys) { error_setg(errp, "subsystem support is unavailable with legacy " "namespace ('drive' property)"); @@ -8755,8 +8828,8 @@ static void nvme_init_state(NvmeCtrl *n) n->conf_msix_qsize =3D n->params.msix_qsize; } =20 - n->sq =3D g_new0(NvmeSQueue *, n->params.max_ioqpairs + 1); - n->cq =3D g_new0(NvmeCQueue *, n->params.max_ioqpairs + 1); + n->sq =3D g_new0(NvmeSQueue *, n->num_queues); + n->cq =3D g_new0(NvmeCQueue *, n->num_queues); n->temperature =3D NVME_TEMPERATURE; n->features.temp_thresh_hi =3D NVME_TEMPERATURE_WARNING; n->starttime_ms =3D qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); @@ -8991,7 +9064,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) } =20 if (n->params.msix_exclusive_bar && !pci_is_vf(pci_dev)) { - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, 0, NULL, N= ULL); + bar_size =3D nvme_mbar_size(n->num_queues, 0, NULL, NULL); memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n, "nv= me", bar_size); pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | @@ -9003,7 +9076,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) /* add one to max_ioqpairs to account for the admin queue pair */ if (!pci_is_vf(pci_dev)) { nr_vectors =3D n->params.msix_qsize; - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, + bar_size =3D nvme_mbar_size(n->num_queues, nr_vectors, &msix_table_offset, &msix_pba_offset); } else { @@ -9726,9 +9799,655 @@ static uint32_t nvme_pci_read_config(PCIDevice *dev= , uint32_t address, int len) return pci_default_read_config(dev, address, len); } =20 +static const VMStateDescription nvme_vmstate_cqe =3D { + .name =3D "nvme-cqe", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(result, NvmeCqe), + VMSTATE_UINT32(dw1, NvmeCqe), + VMSTATE_UINT16(sq_head, NvmeCqe), + VMSTATE_UINT16(sq_id, NvmeCqe), + VMSTATE_UINT16(cid, NvmeCqe), + VMSTATE_UINT16(status, NvmeCqe), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr_sgl =3D { + .name =3D "nvme-request-cmd-dptr-sgl", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(addr, NvmeSglDescriptor), + VMSTATE_UINT32(len, NvmeSglDescriptor), + VMSTATE_UINT8_ARRAY(rsvd, NvmeSglDescriptor, 3), + VMSTATE_UINT8(type, NvmeSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr =3D { + .name =3D "nvme-request-cmd-dptr", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(prp1, NvmeCmdDptr), + VMSTATE_UINT64(prp2, NvmeCmdDptr), + VMSTATE_STRUCT(sgl, NvmeCmdDptr, 0, nvme_vmstate_cmd_dptr_sgl, Nvm= eSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd =3D { + .name =3D "nvme-request-cmd", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(opcode, NvmeCmd), + VMSTATE_UINT8(flags, NvmeCmd), + VMSTATE_UINT16(cid, NvmeCmd), + VMSTATE_UINT32(nsid, NvmeCmd), + VMSTATE_UINT64(res1, NvmeCmd), + VMSTATE_UINT64(mptr, NvmeCmd), + VMSTATE_STRUCT(dptr, NvmeCmd, 0, nvme_vmstate_cmd_dptr, NvmeCmdDpt= r), + VMSTATE_UINT32(cdw10, NvmeCmd), + VMSTATE_UINT32(cdw11, NvmeCmd), + VMSTATE_UINT32(cdw12, NvmeCmd), + VMSTATE_UINT32(cdw13, NvmeCmd), + VMSTATE_UINT32(cdw14, NvmeCmd), + VMSTATE_UINT32(cdw15, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static bool nvme_req_pre_load(void *opaque, Error **errp) +{ + memset(opaque, 0x0, sizeof(NvmeRequest)); + return true; +} + +static const VMStateDescription nvme_vmstate_request =3D { + .name =3D "nvme-request", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_req_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(status, NvmeRequest), + VMSTATE_STRUCT(cqe, NvmeRequest, 0, nvme_vmstate_cqe, NvmeCqe), + VMSTATE_STRUCT(cmd, NvmeRequest, 0, nvme_vmstate_cmd, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_bar =3D { + .name =3D "nvme-bar", + .minimum_version_id =3D 1, + .version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(cap, NvmeBar), + VMSTATE_UINT32(vs, NvmeBar), + VMSTATE_UINT32(intms, NvmeBar), + VMSTATE_UINT32(intmc, NvmeBar), + VMSTATE_UINT32(cc, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd24, NvmeBar, 4), + VMSTATE_UINT32(csts, NvmeBar), + VMSTATE_UINT32(nssr, NvmeBar), + VMSTATE_UINT32(aqa, NvmeBar), + VMSTATE_UINT64(asq, NvmeBar), + VMSTATE_UINT64(acq, NvmeBar), + VMSTATE_UINT32(cmbloc, NvmeBar), + VMSTATE_UINT32(cmbsz, NvmeBar), + VMSTATE_UINT32(bpinfo, NvmeBar), + VMSTATE_UINT32(bprsel, NvmeBar), + VMSTATE_UINT64(bpmbl, NvmeBar), + VMSTATE_UINT64(cmbmsc, NvmeBar), + VMSTATE_UINT32(cmbsts, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd92, NvmeBar, 3492), + VMSTATE_UINT32(pmrcap, NvmeBar), + VMSTATE_UINT32(pmrctl, NvmeBar), + VMSTATE_UINT32(pmrsts, NvmeBar), + VMSTATE_UINT32(pmrebs, NvmeBar), + VMSTATE_UINT32(pmrswtp, NvmeBar), + VMSTATE_UINT32(pmrmscl, NvmeBar), + VMSTATE_UINT32(pmrmscu, NvmeBar), + VMSTATE_UINT8_ARRAY(css, NvmeBar, 484), + VMSTATE_END_OF_LIST() + }, +}; + +static bool nvme_cqueue_pre_load(void *opaque, Error **errp) +{ + NvmeCQueue *cq =3D opaque; + + QTAILQ_INIT(&cq->req_list); + return true; +} + +static const VMStateDescription nvme_vmstate_cqueue =3D { + .name =3D "nvme-cq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .pre_load_errp =3D nvme_cqueue_pre_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(phase, NvmeCQueue), + VMSTATE_UINT16(cqid, NvmeCQueue), + VMSTATE_UINT16(irq_enabled, NvmeCQueue), + VMSTATE_UINT32(head, NvmeCQueue), + VMSTATE_UINT32(tail, NvmeCQueue), + VMSTATE_UINT32(vector, NvmeCQueue), + VMSTATE_UINT32(size, NvmeCQueue), + VMSTATE_UINT64(dma_addr, NvmeCQueue), + + VMSTATE_QTAILQ_V(req_list, NvmeCQueue, 1, nvme_vmstate_request, + NvmeRequest, entry), + + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_squeue =3D { + .name =3D "nvme-sq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(sqid, NvmeSQueue), + VMSTATE_UINT16(cqid, NvmeSQueue), + VMSTATE_UINT32(head, NvmeSQueue), + VMSTATE_UINT32(tail, NvmeSQueue), + VMSTATE_UINT32(size, NvmeSQueue), + VMSTATE_UINT64(dma_addr, NvmeSQueue), + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event_result =3D { + .name =3D "nvme-async-event-result", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(event_type, NvmeAerResult), + VMSTATE_UINT8(event_info, NvmeAerResult), + VMSTATE_UINT8(log_page, NvmeAerResult), + VMSTATE_UINT8(resv, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event =3D { + .name =3D "nvme-async-event", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_STRUCT(result, NvmeAsyncEvent, 0, nvme_vmstate_async_event= _result, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_hbs =3D { + .name =3D "nvme-hbs", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(acre, NvmeHostBehaviorSupport), + VMSTATE_UINT8(etdas, NvmeHostBehaviorSupport), + VMSTATE_UINT8(lbafee, NvmeHostBehaviorSupport), + VMSTATE_UINT8(rsvd3, NvmeHostBehaviorSupport), + VMSTATE_UINT16(cdfe, NvmeHostBehaviorSupport), + VMSTATE_UINT8_ARRAY(rsvd6, NvmeHostBehaviorSupport, 506), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_atomic =3D { + .name =3D "nvme-atomic", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(atomic_max_write_size, NvmeAtomic), + VMSTATE_UINT64(atomic_boundary, NvmeAtomic), + VMSTATE_UINT64(atomic_nabo, NvmeAtomic), + VMSTATE_BOOL(atomic_writes, NvmeAtomic), + VMSTATE_END_OF_LIST() + } +}; + +static bool pre_save_validate_aer_req(NvmeRequest *req, Error **errp) +{ + /* + * Can't use assert() here, because we don't want + * to just crash QEMU when user requests a migration. + */ + if (!(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ)) { + error_setg(errp, "req->cmd.opcode (%u) !=3D NVME_ADM_CMD_ASYNC_EV_= REQ", req->cmd.opcode); + return false; + } + + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->sq =3D=3D &req->sq->ctrl->admin_sq)) { + error_setg(errp, "req->sq !=3D &req->sq->ctrl->admin_sq"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool pre_save_validate_cq_req(NvmeRequest *req, Error **errp) +{ + if (!(req->ns =3D=3D NULL)) { + error_setg(errp, "req->ns !=3D NULL"); + return false; + } + + if (!(req->aiocb =3D=3D NULL)) { + error_setg(errp, "req->aiocb !=3D NULL"); + return false; + } + + if (!(req->opaque =3D=3D NULL)) { + error_setg(errp, "req->opaque !=3D NULL"); + return false; + } + + if (!(req->atomic_write =3D=3D false)) { + error_setg(errp, "req->atomic_write !=3D false"); + return false; + } + + if (req->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in req->sg.flags"); + return false; + } + + return true; +} + +static bool nvme_ctrl_pre_save(void *opaque, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_pre_save_enter(n); + + /* + * We don't want to have a race with nvme_process_sq(). + * What implicitly protects us from this is BQL. + */ + assert(bql_locked()); + + /* cancel all SQ processing BHs */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_cancel(sq->bh); + } + + /* drain all IO */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns; + + ns =3D nvme_ns(n, i); + if (!ns) { + continue; + } + + trace_pci_nvme_pre_save_ns_drain(n, i); + nvme_ns_drain(ns); + } + + /* + * Now, we should take care of AERs. + * + * 1. Save all queued events (n->aer_queue). + * This is done automatically, see nvme_vmstate VMStateDescription. + * Here we only need to print them for debugging purpose. + * 2. Go over outstanding AER requests (n->aer_reqs) and check they are + * all have expected opcode (NVME_ADM_CMD_ASYNC_EV_REQ) and other f= ields. + * + * We must be really careful here, because in case of further QEMU NVM= e changes, + * we may break migration without noticing it, or worse, introduce sil= ent + * data corruption during migration. + */ + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_pre_save_aer(event->result.event_type, event->r= esult.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeRequest *req =3D n->aer_reqs[i]; + + if (!pre_save_validate_aer_req(req, errp)) { + return false; + } + } + + /* make sure that all in-flight IO requests (except NVME_ADM_CMD_ASYNC= _EV_REQ) are processed */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req; + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + trace_pci_nvme_pre_save_sq_out_req_check(n, i, sq->head, sq->tail,= sq->size); + + QTAILQ_FOREACH(req, &sq->out_req_list, entry) { + assert(req->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ); + } + } + + /* wait when all IO requests completions are written to guest memory */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + qemu_bh_cancel(cq->bh); + /* this should empty cq->req_list unless CQ is full */ + nvme_post_cqes(cq); + + trace_pci_nvme_pre_save_cq_req_check(n, i, cq->head, cq->tail, cq-= >size); + + if (!QTAILQ_EMPTY(&cq->req_list)) { + NvmeRequest *req; + + assert(nvme_cq_full(cq)); + + QTAILQ_FOREACH(req, &cq->req_list, entry) { + trace_pci_nvme_pre_save_cq_unposted_cqe(n, i, nvme_cid(req= ), + nvme_nsid(req->ns), + le32_to_cpu(req->c= qe.result), + le32_to_cpu(req->c= qe.dw1), + req->status, req->= cmd.opcode); + if (!pre_save_validate_cq_req(req, errp)) { + return false; + } + } + } + } + + for (uint32_t nsid =3D 0; nsid <=3D NVME_MAX_NAMESPACES; nsid++) { + NvmeNamespace *ns =3D n->namespaces[nsid]; + + if (!ns) + continue; + + if (ns !=3D &n->namespace) { + error_setg(errp, "only one NVMe namespace is supported for mig= ration"); + return false; + } + } + + return true; +} + +static bool nvme_ctrl_post_load(void *opaque, int version_id, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_post_load_enter(n); + + /* restore CQs first */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + if (cq->cqid !=3D i) { + error_setg(errp, "inconsistent migration stream (cq->cqid !=3D= i)"); + return false; + } + + cq->ctrl =3D n; + nvme_restore_cq(cq); + trace_pci_nvme_post_load_restore_cq(n, i, cq->head, cq->tail, cq->= size); + + if (i =3D=3D 0) { + /* + * Admin CQ lives in n->admin_cq, we don't need + * memory allocated for it in get_ptrs_array_entry() anymore. + * + * nvme_restore_cq() also takes care of: + * n->cq[0] =3D &n->admin_cq; + * so n->cq[0] remains valid. + */ + g_free(cq); + } + } + + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + if (sq->sqid !=3D i) { + error_setg(errp, "inconsistent migration stream (sq->sqid !=3D= i)"); + return false; + } + + if (!n->cq[sq->cqid]) { + error_setg(errp, "inconsistent migration stream (n->cq[sq->cqi= d] is NULL)"); + return false; + } + + sq->ctrl =3D n; + nvme_restore_sq(sq); + trace_pci_nvme_post_load_restore_sq(n, i, sq->head, sq->tail, sq->= size); + + if (i =3D=3D 0) { + /* same as for CQ */ + g_free(sq); + } + } + + /* restore cq->req_list-s */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req_from, *next; + typeof_field(NvmeCQueue, req_list) req_list; + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq || QTAILQ_EMPTY(&cq->req_list)) + continue; + + /* + * We use nvme_vmstate_request VMStateDescription to save/restore + * NvmeRequest structures, but tricky thing here is that + * memory for each cq->req_list item is allocated separately + * during restore. It doesn't work for us. We need to take + * an existing NvmeRequest structure from SQ's req_list pool + * and fill it with data from the newly allocated one (req_from). + * Then, we can safely release allocated memory for it. + */ + + /* make a copy of cq->req_list (QTAILQ head) and clean cq->req_lis= t */ + QTAILQ_INIT(&req_list); + QTAILQ_FOREACH_SAFE(req_from, &cq->req_list, entry, next) { + QTAILQ_REMOVE(&cq->req_list, req_from, entry); + QTAILQ_INSERT_TAIL(&req_list, req_from, entry); + } + QTAILQ_INIT(&cq->req_list); + + QTAILQ_FOREACH_SAFE(req_from, &req_list, entry, next) { + uint16_t sqid =3D le16_to_cpu(req_from->cqe.sq_id); + NvmeRequest *req; + NvmeSQueue *sq; + + assert(!nvme_check_sqid(n, sqid)); + sq =3D n->sq[sqid]; + + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&cq->req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + QTAILQ_REMOVE(&req_list, req_from, entry); + g_free(req_from); + } + + qemu_bh_schedule(cq->bh); + } + + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_post_load_aer(event->result.event_type, event->= result.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeSQueue *sq =3D &n->admin_sq; + NvmeRequest *req_from =3D n->aer_reqs[i]; + NvmeRequest *req; + + /* Idea here is the same as for "restore cq->req_list-s" step */ + + /* take an NvmeRequest struct from SQ */ + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + n->aer_reqs[i] =3D req; + g_free(req_from); + } + + /* + * We need to attach namespaces (currently, only one namespace is + * supported for migration). + * This logic comes from nvme_start_ctrl(). + */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + + if (!ns || (!ns->params.shared && ns->ctrl !=3D n)) { + continue; + } + + if (nvme_csi_supported(n, ns->csi) && !ns->params.detached) { + if (!ns->attached || ns->params.shared) { + nvme_attach_ns(n, ns); + } + } + } + + /* schedule SQ processing */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_schedule(sq->bh); + } + + return true; +} + static const VMStateDescription nvme_vmstate =3D { .name =3D "nvme", - .unmigratable =3D 1, + .minimum_version_id =3D 1, + .version_id =3D 1, + .pre_save_errp =3D nvme_ctrl_pre_save, + .post_load_errp =3D nvme_ctrl_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_PCI_DEVICE(parent_obj, NvmeCtrl), + VMSTATE_MSIX(parent_obj, NvmeCtrl), + VMSTATE_STRUCT(bar, NvmeCtrl, 0, nvme_vmstate_bar, NvmeBar), + + VMSTATE_BOOL(qs_created, NvmeCtrl), + VMSTATE_UINT32(page_size, NvmeCtrl), + VMSTATE_UINT16(page_bits, NvmeCtrl), + VMSTATE_UINT16(max_prp_ents, NvmeCtrl), + VMSTATE_UINT32(max_q_ents, NvmeCtrl), + VMSTATE_UINT8(outstanding_aers, NvmeCtrl), + VMSTATE_UINT32(irq_status, NvmeCtrl), + VMSTATE_INT32(cq_pending, NvmeCtrl), + + VMSTATE_UINT64(host_timestamp, NvmeCtrl), + VMSTATE_UINT64(timestamp_set_qemu_clock_ms, NvmeCtrl), + VMSTATE_UINT64(starttime_ms, NvmeCtrl), + VMSTATE_UINT16(temperature, NvmeCtrl), + VMSTATE_UINT8(smart_critical_warning, NvmeCtrl), + + VMSTATE_UINT32(conf_msix_qsize, NvmeCtrl), + VMSTATE_UINT32(conf_ioqpairs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_dbs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_eis, NvmeCtrl), + VMSTATE_BOOL(dbbuf_enabled, NvmeCtrl), + + VMSTATE_UINT8(aer_mask, NvmeCtrl), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC( + aer_reqs, NvmeCtrl, outstanding_aers, 0, nvme_vmstate_request,= NvmeRequest), + VMSTATE_QTAILQ_V(aer_queue, NvmeCtrl, 1, nvme_vmstate_async_event, + NvmeAsyncEvent, entry), + VMSTATE_INT32(aer_queued, NvmeCtrl), + + VMSTATE_STRUCT(namespace, NvmeCtrl, 0, nvme_vmstate_ns, NvmeNamesp= ace), + + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + sq, NvmeCtrl, num_queues, 0, nvme_vmstate_squeue, NvmeSQueue), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + cq, NvmeCtrl, num_queues, 0, nvme_vmstate_cqueue, NvmeCQueue), + + VMSTATE_UINT16(features.temp_thresh_hi, NvmeCtrl), + VMSTATE_UINT16(features.temp_thresh_low, NvmeCtrl), + VMSTATE_UINT32(features.async_config, NvmeCtrl), + VMSTATE_STRUCT(features.hbs, NvmeCtrl, 0, nvme_vmstate_hbs, NvmeHo= stBehaviorSupport), + + VMSTATE_UINT32(dn, NvmeCtrl), + VMSTATE_STRUCT(atomic, NvmeCtrl, 0, nvme_vmstate_atomic, NvmeAtomi= c), + + VMSTATE_END_OF_LIST() + }, }; =20 static void nvme_class_init(ObjectClass *oc, const void *data) diff --git a/hw/nvme/ns.c b/hw/nvme/ns.c index b0106eaa5c8..3e91ac9eab1 100644 --- a/hw/nvme/ns.c +++ b/hw/nvme/ns.c @@ -20,6 +20,7 @@ #include "qemu/bitops.h" #include "system/system.h" #include "system/block-backend.h" +#include "migration/vmstate.h" =20 #include "nvme.h" #include "trace.h" @@ -886,6 +887,164 @@ static void nvme_ns_realize(DeviceState *dev, Error *= *errp) } } =20 +static const VMStateDescription nvme_vmstate_lbaf =3D { + .name =3D "nvme_lbaf", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(ms, NvmeLBAF), + VMSTATE_UINT8(ds, NvmeLBAF), + VMSTATE_UINT8(rp, NvmeLBAF), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns =3D { + .name =3D "nvme_id_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(nsze, NvmeIdNs), + VMSTATE_UINT64(ncap, NvmeIdNs), + VMSTATE_UINT64(nuse, NvmeIdNs), + VMSTATE_UINT8(nsfeat, NvmeIdNs), + VMSTATE_UINT8(nlbaf, NvmeIdNs), + VMSTATE_UINT8(flbas, NvmeIdNs), + VMSTATE_UINT8(mc, NvmeIdNs), + VMSTATE_UINT8(dpc, NvmeIdNs), + VMSTATE_UINT8(dps, NvmeIdNs), + VMSTATE_UINT8(nmic, NvmeIdNs), + VMSTATE_UINT8(rescap, NvmeIdNs), + VMSTATE_UINT8(fpi, NvmeIdNs), + VMSTATE_UINT8(dlfeat, NvmeIdNs), + VMSTATE_UINT16(nawun, NvmeIdNs), + VMSTATE_UINT16(nawupf, NvmeIdNs), + VMSTATE_UINT16(nacwu, NvmeIdNs), + VMSTATE_UINT16(nabsn, NvmeIdNs), + VMSTATE_UINT16(nabo, NvmeIdNs), + VMSTATE_UINT16(nabspf, NvmeIdNs), + VMSTATE_UINT16(noiob, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nvmcap, NvmeIdNs, 16), + VMSTATE_UINT16(npwg, NvmeIdNs), + VMSTATE_UINT16(npwa, NvmeIdNs), + VMSTATE_UINT16(npdg, NvmeIdNs), + VMSTATE_UINT16(npda, NvmeIdNs), + VMSTATE_UINT16(nows, NvmeIdNs), + VMSTATE_UINT16(mssrl, NvmeIdNs), + VMSTATE_UINT32(mcl, NvmeIdNs), + VMSTATE_UINT8(msrc, NvmeIdNs), + VMSTATE_UINT8_ARRAY(rsvd81, NvmeIdNs, 18), + VMSTATE_UINT8(nsattr, NvmeIdNs), + VMSTATE_UINT16(nvmsetid, NvmeIdNs), + VMSTATE_UINT16(endgid, NvmeIdNs), + VMSTATE_UINT8_ARRAY(nguid, NvmeIdNs, 16), + VMSTATE_UINT64(eui64, NvmeIdNs), + VMSTATE_STRUCT_ARRAY(lbaf, NvmeIdNs, NVME_MAX_NLBAF, 1, + nvme_vmstate_lbaf, NvmeLBAF), + VMSTATE_UINT8_ARRAY(vs, NvmeIdNs, 3712), + + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_nvm =3D { + .name =3D "nvme_id_ns_nvm", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(lbstm, NvmeIdNsNvm), + VMSTATE_UINT8(pic, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd9, NvmeIdNsNvm, 3), + VMSTATE_UINT32_ARRAY(elbaf, NvmeIdNsNvm, NVME_MAX_NLBAF), + VMSTATE_UINT32(npdgl, NvmeIdNsNvm), + VMSTATE_UINT32(nprg, NvmeIdNsNvm), + VMSTATE_UINT32(npra, NvmeIdNsNvm), + VMSTATE_UINT32(nors, NvmeIdNsNvm), + VMSTATE_UINT32(npdal, NvmeIdNsNvm), + VMSTATE_UINT8_ARRAY(rsvd288, NvmeIdNsNvm, 3808), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_id_ns_ind =3D { + .name =3D "nvme_id_ns_ind", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(nsfeat, NvmeIdNsInd), + VMSTATE_UINT8(nmic, NvmeIdNsInd), + VMSTATE_UINT8(rescap, NvmeIdNsInd), + VMSTATE_UINT8(fpi, NvmeIdNsInd), + VMSTATE_UINT32(anagrpid, NvmeIdNsInd), + VMSTATE_UINT8(nsattr, NvmeIdNsInd), + VMSTATE_UINT8(rsvd9, NvmeIdNsInd), + VMSTATE_UINT16(nvmsetid, NvmeIdNsInd), + VMSTATE_UINT16(endgrpid, NvmeIdNsInd), + VMSTATE_UINT8(nstat, NvmeIdNsInd), + VMSTATE_UINT8_ARRAY(rsvd15, NvmeIdNsInd, 4081), + VMSTATE_END_OF_LIST() + } +}; + +typedef struct TmpNvmeNamespace { + NvmeNamespace *parent; + bool enable_write_cache; +} TmpNvmeNamespace; + +static bool nvme_ns_tmp_pre_save(void *opaque, Error **errp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + tns->enable_write_cache =3D blk_enable_write_cache(tns->parent->blkcon= f.blk); + + return true; +} + +static bool nvme_ns_tmp_post_load(void *opaque, int version_id, Error **er= rp) +{ + struct TmpNvmeNamespace *tns =3D opaque; + + blk_set_enable_write_cache(tns->parent->blkconf.blk, tns->enable_write= _cache); + + return true; +} + +static const VMStateDescription nvme_vmstate_ns_tmp =3D { + .name =3D "nvme_ns_tmp", + .pre_save_errp =3D nvme_ns_tmp_pre_save, + .post_load_errp =3D nvme_ns_tmp_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_BOOL(enable_write_cache, TmpNvmeNamespace), + VMSTATE_END_OF_LIST() + } +}; + +const VMStateDescription nvme_vmstate_ns =3D { + .name =3D "nvme_ns", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_WITH_TMP(NvmeNamespace, TmpNvmeNamespace, nvme_vmstate_ns_= tmp), + + VMSTATE_STRUCT(id_ns, NvmeNamespace, 0, nvme_vmstate_id_ns, NvmeId= Ns), + VMSTATE_STRUCT(id_ns_nvm, NvmeNamespace, 0, nvme_vmstate_id_ns_nvm= , NvmeIdNsNvm), + VMSTATE_STRUCT(id_ns_ind, NvmeNamespace, 0, nvme_vmstate_id_ns_ind= , NvmeIdNsInd), + VMSTATE_STRUCT(lbaf, NvmeNamespace, 0, nvme_vmstate_lbaf, NvmeLBAF= ), + VMSTATE_UINT32(nlbaf, NvmeNamespace), + VMSTATE_UINT8(csi, NvmeNamespace), + VMSTATE_UINT16(status, NvmeNamespace), + VMSTATE_UINT8(pif, NvmeNamespace), + + VMSTATE_UINT16(zns.zrwas, NvmeNamespace), + VMSTATE_UINT16(zns.zrwafg, NvmeNamespace), + VMSTATE_UINT32(zns.numzrwa, NvmeNamespace), + + VMSTATE_UINT32(features.err_rec, NvmeNamespace), + VMSTATE_STRUCT(atomic, NvmeNamespace, 0, nvme_vmstate_atomic, Nvme= Atomic), + VMSTATE_END_OF_LIST() + } +}; + static const Property nvme_ns_props[] =3D { DEFINE_BLOCK_PROPERTIES(NvmeNamespace, blkconf), DEFINE_PROP_BOOL("detached", NvmeNamespace, params.detached, false), @@ -937,6 +1096,7 @@ static void nvme_ns_class_init(ObjectClass *oc, const = void *data) dc->bus_type =3D TYPE_NVME_BUS; dc->realize =3D nvme_ns_realize; dc->unrealize =3D nvme_ns_unrealize; + dc->vmsd =3D &nvme_vmstate_ns; device_class_set_props(dc, nvme_ns_props); dc->desc =3D "Virtual NVMe namespace"; } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 457b6637249..2e7597cded3 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -444,6 +444,11 @@ typedef struct NvmeRequest { NvmeSg sg; bool atomic_write; QTAILQ_ENTRY(NvmeRequest)entry; + /* + * If you add a new field here, please make sure to update + * nvme_vmstate_request, pre_save_validate_aer_req() and + * pre_save_validate_cq_req(). + */ } NvmeRequest; =20 typedef struct NvmeBounceContext { @@ -638,6 +643,7 @@ typedef struct NvmeCtrl { =20 NvmeNamespace namespace; NvmeNamespace *namespaces[NVME_MAX_NAMESPACES + 1]; + uint32_t num_queues; NvmeSQueue **sq; NvmeCQueue **cq; NvmeSQueue admin_sq; @@ -749,4 +755,7 @@ void nvme_atomic_configure_max_write_size(bool dn, uint= 16_t awun, void nvme_ns_atomic_configure_boundary(bool dn, uint16_t nabsn, uint16_t nabspf, NvmeAtomic *atomic= ); =20 +extern const VMStateDescription nvme_vmstate_atomic; +extern const VMStateDescription nvme_vmstate_ns; + #endif /* HW_NVME_NVME_H */ diff --git a/hw/nvme/trace-events b/hw/nvme/trace-events index 6be0bfa1c1f..f97a6a11f36 100644 --- a/hw/nvme/trace-events +++ b/hw/nvme/trace-events @@ -7,6 +7,16 @@ pci_nvme_dbbuf_config(uint64_t dbs_addr, uint64_t eis_addr= ) "dbs_addr=3D0x%"PRIx64 pci_nvme_map_addr(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %"PRI= u64"" pci_nvme_map_addr_cmb(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %= "PRIu64"" pci_nvme_map_prp(uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t= prp2, int num_prps) "trans_len %"PRIu64" len %"PRIu32" prp1 0x%"PRIx64" pr= p2 0x%"PRIx64" num_prps %d" +pci_nvme_pre_save_enter(void *n) "n=3D%p" +pci_nvme_pre_save_ns_drain(void *n, int i) "n=3D%p i=3D%d" +pci_nvme_pre_save_sq_out_req_check(void *n, int i, uint32_t head, uint32_t= tail, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" = size=3D0x%"PRIx32"" +pci_nvme_pre_save_cq_req_check(void *n, int i, uint32_t head, uint32_t tai= l, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_pre_save_cq_unposted_cqe(void *n, int i, uint16_t cid, uint32_t n= sid, uint32_t dw0, uint32_t dw1, uint16_t status, uint8_t opc) "n=3D%p i=3D= %d cid %"PRIu16" nsid %"PRIu32" dw0 0x%"PRIx32" dw1 0x%"PRIx32" status 0x%"= PRIx16" opc 0x%"PRIx8"" +pci_nvme_pre_save_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type 0= x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" +pci_nvme_post_load_enter(void *n) "n=3D%p" +pci_nvme_post_load_restore_cq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_restore_sq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type = 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" pci_nvme_map_sgl(uint8_t typ, uint64_t len) "type 0x%"PRIx8" len %"PRIu64"" pci_nvme_io_cmd(uint16_t cid, uint32_t nsid, uint16_t sqid, uint8_t opcode= , const char *opname) "cid %"PRIu16" nsid 0x%"PRIx32" sqid %"PRIu16" opc 0x= %"PRIx8" opname '%s'" pci_nvme_admin_cmd(uint16_t cid, uint16_t sqid, uint8_t opcode, const char= *opname) "cid %"PRIu16" sqid %"PRIu16" opc 0x%"PRIx8" opname '%s'" --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512405; cv=none; d=zohomail.com; s=zohoarc; b=jq8Jo6touDZlNrPcl3V73sc+ucItHq2Bbol6YlsUfy8MZfwync3gDbtZdKBuyteiGqg1Pz6aptmZEz8oeNksZG84ALMg9xiDlyf8QiIa0lio3ukCn/HpiWnLYAM+r+JJVCOw6pS6o3F7OHMJ/lq8Vve7WNe/OhV3zEJ1hnJmFZU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512405; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=Dcvy+PC0uzF6rnrCizpYJCxMnObVtebMYbbX3sHTtykGZsPIcqIMiXK79p2QFhugb48PmgtUkNxrCaOY6YlEPEv/G2LeRdswe+YnKRUB2pPLpQGDWH9K9xT+gXJREGpPJPgm8y3bHE0co5gyjAc2jmK9DGO0qJW4YDIMHeMVy4k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512405526847.5419006108066; Mon, 11 May 2026 08:13:25 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJe-0006NE-Lp; Mon, 11 May 2026 11:13:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIR-0005qN-DE for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:53 -0400 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIP-0001rS-5p for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:51 -0400 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-48896199cbaso40730775e9.1 for ; Mon, 11 May 2026 08:11:48 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512307; x=1779117107; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=DRPLWYuxyPz3o+ZcBBxCw57c06OTm/xVmjq6u82O+aoHLcblR2gg5IuRdtkJWdW058 Z7TyRVe2f01yrNr19VPknXHaGwM32cWhUk1dayzsW4G/7x8iUrPYpfYyG+sp9nKyIm3F SVamW+vBfffa08a4Gtn76YJFkUjFvS4TxCD7M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512307; x=1779117107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=sBpcKOoEk+DQ0qOwh5vSLzEuS5T0qQxiJFTEUttDnck=; b=mUQn8KrkasZnjMZexl5uqEJ6+kyJiaZDMUlv1FbUN4CqLnlSLeTCKUdcuc8pWHSjbi WX66W4/FrdL6RNNDked1JrDn0yJcnWM6T2FZbK2GTP6wZKyu8/myDFOd/X99DgskqEBN SrBQLkvVZQHozatCzf+ioAPCm2hrZEXycj11x9UjQRsSDLDyZnp7wU9Rp6Msb0vc9u+B OHV5Z2AIbcjfgLdO6CZj9a61ABN9jB4rj3xVMayDyDga/PRIMaMlVy2lt0j9W5kY8Tch GXYpjfp3nEJ7xtqWJtedOhZcSyIW3sC82V2am0LwwPXs/qyPTuDkZh4r6+FpuNZpPA4I 2tBw== X-Gm-Message-State: AOJu0YyKX+N8UmGMx1HDra81kDXe2ijyFWxJUkhAXV1fat4NOOjhGnpg 9P0ZDG8pDRfL8EDMWaWEHIIaI0f2T6tZrxgudBJ7HZ8Sdw74a/L9JiwGEW6UygrFRULxc1frI+3 6nwLf1mw= X-Gm-Gg: Acq92OETWyVwrXTFmTyS9g5FVDVOj42EluFwxkspLrPKA4n6pVahxsy1G2zxSwNoy19 hImdc+mb80aqT7Ami2Kx5CJsMmpUqDXJvSPrEo5iC26/RrV7NOH2As73bBu3kySdw+V2fnKgkAG fkwUDC0CtDOsnJPEaqeHaGDzme5j3yKBo3tziEHzKCTp0xgfhR/gQZg0zEkrnRSu0q/ynbnnbbi a4793woujY3rAoMiMHMRsXiA8Xl0FAQHhWafvZkTmJUNM/8uEaXDQ/bLKajfzG65z+Vm6VreZ4L HOxZSleSvnvZ8mN9EwINV8pSb+ngveGlctryTDsGufpk8pOmn7wSOfAZvvlgEW6jE+r2Cn9rj8u Y8OB7b5aLK5Cv8HafNOml3v04QMuB4BFfjaRLfedx1SMfomL7ITPY6Z2udciHIFK452dr0vnq1v U2zB4xpAdmhw4/gOqUvnwara2wxScjvKS3KZrhxKd4LxWHt3G4tMKrroEBO0KZeMXSB9KDwVJ5a aR2oUu14Hk7AXt3hNCA X-Received: by 2002:a05:600c:4e43:b0:485:30d4:6b9e with SMTP id 5b1f17b1804b1-48e51f4c1e5mr419887325e9.21.1778512302278; Mon, 11 May 2026 08:11:42 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn Subject: [PATCH v7 7/8] tests/functional/x86_64: add migration test for NVMe device Date: Mon, 11 May 2026 17:11:30 +0200 Message-ID: <20260511151131.483217-8-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32d; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512407812154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce a very simple test to ensure that NVMe device migration works fine. Test plan is simple: 1. prepare VM with NVMe device 2. run workload that produces relatively heavy IO on the device 3. migrate VM 4. ensure that workload is alive and finishes without errors Test can be run as simple as: $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build In the future we can extend this approach, and introduce some fio-based tests. And probably, it makes sense to make this test to apply not only to NVMe device, but also virtio-{blk,scsi}, ide, sata and other migratable devices. Signed-off-by: Alexander Mikhalitsyn --- tests/functional/x86_64/meson.build | 1 + .../functional/x86_64/test_nvme_migration.py | 159 ++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100755 tests/functional/x86_64/test_nvme_migration.py diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/= meson.build index 1ed10ad6c29..fd77f19d726 100644 --- a/tests/functional/x86_64/meson.build +++ b/tests/functional/x86_64/meson.build @@ -37,6 +37,7 @@ tests_x86_64_system_thorough =3D [ 'linux_initrd', 'multiprocess', 'netdev_ethtool', + 'nvme_migration', 'replay', 'reverse_debug', 'tuxrun', diff --git a/tests/functional/x86_64/test_nvme_migration.py b/tests/functio= nal/x86_64/test_nvme_migration.py new file mode 100755 index 00000000000..3788a8e3473 --- /dev/null +++ b/tests/functional/x86_64/test_nvme_migration.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# x86_64 NVMe migration test + +from migration import MigrationTest +from qemu_test import QemuSystemTest, Asset +from qemu_test import wait_for_console_pattern +from qemu_test import exec_command, exec_command_and_wait_for_pattern + + +class X8664NVMeMigrationTest(MigrationTest): + ASSET_KERNEL =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/vmlinuz'), + 'd4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129') + + ASSET_INITRD =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/initrd.img'), + '277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b') + + ASSET_DISKIMAGE =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Cloud/x86_64/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2'), + 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0') + + DEFAULT_KERNEL_PARAMS =3D ('root=3D/dev/nvme0n1p1 console=3DttyS0 net.= ifnames=3D0 ' + 'rd.rescue quiet') + + def wait_for_console_pattern(self, success_message, vm): + wait_for_console_pattern( + self, + success_message, + failure_message=3D"Kernel panic - not syncing", + vm=3Dvm, + ) + + def exec_command_and_check(self, command, vm): + prompt =3D '# ' + exec_command_and_wait_for_pattern(self, + f"{command} && echo OK || echo FAI= L", + 'FAIL', vm=3Dvm) + # Note, that commands we send to the console are echo-ed back, so = if we have a word "FAIL" + # in the command itself, we should expect to see it once. + wait_for_console_pattern(self, 'OK', failure_message=3D"FAIL", vm= =3Dvm) + self.wait_for_console_pattern(prompt, vm) + + def configure_machine(self, vm): + kernel_path =3D self.ASSET_KERNEL.fetch() + initrd_path =3D self.ASSET_INITRD.fetch() + diskimage_path =3D self.ASSET_DISKIMAGE.fetch() + + vm.set_console() + vm.add_args("-cpu", "max") + vm.add_args("-m", "2G") + vm.add_args("-accel", "kvm") + + vm.add_args('-drive', + f'file=3D{diskimage_path},if=3Dnone,id=3Ddrv0,sna= pshot=3Don') + vm.add_args('-device', 'nvme,bus=3Dpcie.0,' + + 'drive=3Ddrv0,id=3Dnvme-disk0,serial=3Dnvmemigrat= etest,bootindex=3D1') + + vm.add_args( + "-kernel", + kernel_path, + "-initrd", + initrd_path, + "-append", + self.DEFAULT_KERNEL_PARAMS + ) + + def launch_source_vm(self, vm): + vm.launch() + + self.wait_for_console_pattern('Entering emergency mode.', vm) + prompt =3D '# ' + self.wait_for_console_pattern(prompt, vm) + + # Synchronize on NVMe driver creating the root device + exec_command_and_wait_for_pattern(self, + "while ! (dmesg -c | grep nvme0n1:) ; do sleep 1 ;= done", + "nvme0n1", vm=3Dvm) + self.wait_for_console_pattern(prompt, vm) + + # prepare system + exec_command_and_wait_for_pattern(self, 'mount /dev/nvme0n1p1 /sys= root', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'chroot /sysroot', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t proc proc /proc', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs /sys= ', + prompt, vm=3Dvm) + + # Run workload before migration to check if it continues to run pr= operly after migration + # + # Workload is simple: it continuously calculates checksums of all = files in /usr/bin + # to generate some I/O load on the NVMe disk and at the same time = it drops caches to + # make sure that we have some read I/O on the disk as well. + # If there are any issues with the migration of the NVMe device, w= e should see errors + # in dmesg and consequently in the workload log. + exec_command_and_wait_for_pattern(self, + "(while [ ! -f /tmp/test_nvme_migr= ation_workload.stop ]; do \ + rm -f /tmp/test_nvme_migration= _workload.iteration_finished; \ + echo 3 > /proc/sys/vm/drop_cac= hes; \ + find /usr/bin -type f -exec ck= sum {} \\;; \ + touch /tmp/test_nvme_migration= _workload.iteration_finished; \ + done) > /dev/null 2> /tmp/test_nvm= e_migration_workload.errors &", + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'echo $! > /tmp/test_nvme_= migration_workload.pid', + prompt, vm=3Dvm) + + # check if process is alive and running + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + def assert_dest_vm(self, vm): + prompt =3D '# ' + + # check if process is alive and running after migration, if not - = fail the test + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + # signal workload to stop + exec_command_and_wait_for_pattern(self, 'touch /tmp/test_nvme_migr= ation_workload.stop', + prompt, vm=3Dvm) + + # wait workload to finish, because we want to examine log to see i= f there are any errors + exec_command_and_wait_for_pattern(self, + "while [ ! -f /tmp/test_nvme_migra= tion_workload.iteration_finished ]; do sleep 1; done;", + prompt, vm=3Dvm) + + exec_command_and_wait_for_pattern(self, 'cat /tmp/test_nvme_migrat= ion_workload.errors', + prompt, vm=3Dvm) + + # fail the test if non-empty + self.exec_command_and_check("[ ! -s /tmp/test_nvme_migration_workl= oad.errors ]", vm) + + def test_migration_with_tcp_localhost(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_tcp_localhost() + + def test_migration_with_unix(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_unix() + + def test_migration_with_exec(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_exec() + + +if __name__ =3D=3D '__main__': + MigrationTest.main() --=20 2.47.3 From nobody Sat May 30 17:44:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1778512455; cv=none; d=zohomail.com; s=zohoarc; b=AK0qqRE3vTDciDh0O3pvYbFj6DaOP70Vp5oUSIYhHsbCme4Eax/wQtx9KKH+e+uz9zX8iHMXDov94fdIiwFBFwjQcrxkwHxCfzscsQShBUwZPWPY9FQedPp8Ksfk5Xgd+6SzvEVfbYqYj5Z4eITxz1vC2rl+17DmQ7ujF6fEhRY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1778512455; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SJxkUm4jKMAc7kPfn1S0XWI7olLmjTreamMkr57EPvI=; b=FqlEYi/rq6tUAGdHnziTh4glsks5R/79p/X97bQjsiYgVMMMgcEFXEJK1zFFGO5yzaNW21Wuone3cNaYh3OU9zhAEUUPO+epFwi8r73JAiYRdHUkD7Vg547c4jXWJasJ494C/+ejbpx2CfAYTSuEcqkHSPBtP5CdcebQHBzLjrs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1778512455795805.4710458293766; Mon, 11 May 2026 08:14:15 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wMSJv-0007Al-0W; Mon, 11 May 2026 11:13:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wMSIh-0005tz-4b for qemu-devel@nongnu.org; Mon, 11 May 2026 11:12:08 -0400 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wMSIP-0001rW-Q1 for qemu-devel@nongnu.org; Mon, 11 May 2026 11:11:52 -0400 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-488ab2db91aso49105645e9.3 for ; Mon, 11 May 2026 08:11:49 -0700 (PDT) Received: from alex-laptop.lan (p200300cf57228c00267b069c8b2d0c7b.dip0.t-ipconnect.de. [2003:cf:5722:8c00:267b:69c:8b2d:c7b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e6d8d6813sm65766925e9.6.2026.05.11.08.11.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 08:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1778512308; x=1779117108; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SJxkUm4jKMAc7kPfn1S0XWI7olLmjTreamMkr57EPvI=; b=e+GtKdC/OSG4Q3gytkHxFdnXeAjjgBaCDAsErn+D5SC8buw/Z+bBkeMpmzl+9R9h95 cu0KsNFvWeY56waphaMvKWr+0VRqXqyOZ1R8KjSL62gdfOqvHS+hyEHzs5ap2igPZ+wc J+ZPgNyrNT9ENh+7z3jgPVHmKw7VBk6gHyCgA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778512308; x=1779117108; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=SJxkUm4jKMAc7kPfn1S0XWI7olLmjTreamMkr57EPvI=; b=OXPlbxqzByvJcB4RJx/SazWUF0UGsWI1MVs/AJXyku9DWyf6p0eizf3iYYNv2zYzvN 1Lco30SjHzctohcZ2PAHixcmz3dC4RCwbWnqfxHFjjRLp3WEwmx9wXgH4KV8ILrLmsr1 s4G+07wpjTtbtYrQoXXquSgADflDiY+ryYK98jUevt+5/PBhKuubCxGvIJyCQGrhFj4g C3SFmyjlSGVRejf/HNHC9Ui9JtBZB95SikhRdD0mcIZBKnk/NLEfPLLrcCtSfKbA1xPh EGFCrYO2dwGg+RoWYwq6SHTQM7CfFy+B4cMoSTJuEkLk6MMyErVay1hQXqn+KMNTGY7o R3JA== X-Gm-Message-State: AOJu0YxWL9hUFRaJ6US+67qoE5PjutzZpPSIFLh8SfYDlat9ejN75aZO S5cxhSW05M7eGHwmFrTsaYsgWjkUa+KOteOdnV0oeSAa6JEYXRD9tXRL77K6dbvpmCYr7IN9Z3d vWpVaGFg= X-Gm-Gg: Acq92OEZ95uwn64z5BeZNgxMufkGayH5H6KNhjLt0TUYumT1mLrij/0bABHXP1zhyRE 3gDYl+2IHVU/d8rfBKt6dFnGAaZnXqbm6A5anDbSswNsHS01hQ0zBxLQppc1K9QNztlwq/o1GOw WlzNu+TFcu9Uk7LFY3LXyuOkxAHRhiACk6AZoOENk+mgo0u3bfv5MPoNVkdA/57hy0z1tloUOM/ MTig7/pAysssE7yM/89kAfDoXrkGud0mcVM1J8TIkwTbIfMGJKH16vW/rClL/Vt+CCjKN25uqUs +883ifyihuHXW02LIeWn9gTqap0XuEzW6P5WYzbzPRV7Fe2AGGGWvAqn5TmawaqH5dkg3d98X0p Mvqp4h7K2tZnOKKwYTg3niklkSlHlC4LWh6onXGoZgLvF6Di27SBj2dj0J2+Z3PtRPifAVkWbaE gZdKQZb8jZuIm1AD4KnPAhhUKq5puiISCt0ehMxMwiqpmjl+0Y4yVVCzcdcKX32hTyXbpW6n44S fX5oaQkziv/IRWyHtVxtW1frPZtPNw= X-Received: by 2002:a05:600c:a411:b0:488:af48:af11 with SMTP id 5b1f17b1804b1-48e51e08ce3mr319709765e9.1.1778512303529; Mon, 11 May 2026 08:11:43 -0700 (PDT) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Hanna Reitz , qemu-block@nongnu.org, Laurent Vivier , Paolo Bonzini , Kevin Wolf , Keith Busch , Fam Zheng , =?UTF-8?q?St=C3=A9phane=20Graber?= , Klaus Jensen , Zhao Liu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fabiano Rosas , Peter Xu , Jesper Devantier , Alexander Mikhalitsyn , Alexander Mikhalitsyn Subject: [PATCH v7 8/8] tests/qtest/nvme-test: add migration test with full CQ Date: Mon, 11 May 2026 17:11:31 +0200 Message-ID: <20260511151131.483217-9-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511151131.483217-1-alexander@mihalicyn.com> References: <20260511151131.483217-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32c; envelope-from=alexander@mihalicyn.com; helo=mail-wm1-x32c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1778512456993158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn As suggested by Stefan [1], let's add a migration test to cover rare scenario when CQ is full of non-processed CQEs and migration happens. To run this test: $ meson test -C build 'qtest-x86_64/qos-test' Link: https://lore.kernel.org/qemu-devel/20260408183529.GB319710@fedora/ [1] Suggested-by: Stefan Hajnoczi Signed-off-by: Alexander Mikhalitsyn --- v7: - fixed endianness bugs (and tested on s390x machine) - code style changes (don't use ptr type for physical addresses) v6: - test added --- tests/qtest/nvme-test.c | 395 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 395 insertions(+) diff --git a/tests/qtest/nvme-test.c b/tests/qtest/nvme-test.c index 4aec1651e6e..4d063ee971f 100644 --- a/tests/qtest/nvme-test.c +++ b/tests/qtest/nvme-test.c @@ -8,9 +8,12 @@ */ =20 #include "qemu/osdep.h" +#include +#include "qemu/bswap.h" #include "qemu/module.h" #include "qemu/units.h" #include "libqtest.h" +#include "libqtest-single.h" #include "libqos/qgraph.h" #include "libqos/pci.h" #include "block/nvme.h" @@ -142,6 +145,396 @@ static void nvmetest_pmr_reg_test(void *obj, void *da= ta, QGuestAllocator *alloc) qpci_iounmap(pdev, pmr_bar); } =20 +#define PAGE_SIZE 4096 + +typedef struct nvme_ctrl nvme_ctrl; + +typedef struct nvme_queue { + nvme_ctrl *ctrl; + uint64_t doorbell; + uint32_t size; +} nvme_queue; + +typedef struct nvme_cq { + nvme_queue common; + uint64_t phys_cqe; /* NvmeCqe* */ + uint16_t head; + uint8_t phase; +} nvme_cq; + +typedef struct nvme_sq { + nvme_queue common; + uint64_t phys_sqe; /* NvmeCmd* */ + nvme_cq *cq; + uint16_t head; + uint16_t tail; +} nvme_sq; + +struct nvme_ctrl { + QGuestAllocator *alloc; + QPCIDevice *pdev; + QPCIBar bar; + + uint32_t db_stride; + + nvme_sq admin_sq; + nvme_cq admin_cq; +}; + +#define PHYS_ADDR_OF(T, base_phys_addr, accessor) \ + (uint64_t)&((T *)base_phys_addr)accessor + +static void nvme_init_queue_common(nvme_ctrl *ctrl, nvme_queue *q, + uint16_t db_idx, uint32_t size) +{ + q->ctrl =3D ctrl; + q->doorbell =3D (sizeof(NvmeBar) + db_idx * ctrl->db_stride); + g_test_message(" q %p db_idx %u doorbell %lx", q, db_idx, q->doorbell); + q->size =3D size; +} + +static void nvme_init_sq(nvme_ctrl *ctrl, nvme_sq *sq, uint16_t db_idx, + uint32_t size, nvme_cq *cq) +{ + nvme_init_queue_common(ctrl, &sq->common, db_idx, size); + + sq->phys_sqe =3D guest_alloc(ctrl->alloc, PAGE_SIZE); + g_assert(sq->phys_sqe); + + g_test_message("sq %p db_idx %u sqe 0x%" PRIx64, sq, db_idx, sq->phys_= sqe); + sq->cq =3D cq; + sq->head =3D 0; + sq->tail =3D 0; +} + +static void nvme_init_cq(nvme_ctrl *ctrl, nvme_cq *cq, uint16_t db_idx, + uint32_t size) +{ + nvme_init_queue_common(ctrl, &cq->common, db_idx, size); + + cq->phys_cqe =3D guest_alloc(ctrl->alloc, PAGE_SIZE); + g_assert(cq->phys_cqe); + + g_test_message("cq %p db_idx %u cqe 0x%" PRIx64, cq, db_idx, cq->phys_= cqe); + cq->head =3D 0; + cq->phase =3D 1; +} + +static int nvme_cqe_pending(nvme_cq *cq) +{ + uint16_t status =3D qtest_readw(cq->common.ctrl->pdev->bus->qts, + PHYS_ADDR_OF(NvmeCqe, cq->phys_cqe, [cq-= >head].status)); + return (status & 1) =3D=3D cq->phase; +} + +static int nvme_is_cqe_success(NvmeCqe *cqe) +{ + return (le16_to_cpu(cqe->status) >> 1) =3D=3D NVME_SUCCESS; +} + +static NvmeCqe nvme_handle_cqe(nvme_sq *sq) +{ + nvme_cq *cq =3D sq->cq; + uint64_t phys_cqe =3D PHYS_ADDR_OF(NvmeCqe, cq->phys_cqe, [cq->head]);= /* NvmeCqe* */ + NvmeCqe cqe; + uint16_t cq_next_head; + + g_assert(nvme_cqe_pending(cq)); + + qtest_memread(sq->common.ctrl->pdev->bus->qts, phys_cqe, &cqe, sizeof(= cqe)); + + cq_next_head =3D (cq->head + 1) % cq->common.size; + g_test_message("cq %p head %u -> %u", cq, cq->head, cq_next_head); + if (cq_next_head < cq->head) { + cq->phase ^=3D 1; + } + cq->head =3D cq_next_head; + + if (cqe.sq_head !=3D sq->head) { + sq->head =3D cqe.sq_head; + g_test_message("sq %p head =3D %u", sq, sq->head); + } + + qpci_io_writel(cq->common.ctrl->pdev, cq->common.ctrl->bar, cq->common= .doorbell, cq->head); + + return cqe; +} + +static NvmeCqe nvme_wait(nvme_sq *sq) +{ + int i; + bool ready =3D false; + + for (i =3D 0; i < 10; i++) { + if (nvme_cqe_pending(sq->cq)) { + ready =3D true; + break; + } + + g_usleep(1000); + } + + g_assert(ready); + + return nvme_handle_cqe(sq); +} + +static uint64_t nvme_get_next_sqe(nvme_sq *sq, uint8_t opcode, uint16_t ci= d, uint64_t prp1) +{ + uint64_t phys_sqe =3D PHYS_ADDR_OF(NvmeCmd, sq->phys_sqe, [sq->tail]); + + if (((sq->tail + 1) % sq->common.size) =3D=3D sq->head) { + /* no space in SQ */ + g_test_message("%s head %d tail %d", __func__, sq->head, sq->tail); + g_assert(false); + return 0; + } + + qtest_memset(sq->common.ctrl->pdev->bus->qts, + phys_sqe, 0, sizeof(NvmeCmd)); + + #define GUEST_MEM_WRITE(fn, phys_addr, val) \ + fn(sq->common.ctrl->pdev->bus->qts, phys_addr, (val)) + + GUEST_MEM_WRITE(qtest_writeb, PHYS_ADDR_OF(NvmeCmd, phys_sqe, ->opcode= ), opcode); + GUEST_MEM_WRITE(qtest_writew, PHYS_ADDR_OF(NvmeCmd, phys_sqe, ->cid), = cid); + GUEST_MEM_WRITE(qtest_writeq, PHYS_ADDR_OF(NvmeCmd, phys_sqe, ->dptr.p= rp1), prp1); + + #undef GUEST_MEM_WRITE + + g_test_message("sq %p next_sqe %u sqe 0x%" PRIx64, sq, sq->tail, phys_= sqe); + return phys_sqe; +} + +static void nvme_commit_sqe(nvme_sq *sq) +{ + g_test_message("sq %p commit sqe tail %u", sq, sq->tail); + sq->tail =3D (sq->tail + 1) % sq->common.size; + qpci_io_writel(sq->common.ctrl->pdev, sq->common.ctrl->bar, sq->common= .doorbell, sq->tail); +} + +static uint64_t nvme_admin_identify_ctrl(nvme_ctrl *ctrl, uint16_t cid, bo= ol no_wait) +{ + uint64_t phys_cmd_identify; /* NvmeCmd* */ + uint64_t phys_identify; /* NvmeIdCtrl* */ + NvmeCqe cqe; + + g_test_message("sending req cid %u no_wait %d", cid, no_wait); + + phys_identify =3D guest_alloc(ctrl->alloc, PAGE_SIZE); + g_assert(phys_identify); + + phys_cmd_identify =3D nvme_get_next_sqe(&ctrl->admin_sq, + NVME_ADM_CMD_IDENTIFY, cid, + phys_identify); + g_assert(phys_cmd_identify); + + #define GUEST_MEM_WRITE(fn, phys_addr, val) \ + fn(ctrl->pdev->bus->qts, phys_addr, (val)) + + GUEST_MEM_WRITE(qtest_writel, PHYS_ADDR_OF(NvmeCmd, phys_cmd_identify,= ->nsid), 0); + GUEST_MEM_WRITE(qtest_writel, PHYS_ADDR_OF(NvmeIdentify, phys_cmd_iden= tify, ->cns), NVME_ID_CNS_CTRL); + + #undef GUEST_MEM_WRITE + + nvme_commit_sqe(&ctrl->admin_sq); + + if (no_wait) { + return phys_identify; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + g_assert(le16_to_cpu(cqe.cid) =3D=3D cid); + + return phys_identify; +} + +static void nvme_wait_ready(nvme_ctrl *ctrl, int val) +{ + int i; + + for (i =3D 0; i < 10; i++) { + uint32_t csts =3D qpci_io_readl(ctrl->pdev, ctrl->bar, NVME_REG_CS= TS); + g_test_message("%s: csts %x", __func__, csts); + + if (NVME_CSTS_RDY(csts) =3D=3D val) { + return; + } + + g_usleep(1000); + } + + g_assert(false); +} + +static void test_migrate_setup_nvme_ctrl(nvme_ctrl *ctrl) +{ + uint64_t cap; + + /* disable controller */ + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, 0); + nvme_wait_ready(ctrl, 0); + + cap =3D qpci_io_readq(ctrl->pdev, ctrl->bar, NVME_REG_CAP); + ctrl->db_stride =3D 4 << NVME_CAP_DSTRD(cap); + + nvme_init_cq(ctrl, &ctrl->admin_cq, 1, 2 /* CQEs num */); + nvme_init_sq(ctrl, &ctrl->admin_sq, 0, 4 /* SQEs num */, &ctrl->admin_= cq); + + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_AQA, + ((ctrl->admin_cq.common.size - 1) << AQA_ACQS_SHIFT) | + ((ctrl->admin_sq.common.size - 1) << AQA_ASQS_SHIFT) + ); + + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ASQ, (uint64_t)ctrl->admin_sq.phys_sqe); + qpci_io_writeq(ctrl->pdev, ctrl->bar, + NVME_REG_ACQ, (uint64_t)ctrl->admin_cq.phys_cqe); + + /* enable controller */ + { + uint32_t cc =3D 0; + NVME_SET_CC_EN(cc, 1); + qpci_io_writel(ctrl->pdev, ctrl->bar, NVME_REG_CC, cc); + } + + nvme_wait_ready(ctrl, 1); +} + +typedef struct test_migrate_req { + uint16_t cid; + bool handle_cqe; + uint64_t phys_identify; /* NvmeIdCtrl* */ +} test_migrate_req; + +static void test_migrate_send_nvme_reqs(nvme_ctrl *ctrl, test_migrate_req = *reqs, + int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + reqs[i].phys_identify =3D nvme_admin_identify_ctrl(ctrl, reqs[i].c= id, + !reqs[i].handle_c= qe); + g_assert(reqs[i].phys_identify); + + if (reqs[i].handle_cqe) { + guest_free(ctrl->alloc, reqs[i].phys_identify); + } + } +} + +static void test_migrate_check_nvme(nvme_ctrl *ctrl, test_migrate_req *req= s, int num) +{ + int i; + + for (i =3D 0; i < num; i++) { + NvmeCqe cqe; + + if (reqs[i].handle_cqe) { + continue; + } + + cqe =3D nvme_wait(&ctrl->admin_sq); + g_assert(nvme_is_cqe_success(&cqe)); + + g_assert_cmpint(le16_to_cpu(cqe.cid), =3D=3D, reqs[i].cid); + + #define GUEST_MEM_READB(phys_addr) \ + qtest_readb(ctrl->pdev->bus->qts, (phys_addr)) + + g_assert_cmpint(GUEST_MEM_READB(PHYS_ADDR_OF(NvmeIdCtrl, reqs[i].p= hys_identify, ->ieee[0])), =3D=3D, 0x0); + g_assert_cmpint(GUEST_MEM_READB(PHYS_ADDR_OF(NvmeIdCtrl, reqs[i].p= hys_identify, ->ieee[1])), =3D=3D, 0x54); + g_assert_cmpint(GUEST_MEM_READB(PHYS_ADDR_OF(NvmeIdCtrl, reqs[i].p= hys_identify, ->ieee[2])), =3D=3D, 0x52); + + #undef GUEST_MEM_READB + + guest_free(ctrl->alloc, reqs[i].phys_identify); + } +} + +static void test_migrate(void *obj, void *data, QGuestAllocator *alloc) +{ + g_autofree gchar *tmpfs =3D NULL; + GError *err =3D NULL; + g_autofree gchar *mig_path; + g_autofree gchar *uri; + GString *dest_cmdline; + QTestState *to; + QDict *rsp; + QNvme *nvme =3D obj; + QPCIDevice *pdev =3D &nvme->dev; + g_autofree nvme_ctrl *ctrl; + test_migrate_req test_reqs[] =3D { + { 123, true }, + { 456, false }, + { 300, false }, + { 333, false } + }; + + /* create temporary dir and prepare unix socket path for migration */ + tmpfs =3D g_dir_make_tmp("nvme-test-XXXXXX", &err); + if (!tmpfs) { + g_test_message("Can't create temporary directory in %s: %s", + g_get_tmp_dir(), err->message); + g_error_free(err); + } + g_assert(tmpfs); + + mig_path =3D g_strdup_printf("%s/socket.mig", tmpfs); + uri =3D g_strdup_printf("unix:%s", mig_path); + + /* enable NVMe PCI device */ + qpci_device_enable(pdev); + + ctrl =3D g_malloc0(sizeof(*ctrl)); + ctrl->alloc =3D alloc; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D global_qtest); + + test_migrate_setup_nvme_ctrl(ctrl); + test_migrate_send_nvme_reqs(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + dest_cmdline =3D g_string_new(qos_get_current_command_line()); + g_string_append_printf(dest_cmdline, " -incoming %s", uri); + + /* Create destination VM */ + to =3D qtest_init(dest_cmdline->str); + + /* Get access to PCI device from destination VM */ + nvme =3D qos_allocate_objects(to, &ctrl->alloc); + pdev =3D &nvme->dev; + ctrl->pdev =3D pdev; + ctrl->bar =3D qpci_iomap(ctrl->pdev, 0, NULL); + g_assert(pdev->bus->qts =3D=3D to); + + /* Migrate VM */ + rsp =3D qmp("{ 'execute': 'migrate', 'arguments': { 'uri': %s } }", ur= i); + g_assert(qdict_haskey(rsp, "return")); + qobject_unref(rsp); + + /* Wait when source VM is stopped */ + qmp_eventwait("STOP"); + + /* Copy guest physical memory allocator state */ + migrate_allocator(alloc, ctrl->alloc); + + /* Wait for destination VM to become alive */ + qtest_qmp_eventwait(to, "RESUME"); + + test_migrate_check_nvme(ctrl, test_reqs, ARRAY_SIZE(test_reqs)); + + qpci_iounmap(ctrl->pdev, ctrl->bar); + + qtest_quit(to); + g_unlink(mig_path); + g_rmdir(tmpfs); + g_string_free(dest_cmdline, true); +} + static void nvme_register_nodes(void) { QOSGraphEdgeOptions opts =3D { @@ -168,6 +561,8 @@ static void nvme_register_nodes(void) }); =20 qos_add_test("reg-read", "nvme", nvmetest_reg_read_test, NULL); + + qos_add_test("migrate", "nvme", test_migrate, NULL); } =20 libqos_init(nvme_register_nodes); --=20 2.47.3