From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809499; cv=none; d=zohomail.com; s=zohoarc; b=D2OliY/ozhLSyYwHaeHVApbzN2uPpZN/B+obZffY/lM+Gvl7AshaXEmYTltuSYukagwj7xz+YO3AEPSYZllMBKpIUmCi776AMa4gknafYwwECnSizFM2OrSiBLwB188Av/Y4P85rJ1juoF3y8ryboFCPZwi5OL0mU4Wy3jLt/2s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809499; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Aa7YS8YRAQ8yRhQF9Whg1Qcoxr3p3yRf12L+u0hXtWc=; b=A/fGGwdpurwbVK8MgJwprkeNHxQgo5UwEw4lP/wMHUXlxw0Kq4/zCGe7Uwli7vkYX/vvj1eDZo2UiPsaUIY8sU3eiPDkBEWVsH6A62M1n6JMKfRjWf4rD4+Gt1K0SKWcCjzPSQOTtNGivsplIb48Cxc89PV4tAvuuCz7G21v+zI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772809499713201.09542526760072; Fri, 6 Mar 2026 07:04:59 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWiz-0008ER-H5; Fri, 06 Mar 2026 10:04:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWif-0007v5-AC for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:04:03 -0500 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWid-0003Le-Hd for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:04:01 -0500 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-439b8a3f2bcso4479694f8f.3 for ; Fri, 06 Mar 2026 07:03:58 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809437; x=1773414237; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Aa7YS8YRAQ8yRhQF9Whg1Qcoxr3p3yRf12L+u0hXtWc=; b=MU9H0/3cesldAQoj2jQodTuSmj7vCQcTq4wDe+RPIHvJ/R2nw0V1HHR3cL4WLq4nYy mxK9uIxwNlqXM+Flxx06hciNoLuX32LccBh1nNKFfmWEySG6M9ZNeLhsqztzMGSY2NFp MfBtm4nJOSygOTkg64gSj2GjM9uWLJg1vfmDQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809437; x=1773414237; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Aa7YS8YRAQ8yRhQF9Whg1Qcoxr3p3yRf12L+u0hXtWc=; b=Ai7lsfd8f+NZUJbsJDY8eyw086npbhZhyCY8lxTszA08uwqDYAEdF9tG0AWFCXkxkx Aggt+ATFrNWPp6DIZtPxJOLBMG5AYbJGTNQOpvvTbKQCST+TCiv9sr2A2QS3CmwlFUDH DfUSOxu5Clzh5RC4kPN4Nai5a0GrXeoGMMMyv+b4e3lTK1ceYRKTcoIb0LY8IhK1R2+j Sm+lPnaGIiqeCEx9VkUH4uk7uYjkIwB8bRuR9wtHq9Cq7YC8+P5aMKj8kEXhQ+ix9Wrp zs/C2yNyiNWMjucuYe4YtW5Z7fYMsuuF/0HJNHHW2ud06M95V97rtot5YDjbIc/Tx/M1 eFng== X-Gm-Message-State: AOJu0YyRIu1c63u8LykGrrTArHPFMA98hidZawAJqBZv8cELzFbNH1e4 5o35FNhXbdR7mWEnDQen3NezZ78437FsKNUSYgfkB3FvPy4c+o1bAotNxyJh59Pu4MGUzG0Do3K vLzYXgS0= X-Gm-Gg: ATEYQzwjAPnyRnpOI8K6WBFtpZolSYxNvV5qlWM+retlD9uK9Ven8/iVRn+VmnrJWd0 s90g7UHMI4gSW0j6xactjtw4A4IlF3jEI6UAZGngq/geXfOBERJs5xRDIxfkZJj0G1toQteve14 dOMlO0ZTCCNj2GmmJ0kRizgs5Bhk9GatgDnCNjauwTwqWHP12y8K7Q0SgfzMOLUmJKIfLTqA3p+ nuhzbBLfmTANTRCx6Xs/5++oXma9dhBJEe9Z/Tjda80gJj4w874czPdZx+qg7ZdHfVeCW5p0W/U nw5t8O5sogOlYCI8yy8gLdg2YAyaqXcfrAWZ502mu6w9k/i+vFHqO1B1/HmBgjgZS6h7cJ/y9nM AaTv7oKHkfX3xjs5w7HS9XY/dgXZ8cpBbbpgpXILvhp1Z9/1zNrUSMRPzjxBxh9FmXNZBDkotFM RkxI6YGf0JVQ1zp7toP+eo/rkbx4JW13aauWRm7Vpd3D0ae1rcDaPVeV7Tcc5uhC44wS7yrTWxN JSS0ziNVymTMTePo8MFU4E= X-Received: by 2002:a05:6000:1acd:b0:439:c1ca:82be with SMTP id ffacd0b85a97d-439da885f92mr3986014f8f.28.1772809425164; Fri, 06 Mar 2026 07:03:45 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 1/6] migration: add VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT{8, 32}_ALLOC Date: Fri, 6 Mar 2026 16:03:37 +0100 Message-ID: <20260306150342.395923-2-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::436; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x436.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809505487154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Add VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT{8, 32}_ALLOC, which helps to save/restore a dynamic array of pointers to structures. Signed-off-by: Alexander Mikhalitsyn v2: - added VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC --- include/migration/vmstate.h | 33 ++++++++++++++ migration/vmstate-types.c | 88 +++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index 89f9f49d20a..0bbe3317740 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -265,6 +265,7 @@ extern const VMStateInfo vmstate_info_bitmap; extern const VMStateInfo vmstate_info_qtailq; extern const VMStateInfo vmstate_info_gtree; extern const VMStateInfo vmstate_info_qlist; +extern const VMStateInfo vmstate_info_ptrs_array_entry; =20 #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0) /* @@ -537,6 +538,38 @@ extern const VMStateInfo vmstate_info_qlist; .offset =3D vmstate_offset_array(_s, _f, _type*, _n), \ } =20 +/* + * For migrating a dynamically allocated uint32-indexed array + * of pointers to structures (with NULL entries and with auto memory alloc= ation). + * + * _type: type of structure pointed to + * _vmsd: VMSD for structure + * start: size of structure pointed to (for auto memory allocation) + */ +#define VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC(_field, _state, _f= ield_num, _version, _vmsd, _type) { \ + .name =3D (stringify(_field)), \ + .version_id =3D (_version), \ + .num_offset =3D vmstate_offset_value(_state, _field_num, uint8_t), \ + .info =3D &vmstate_info_ptrs_array_entry, \ + .vmsd =3D &(_vmsd), \ + .start =3D sizeof(_type), \ + .size =3D sizeof(_type *), \ + .flags =3D VMS_VARRAY_UINT8|VMS_POINTER, \ + .offset =3D vmstate_offset_pointer(_state, _field, _type *), \ +} + +#define VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC(_field, _state, _= field_num, _version, _vmsd, _type) { \ + .name =3D (stringify(_field)), \ + .version_id =3D (_version), \ + .num_offset =3D vmstate_offset_value(_state, _field_num, uint32_t), \ + .info =3D &vmstate_info_ptrs_array_entry, \ + .vmsd =3D &(_vmsd), \ + .start =3D sizeof(_type), \ + .size =3D sizeof(_type *), \ + .flags =3D VMS_VARRAY_UINT32|VMS_POINTER, \ + .offset =3D vmstate_offset_pointer(_state, _field, _type *), \ +} + #define VMSTATE_VARRAY_OF_POINTER_UINT32(_field, _state, _field_num, _vers= ion, _info, _type) { \ .name =3D (stringify(_field)), = \ .version_id =3D (_version), = \ diff --git a/migration/vmstate-types.c b/migration/vmstate-types.c index 89cb2114721..3335377cd07 100644 --- a/migration/vmstate-types.c +++ b/migration/vmstate-types.c @@ -942,3 +942,91 @@ const VMStateInfo vmstate_info_qlist =3D { .get =3D get_qlist, .put =3D put_qlist, }; + +static int put_ptrs_array_entry(QEMUFile *f, void *ppv, size_t unused_size, + const VMStateField *field, JSONWriter *vmd= esc) +{ + const VMStateDescription *vmsd =3D field->vmsd; + int ret; + Error *local_err =3D NULL; + void *pv; + + /* + * (ppv) is an address of an i-th element of a dynamic array. + * + * (ppv) can not be NULL unless we have some regression/bug in + * vmstate_save_state_v(), because it is result of pointer arithemic l= ike: + * first_elem + size * i. + */ + if (ppv =3D=3D NULL) { + error_report("vmstate: put_ptrs_array_entry must be called with pp= v !=3D NULL"); + return -EINVAL; + } + + /* get a pointer to a structure */ + pv =3D *(void **)ppv; + + if (pv =3D=3D NULL) { + /* write a mark telling that there was a NULL pointer */ + qemu_put_byte(f, false); + return 0; + } + + /* if pointer is not NULL, dump the structure contents with help of vm= sd */ + qemu_put_byte(f, true); + ret =3D vmstate_save_state(f, vmsd, pv, vmdesc, &local_err); + if (ret) { + error_report_err(local_err); + return ret; + } + + return 0; +} + +static int get_ptrs_array_entry(QEMUFile *f, void *ppv, size_t unused_size, + const VMStateField *field) +{ + int ret =3D 0; + Error *local_err =3D NULL; + const VMStateDescription *vmsd =3D field->vmsd; + /* size of structure pointed to by elements of array */ + size_t size =3D field->start; + + if (ppv =3D=3D NULL) { + error_report("vmstate: get_ptrs_array_entry must be called with pp= v !=3D NULL"); + return -EINVAL; + } + + /* + * We start from a clean array, all elements must be NULL, unless + * something we haven't prepared for has changed in vmstate_save_state= _v(). + * Let's check for this just in case. + */ + if (*(void **)ppv !=3D NULL) { + error_report("vmstate: get_ptrs_array_entry must be called with *p= pv =3D=3D NULL"); + return -EINVAL; + } + + if (qemu_get_byte(f)) { + void *pv; + + /* allocate memory for structure */ + pv =3D g_malloc0(size); + ret =3D vmstate_load_state(f, vmsd, pv, vmsd->version_id, &local_e= rr); + if (ret) { + error_report_err(local_err); + g_free(pv); + return ret; + } + + *(void **)ppv =3D pv; + } + + return ret; +} + +const VMStateInfo vmstate_info_ptrs_array_entry =3D { + .name =3D "ptrs_array_entry", + .get =3D get_ptrs_array_entry, + .put =3D put_ptrs_array_entry, +}; --=20 2.47.3 From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809628; cv=none; d=zohomail.com; s=zohoarc; b=NBlKS31b9ZzRfuw5LpiJsEnczTRIFpzH7/ohQPuQwTjWthgwyAZ1ZsF5ES0jYgqBX9S5rdh31Z6GypFspXY4Y5x9YrYo+pMb4wHTL+pUh9oNTUxnKCKriOFUDCleXxvu6WvnHLEmd4JLUxX/NfuOVoboS4fkalIjYUybSerHYO8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809628; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=GIAmNR0oZ0DWaK6vnXUjsxDdMhpQMXKTi6OwuCEcqAU=; b=T05M/OevU1dj88lPlXR9YhNMpMK41+O66mFubzdzP45wrTWgloMbREfRlW+NEYNqFKX1mC7XXPR6/1mmfk753sO3TPZmCCZHkkpVOIVqBIuMW/gczjGt2JSJuGIg5MEyo9wwIy/D3co6XKirwwcsXQMR2PfImLhjpzDwpOryXoU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772809628429836.9437790553736; Fri, 6 Mar 2026 07:07:08 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWiZ-0007Wz-TS; Fri, 06 Mar 2026 10:03:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWiU-0006sV-G8 for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:50 -0500 Received: from mail-ej1-x62b.google.com ([2a00:1450:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWiS-0003JQ-L7 for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:50 -0500 Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-b9423d62cbbso141048366b.1 for ; Fri, 06 Mar 2026 07:03:48 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809427; x=1773414227; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GIAmNR0oZ0DWaK6vnXUjsxDdMhpQMXKTi6OwuCEcqAU=; b=FG+VJwCUv08n6o8sxrCCmRdeuZxhsPt3EQg1Ff3AgwtsN0VxQzQaPNVM1tlT7oSF6e lC4SfEJY7AQxTkKVsg0jzklk1gC5zb6u/FAJcXpZKkf+40t7uJoqXPqTaDEyK5vtFy0/ tMyt1ZQ/7kQtZ7S3p+WhoAYOVXGKOOCJxdcTI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809427; x=1773414227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=GIAmNR0oZ0DWaK6vnXUjsxDdMhpQMXKTi6OwuCEcqAU=; b=pRWl9eV/ppNAfryvGqBrAuvV46DZVhDpl3H3BSiK7EZMZvSpW74iXPLVey0Oz/ZGYE nF6+UBajsbMOM+j+AFRnEUxVpTDSWp0z5vn/g97d4f5lDNZGmKgdqfwXAyzDNhwrfLeH wy1AO8XkT01mH8Js+st0fGmi1OiUc/WfSWtmazuayn45voRk5q4oRDfoZoYQzgYHu4CT ByvFElcyex7cKBoKVqdbHD/2zbL4JT28j+crRUOvNJaKq+KB57vyoonUI3wAJU0lSbT/ hUkbONjev+Y86YVnwqbuZW2ZOkk2qPPDnqeYJgtMnsJZ7o6rybrepFNXoQ/ONnJezZ6W H4OA== X-Gm-Message-State: AOJu0Yxy34ka8aKRLd2VjDCx/5bLIr1w/fFQM4ZUvkfUrsXo8bk6nGz/ DkWgra5fpqBtx/rWoggiB8C7KN0LL629Kmu/VFRyfClpfuIYbBMkjtoWfY9hc8XSOSH2KP+Mt4j 3je+lySU= X-Gm-Gg: ATEYQzw/HdqKY0/5J+WreLuH4AccT4vXDMUnr0tVzfRAHyYx/Ke7uR8y8tnwy1iDNnL jEKePt7vep/TkO4885Lk6VP3k6yau9wAuhs5Aur51VN2CQPOyfHTovbDlrw/jni9VGEye23aRU1 +9uy4QjJJlRQdNhG4sucl+u4wW1cK6fE8rLRpvxHUbMrsadzTIKJ2+tVuyEXXGx9wlrv5gigo5G HdXLysseTrNwhObKKeeX4a5uxshw6B7dSo7nDssLqhzggbIz1jI4tgkdUjqAzjTYYEv0J766Eat L1AaPlbBixFruY9mQBDPS/lv839ufUfk4bGlJtdrv7manchi6/DBmP3iaD8rageWSeFfeluMd5I awe5ag9udZjeug0QAZ7nXZnDk6VmOLtGXlri/lVZEmSHzimSdKSi1h0oiGwBeuFkOkuYHT32vfb Ko0okafT3NLaR2Csx3KFj5d3IJTANNTBZj44Q45/0c1ZLa8EohPOAFMMyZTEVtAUCvnRx5KuNqm bmqW/v4SgcDkFO52g+jVSyJ5OqmpRWmnQ== X-Received: by 2002:a17:907:9605:b0:b87:75c:3660 with SMTP id a640c23a62f3a-b9409ed324dmr393923866b.28.1772809426188; Fri, 06 Mar 2026 07:03:46 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 2/6] tests/functional/migration: add VM launch/configure hooks Date: Fri, 6 Mar 2026 16:03:38 +0100 Message-ID: <20260306150342.395923-3-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::62b; envelope-from=alexander@mihalicyn.com; helo=mail-ej1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809629753158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce configure_machine, launch_source_vm and assert_dest_vm methods to allow child classes to override some pieces of source/dest VMs creation, start and check logic. Signed-off-by: Alexander Mikhalitsyn --- tests/functional/migration.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/functional/migration.py b/tests/functional/migration.py index 2bfb1f77901..759906fd820 100644 --- a/tests/functional/migration.py +++ b/tests/functional/migration.py @@ -41,17 +41,33 @@ def assert_migration(self, src_vm, dst_vm): self.assertEqual(dst_vm.cmd('query-status')['status'], 'running') self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate= ') =20 + # Can be overridden by subclasses to configure both source/dest VMs. + def configure_machine(self, vm): + vm.add_args('-nodefaults') + + # Can be overridden by subclasses to prepare the source VM before migr= ation, + # e.g. by running some workload inside the source VM to see if it cont= inues + # to run properly after migration. + def launch_source_vm(self, vm): + vm.launch() + + # Can be overridden by subclasses to check the destination VM after mi= gration, + # e.g. by checking if the workload is still running after migration. + def assert_dest_vm(self, vm): + pass + def do_migrate(self, dest_uri, src_uri=3DNone): dest_vm =3D self.get_vm('-incoming', dest_uri, name=3D"dest-qemu") - dest_vm.add_args('-nodefaults') + self.configure_machine(dest_vm) dest_vm.launch() if src_uri is None: src_uri =3D dest_uri source_vm =3D self.get_vm(name=3D"source-qemu") - source_vm.add_args('-nodefaults') - source_vm.launch() + self.configure_machine(source_vm) + self.launch_source_vm(source_vm) source_vm.qmp('migrate', uri=3Dsrc_uri) self.assert_migration(source_vm, dest_vm) + self.assert_dest_vm(dest_vm) =20 def _get_free_port(self, ports): port =3D ports.find_free_port() --=20 2.47.3 From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809726; cv=none; d=zohomail.com; s=zohoarc; b=ibg+m7OdB7mLY1202bDp7ulgySxAFRDPgovIDThiHtqMBOVWS6umrcd1UGQQj4wDhcK6rJkt0QL/H+PYYrSukayNIayy28/lLGLFblzPJM4dFuEUwz5qaXrpZYIT5ve/q/6dzf6bkIbCehw7i7ZEVPZ9wk3VSb88axHazsnyZas= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809726; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ZTBRdCe4EKhZ7QFgzj0snqvNaQKXZdaqV+19ND1Hpu0=; b=OwybBFErhrtGl/hsPMH9rO97p7XFB0M8JXPkZFy7VGr0RLX3AyDVj1VqQdK0o8RAkE6VjPGp6/MDVSwKTHFKz8pHxmtbqY2yb9P+KSgV0nVyCk96607IvdHpWVpuVtGvk5V9EXB5E5fI0QX5cQCilMCk/fOJgdJXSbwQ/yALslU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772809726776995.0306112633159; Fri, 6 Mar 2026 07:08:46 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWiX-0007Et-NL; Fri, 06 Mar 2026 10:03:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWiV-0006w8-1N for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:51 -0500 Received: from mail-ej1-x636.google.com ([2a00:1450:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWiT-0003Ja-2q for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:50 -0500 Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-b9431300833so105979466b.0 for ; Fri, 06 Mar 2026 07:03:48 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809427; x=1773414227; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZTBRdCe4EKhZ7QFgzj0snqvNaQKXZdaqV+19ND1Hpu0=; b=SM46xUtGsgh2mVP7iK1ZhTTYLCe26dwEfOWGNxIYzzz8cYfaFrrmgDZ/sFNiW7FbEs nFExOKNnD8Fw/LstbmNl5DFrlg56wFC+SZDWPfeEtbfojLo/IBzFHXHzdESbcmev+wrm /q0E9utZH+AX8fSJMnkQGKi528QeEdNCMjass= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809427; x=1773414227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ZTBRdCe4EKhZ7QFgzj0snqvNaQKXZdaqV+19ND1Hpu0=; b=VONwvqzgSEW2z9XgjSSPuKBixnMP/Uycn/iZvmZgZofQwgQqQS4bpFDsik1SJYHu1h H99z7cJyYR2VdPWec7VbWGWhqf/aul8cpEAQMK4tdC0Jd1Kl5xubskC6H1two4nSRCOd 1B5msbTMIZ4GB5h6D6KCQ9bD/s3Ch464NaQ45TBWA8SszMy4997bGJlL8I3wi06tZ9SM f9dWFHH/1872vurORcDKdXxiY1pq6+W7i5MK06yCSMgqa7uIYtB9VlBGpR+nRpARGD9f wSsYFntv+L0wDniK0XXQNLGbZVSLtq2KXBTDRdiXav1Ro59wBa+jvgIEmXfTgJuUKbCb QmDg== X-Gm-Message-State: AOJu0YwyXmzDqNFOAwajgvJ833moWiMMsCGM607t0oSuIMrQuz+FVZta bxbciZOhpgjz60maxMMXiLhDftLF1RoLd30gAL2G+y+c1nQKiLc3dANNHnlwvjbD+P5kEGc4mYm 3EO56VCQ= X-Gm-Gg: ATEYQzzVC1TRlwRKM+T6T9S6A1OAeKRzDUm5c26a9pqgUflgz/K/ty/d/rCkEs8xZD6 pGu3rIxAqnFjJ1cuoy1eH44aJ+g4QjUeV8G9M88eZI1KLmyOZRA8ugsEEPCyxHzezcMf7tFpkPd rtlU5L9agXhi4QiyQNWuZQIXUpc4x1lQO7Hg5oLAUsXkFtJJ+2+jCEZNOt87miiJtBy5GKyZ3OC nEGlJ4oJZkpCaseERAw31dVnA97pmrlioZVJpuACX6EiWfVltfbDh3VMXZpCei0z3ir/JFeusk+ P+sBR54r4/ioXU/ChTTeSJTyoOC+Q3CPVedOd1ZQ+aV8bXYs2/GTDOK6GYeb8JERnAJCEcYn3Ms joqUXsr0P9TDy5l5S21q7Ev4wlZxfW6/fBmRipaHPAGrPvQXGK/71CrT5HaPiFIUH5QCw574hN1 mwWan8I3gzeWAkFO8+2UetEnXpLns9QcVSdB3l3UsZz8iWvBZwlZP/1UJObleRQ0fmonkx1m+Po i/poO39AUZTA2k10gG3SKg= X-Received: by 2002:a17:907:6092:b0:b90:ba11:1694 with SMTP id a640c23a62f3a-b942dfbd66fmr150488866b.50.1772809427194; Fri, 06 Mar 2026 07:03:47 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 3/6] hw/nvme: add migration blockers for non-supported cases Date: Fri, 6 Mar 2026 16:03:39 +0100 Message-ID: <20260306150342.395923-4-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::636; envelope-from=alexander@mihalicyn.com; helo=mail-ej1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809729642154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Let's block migration for cases we don't support: - SR-IOV - CMB - PMR - SPDM No functional changes here, because NVMe migration is not supported at all as of this commit. Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 35 +++++++++++++++++++++++++++++++++++ hw/nvme/nvme.h | 3 +++ 2 files changed, 38 insertions(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index cc4593cd427..4694bdb4d02 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -207,6 +207,7 @@ #include "hw/pci/msix.h" #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" +#include "migration/blocker.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -8962,6 +8963,14 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pc= i_dev, Error **errp) pcie_endpoint_cap_init(pci_dev, 0x80); pcie_cap_flr_init(pci_dev); if (n->params.sriov_max_vfs) { + if (n->migration_blocker =3D=3D NULL) { + error_setg(&n->migration_blocker, + "Migration is disabled when SR-IOV capability is se= t"); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + pcie_ari_init(pci_dev, 0x100); } =20 @@ -9025,6 +9034,14 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pc= i_dev, Error **errp) if (pci_dev->spdm_port) { uint16_t doe_offset =3D PCI_CONFIG_SPACE_SIZE; =20 + if (n->migration_blocker =3D=3D NULL) { + error_setg(&n->migration_blocker, + "Migration is disabled when SPDM responder is used"= ); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + switch (pci_dev->spdm_trans) { case SPDM_SOCKET_TRANSPORT_TYPE_PCI_DOE: if (n->params.sriov_max_vfs) { @@ -9053,10 +9070,26 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *p= ci_dev, Error **errp) } =20 if (n->params.cmb_size_mb) { + if (n->migration_blocker =3D=3D NULL) { + error_setg(&n->migration_blocker, + "Migration is disabled when CMB feature is used"); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + nvme_init_cmb(n, pci_dev); } =20 if (n->pmr.dev) { + if (n->migration_blocker =3D=3D NULL) { + error_setg(&n->migration_blocker, + "Migration is disabled when PMR feature is used"); + if (migrate_add_blocker(&n->migration_blocker, errp) < 0) { + return false; + } + } + if (!nvme_init_pmr(n, pci_dev, errp)) { return false; } @@ -9365,6 +9398,8 @@ static void nvme_exit(PCIDevice *pci_dev) } =20 memory_region_del_subregion(&n->bar0, &n->iomem); + + migrate_del_blocker(&n->migration_blocker); } =20 static const Property nvme_props[] =3D { diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index d66f7dc82d5..457b6637249 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -666,6 +666,9 @@ typedef struct NvmeCtrl { =20 /* Socket mapping to SPDM over NVMe Security In/Out commands */ int spdm_socket; + + /* Migration-related stuff */ + Error *migration_blocker; } NvmeCtrl; =20 typedef enum NvmeResetType { --=20 2.47.3 From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809484; cv=none; d=zohomail.com; s=zohoarc; b=N4m9FOmbpuLiS8Hp29bbIEKr1aG+EBCTkmtkffCTmT5RV4wIrd7wUvRmKAj6MkQV7tMuhEhQ+emThFikv4Ul/w3Idnwt0+zlzA/4BP6aPwwWyhW9OD4Goug6Vr0U8EZJjFUchEaQ5Z5yQMsg6K7kcMEaNzsD7wWdgsptqejKEt8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809484; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=gpP959L10hBsg4INOewg8w8g5XRXZUc7QBJng71AuUU=; b=ggl5j2kWfeWoxQvqr2ZgyY7jomCyZLa0gx6n3BsimcexywDKBOwCNYL9iZ5IKjq7D3qZyBlmucujtpWvufNdh0NK0CXF40kR0VtC/pYAaHm+Al0IEv8mIEiBgOo0xgp/r/E7kqp+4gBJDZdw1y0Kq+D7GO/R6kRz3RBe2tG7/YE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772809484025330.94645806066353; Fri, 6 Mar 2026 07:04:44 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWie-0007q4-4M; Fri, 06 Mar 2026 10:04:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWia-0007i8-Qu for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:56 -0500 Received: from mail-wr1-x432.google.com ([2a00:1450:4864:20::432]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWiY-0003Kz-Vb for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:56 -0500 Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-4327790c4e9so7181087f8f.2 for ; Fri, 06 Mar 2026 07:03:54 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809433; x=1773414233; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gpP959L10hBsg4INOewg8w8g5XRXZUc7QBJng71AuUU=; b=Axyj4MNH+RGBrjyeNA93sq56r1V3eRQBhUC64tYn2zv1hQV7AiNN1xaONZA3+J5crk FMxlr5U3B93fG60iAA80z2HcnSwjXQh7FG0rtDbQ2FFaoQVTjp3OsMDbHiQ5TNqnAL39 It1Hr6UcXTzbx+sm3eTDUthk3goh9Kud9vfJs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809433; x=1773414233; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=gpP959L10hBsg4INOewg8w8g5XRXZUc7QBJng71AuUU=; b=cfOJ9vibXiz/XmP8sFwI5oBDQOyPAyt5BB1eQYAg81Ntb15X2RzPBU4oUf/fJPxBLR +r7CJb4zldD+DebOELSHeA46cMRCZM7OENkPn87UEzE4u5YgtFTkUhwrQrc6uuvJNwnM VRPfFN5l3TPoq/y/nGrsKfatl5cG4ezwrrRxhjVOVVN18Eajp4Wv+Hzts8prZIVXJEXT +PTI6wDslixKT3piUf+rP1Q1xQ+OHLC3CtFifMbzSkG0SbVJowBPTdnZwC+UYAKqQO7D jouY4tTznjveg+TEHRQcJCf6tmghboR2gMWQ9C/46sdGVJPg06dE1MsXH98b3Qw1RxzY 4unQ== X-Gm-Message-State: AOJu0Yy/xuUzxSkBpYFUnBFaI5RWlLinxbduzjTAJde/6XgSh11Sm7Xq zf1vPbHfMdd1embxgYBhspge0EaJxngHBJYeO4+3X+ygQPjZBm3zuTokAJsS6b2iXn7xOYhLeXK bFRiS0Qo= X-Gm-Gg: ATEYQzx5r+HK4tWPAqFN0uiLjvtY8OAoLln/RFGiIJv//UpNXz21mwpEQMjqQqpJPb4 WzEVYr4ZzIE6+br/X3mcBthwV3bqXaqX5YQQVWIvLG78w7sd7diRb3fM5WWuj/M75F3DTz4en6B SuzT01w7UBEXsNkbiXO4VycQZP+Gbg76oGEQFuM1ssw5jjA/cQudk+2CkCjjnlCbWLo8U37QCay AU8j9C/QRnFrc8Dtn29iVYy2AYjYvwKDWIjUVEzzxS0/qTkfPi5OGwjTWVsNSpqL1aWVeXkp8nC wLMzUOGIOHlrgneAjVbXmcT6c5BYZ5JdYrMcV28ZQkX5wmIWwMPtMWNFGFgSxoke2kR3bh0BUvS fwqxWMyOtdVAs749j1kjSfQkgnmwMJCi0xUCrmDmKnnn9L7pCb3BFkaUeQYW13idyxCbFfcGZS7 erIZPPCFmmvRqma5MHhXyCmgz3c5jbXMmK7FnZk7vAsXtd/DNa19f0Z7i0IVt9BxUeTu9jFe1sa QsboZ05zUbGNmRbUJereww= X-Received: by 2002:a05:6000:24c1:b0:439:bf2f:123e with SMTP id ffacd0b85a97d-439da55aa93mr3929690f8f.11.1772809428239; Fri, 06 Mar 2026 07:03:48 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 4/6] hw/nvme: split nvme_init_sq/nvme_init_cq into helpers Date: Fri, 6 Mar 2026 16:03:40 +0100 Message-ID: <20260306150342.395923-5-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::432; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x432.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809486224154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Signed-off-by: Alexander Mikhalitsyn --- hw/nvme/ctrl.c | 57 +++++++++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 21 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 4694bdb4d02..89cc26d745b 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -4852,18 +4852,14 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t sqid, uint16_t cqid, uint16_t size) +static void __nvme_init_sq(NvmeSQueue *sq) { + NvmeCtrl *n =3D sq->ctrl; + uint16_t sqid =3D sq->sqid; + uint16_t cqid =3D sq->cqid; int i; NvmeCQueue *cq; =20 - sq->ctrl =3D n; - sq->dma_addr =3D dma_addr; - sq->sqid =3D sqid; - sq->size =3D size; - sq->cqid =3D cqid; - sq->head =3D sq->tail =3D 0; sq->io_req =3D g_new0(NvmeRequest, sq->size); =20 QTAILQ_INIT(&sq->req_list); @@ -4893,6 +4889,18 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, n->sq[sqid] =3D sq; } =20 +static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t sqid, uint16_t cqid, uint16_t size) +{ + sq->ctrl =3D n; + sq->dma_addr =3D dma_addr; + sq->sqid =3D sqid; + sq->size =3D size; + sq->cqid =3D cqid; + sq->head =3D sq->tail =3D 0; + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5553,24 +5561,16 @@ static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeReques= t *req) return NVME_SUCCESS; } =20 -static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, - uint16_t cqid, uint16_t vector, uint16_t size, - uint16_t irq_enabled) +static void __nvme_init_cq(NvmeCQueue *cq) { + NvmeCtrl *n =3D cq->ctrl; PCIDevice *pci =3D PCI_DEVICE(n); + uint16_t cqid =3D cq->cqid; =20 - if (msix_enabled(pci) && irq_enabled) { - msix_vector_use(pci, vector); + if (msix_enabled(pci) && cq->irq_enabled) { + msix_vector_use(pci, cq->vector); } =20 - cq->ctrl =3D n; - cq->cqid =3D cqid; - cq->size =3D size; - cq->dma_addr =3D dma_addr; - cq->phase =3D 1; - cq->irq_enabled =3D irq_enabled; - cq->vector =3D vector; - cq->head =3D cq->tail =3D 0; QTAILQ_INIT(&cq->req_list); QTAILQ_INIT(&cq->sq_list); if (n->dbbuf_enabled) { @@ -5588,6 +5588,21 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, &DEVICE(cq->ctrl)->mem_reentrancy_guard); } =20 +static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr, + uint16_t cqid, uint16_t vector, uint16_t size, + uint16_t irq_enabled) +{ + cq->ctrl =3D n; + cq->cqid =3D cqid; + cq->size =3D size; + cq->dma_addr =3D dma_addr; + cq->phase =3D 1; + cq->irq_enabled =3D irq_enabled; + cq->vector =3D vector; + cq->head =3D cq->tail =3D 0; + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; --=20 2.47.3 From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809734; cv=none; d=zohomail.com; s=zohoarc; b=foFrM7HgEiHxYe5GpoQQhUV8eRVCaelyvS8hgxkOzjY/DQwoqfAhCNNqXQzsWsKQTkTg76h4fVqNFOvqaWskOGE+PPoGO5zHezZCdUUMJoSeaSQpgrwfgQWSFq8lbgdEp+l4uS1dwCAZP4BlSyPo1PnUud765nU9xSBxknc3gdU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809734; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=xj9/f+SaUMvAcrT6CuXG1fVnLMkvBtAqefrCAJGy/GQ=; b=X9OClh4awq50jz+2JOW6uP0/hOWXipK31wbVacZq3oGbzS4+oByklHa7fOwmVuigWXryD0bOwa4q/tUYyTDxUKoXlIC/PyvpIr6CtHjPc4rS0/578l7U5ergtsnQ8CR0XN8UOj48kGftDLzEWyy/3pizxcNJRQ5fte9ZdruNiUI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17728097342901011.1092658119883; Fri, 6 Mar 2026 07:08:54 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWir-0008BS-Gz; Fri, 06 Mar 2026 10:04:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWid-0007rq-Ok for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:04:00 -0500 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWiZ-0003LA-RW for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:58 -0500 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-439bcec8613so4152831f8f.3 for ; Fri, 06 Mar 2026 07:03:55 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809434; x=1773414234; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xj9/f+SaUMvAcrT6CuXG1fVnLMkvBtAqefrCAJGy/GQ=; b=P9Qu6OZ/TMkBtQ1Q6XDQh4iAEyorieh0tBdL3wS/2hzOgo81A7h36OehVkZuHE+ko1 A2dVQFTNOZtx8kXaIQ0GSLsKN02Gk9+z+z2JCQ9ooXudJljApX84u+GtL78mWNvoBDF0 bQdwmpRcVHe72HKmqflLCGCA/zhI4whctX4Js= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809434; x=1773414234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=xj9/f+SaUMvAcrT6CuXG1fVnLMkvBtAqefrCAJGy/GQ=; b=MX8cLYlfkn6ykj1qn9GhchLxrFSCmtdOIbXEs5RZzdSlv9K3xbu2IVGETpKR8snNMO xSdMReWpIKJF2BMlS//KBpxeD8EVhh+M/yqMBU06fm4hBe9sxcfiSRYs2zFb5Ku4G26s 31u6y8pHuV+dnSJw3Opi7eY0rtC0nmKsnhKltxjmmMyCCoifzqh4/erQoHoYW0NFvfk+ +rfWkl/wl8nekdEgSbPINw6VxeNxh6SK8oAMpVR+5XOs2wQS0H70AUWSWTgqudsqtsZJ 4Qzi/5zI02cXn8byacK3kKxNSEWaKxjesgd0lvWIeagv15d0x3YCYJjxYzz5QMWlDLuR XVOQ== X-Gm-Message-State: AOJu0Yw41zYGqEKN8aIyIQfRYB8hKfve2Gf8rHKI5n2Li7H+OUVHLKwY 16utFokIiQ96XttRIMdzhBTbOio6QpcWOG7wvreAAnN+eGJ1sU+eOn5O5idmLXCihZcd51Iz7Hf 2ze4kg/0= X-Gm-Gg: ATEYQzx1u9ZJfMj6Sos7zfkmPKZ/1l1/uLBa8K0VBPdjiO8+lJfxpi7UuZMKMiO3I9l 1vdr9QFK1atnZBMdL5bfNJyNS4RVcbJj7tfMPsMZie8XMTmrBa2IVE6jrhhYlP9+/PbfjxhPq84 /yZmPFag02rd7sxRqDQ8OKoW5Xpe3YpMySP27Cz0lXZyliS6JKysm43DrPSccrl1lyFR0un/nx/ jgRj3HM5IdP8in1wl/8ZqJhfFXO1bi00i0BIhcye3tcsfyAgze1nnk70qgJhJZ8CTMM5jTpXojI n8RvhjoOL652F9HzQoNukSMbn2p2H6x+4UDLXulfUHzr13qj+LYPZiXc0/CIqGZ4ZFXeNj2qg+f OCH6rLmJnaqAiTm4JqSb/2n42RBezNi0NcL1RBK+nz9Z2ciHvYPIEjXDjat3TLVMk4YF3MaffZg baLEkeyhdB5aUGyGxYlSDSq0K1ElxA1kWyaGljXtQyZ7YTFwBcHjtSk7gHMZ2T75sxN64fy58Wi KcIdk2VyHuRF4x5mwbODts= X-Received: by 2002:a05:6000:4210:b0:439:ca9b:1f61 with SMTP id ffacd0b85a97d-439da351d92mr3985824f8f.17.1772809429292; Fri, 06 Mar 2026 07:03:49 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 5/6] hw/nvme: add basic live migration support Date: Fri, 6 Mar 2026 16:03:41 +0100 Message-ID: <20260306150342.395923-6-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x42a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809736049158500 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn It has some limitations: - only one NVMe namespace is supported - SMART counters are not preserved - CMB is not supported - PMR is not supported - SPDM is not supported - SR-IOV is not supported Signed-off-by: Alexander Mikhalitsyn v2: - AERs are now fully supported --- hw/nvme/ctrl.c | 573 ++++++++++++++++++++++++++++++++++++++++++- hw/nvme/nvme.h | 2 + hw/nvme/trace-events | 11 + 3 files changed, 577 insertions(+), 9 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 89cc26d745b..cdfda1d3be8 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -208,6 +208,7 @@ #include "hw/pci/pcie_sriov.h" #include "system/spdm-socket.h" #include "migration/blocker.h" +#include "migration/qemu-file-types.h" #include "migration/vmstate.h" =20 #include "nvme.h" @@ -4901,6 +4902,25 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_sq(sq); } =20 +static void nvme_restore_sq(NvmeSQueue *sq_from) +{ + NvmeCtrl *n =3D sq_from->ctrl; + NvmeSQueue *sq =3D sq_from; + + if (sq_from->sqid =3D=3D 0) { + sq =3D &n->admin_sq; + sq->ctrl =3D n; + sq->dma_addr =3D sq_from->dma_addr; + sq->sqid =3D sq_from->sqid; + sq->size =3D sq_from->size; + sq->cqid =3D sq_from->cqid; + sq->head =3D sq_from->head; + sq->tail =3D sq_from->tail; + } + + __nvme_init_sq(sq); +} + static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeRequest *req) { NvmeSQueue *sq; @@ -5603,6 +5623,27 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n= , uint64_t dma_addr, __nvme_init_cq(cq); } =20 +static void nvme_restore_cq(NvmeCQueue *cq_from) +{ + NvmeCtrl *n =3D cq_from->ctrl; + NvmeCQueue *cq =3D cq_from; + + if (cq_from->cqid =3D=3D 0) { + cq =3D &n->admin_cq; + cq->ctrl =3D n; + cq->cqid =3D cq_from->cqid; + cq->size =3D cq_from->size; + cq->dma_addr =3D cq_from->dma_addr; + cq->phase =3D cq_from->phase; + cq->irq_enabled =3D cq_from->irq_enabled; + cq->vector =3D cq_from->vector; + cq->head =3D cq_from->head; + cq->tail =3D cq_from->tail; + } + + __nvme_init_cq(cq); +} + static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeRequest *req) { NvmeCQueue *cq; @@ -7291,7 +7332,7 @@ static uint16_t nvme_dbbuf_config(NvmeCtrl *n, const = NvmeRequest *req) n->dbbuf_eis =3D eis_addr; n->dbbuf_enabled =3D true; =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { NvmeSQueue *sq =3D n->sq[i]; NvmeCQueue *cq =3D n->cq[i]; =20 @@ -7731,7 +7772,7 @@ static int nvme_atomic_write_check(NvmeCtrl *n, NvmeC= md *cmd, /* * Walk the queues to see if there are any atomic conflicts. */ - for (i =3D 1; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 1; i < n->num_queues; i++) { NvmeSQueue *sq; NvmeRequest *req; NvmeRwCmd *req_rw; @@ -7801,6 +7842,10 @@ static void nvme_process_sq(void *opaque) NvmeCmd cmd; NvmeRequest *req; =20 + if (qatomic_read(&n->stop_processing_sq)) { + return; + } + if (n->dbbuf_enabled) { nvme_update_sq_tail(sq); } @@ -7809,6 +7854,10 @@ static void nvme_process_sq(void *opaque) NvmeAtomic *atomic; bool cmd_is_atomic; =20 + if (qatomic_read(&n->stop_processing_sq)) { + return; + } + addr =3D sq->dma_addr + (sq->head << NVME_SQES); if (nvme_addr_read(n, addr, (void *)&cmd, sizeof(cmd))) { trace_pci_nvme_err_addr_read(addr); @@ -7917,12 +7966,12 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) nvme_ns_drain(ns); } =20 - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->sq[i] !=3D NULL) { nvme_free_sq(n->sq[i], n); } } - for (i =3D 0; i < n->params.max_ioqpairs + 1; i++) { + for (i =3D 0; i < n->num_queues; i++) { if (n->cq[i] !=3D NULL) { nvme_free_cq(n->cq[i], n); } @@ -8592,6 +8641,8 @@ static bool nvme_check_params(NvmeCtrl *n, Error **er= rp) params->max_ioqpairs =3D params->num_queues - 1; } =20 + n->num_queues =3D params->max_ioqpairs + 1; + if (n->namespace.blkconf.blk && n->subsys) { error_setg(errp, "subsystem support is unavailable with legacy " "namespace ('drive' property)"); @@ -8746,8 +8797,8 @@ static void nvme_init_state(NvmeCtrl *n) n->conf_msix_qsize =3D n->params.msix_qsize; } =20 - n->sq =3D g_new0(NvmeSQueue *, n->params.max_ioqpairs + 1); - n->cq =3D g_new0(NvmeCQueue *, n->params.max_ioqpairs + 1); + n->sq =3D g_new0(NvmeSQueue *, n->num_queues); + n->cq =3D g_new0(NvmeCQueue *, n->num_queues); n->temperature =3D NVME_TEMPERATURE; n->features.temp_thresh_hi =3D NVME_TEMPERATURE_WARNING; n->starttime_ms =3D qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); @@ -8990,7 +9041,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) } =20 if (n->params.msix_exclusive_bar && !pci_is_vf(pci_dev)) { - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, 0, NULL, N= ULL); + bar_size =3D nvme_mbar_size(n->num_queues, 0, NULL, NULL); memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n, "nv= me", bar_size); pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | @@ -9002,7 +9053,7 @@ static bool nvme_init_pci(NvmeCtrl *n, PCIDevice *pci= _dev, Error **errp) /* add one to max_ioqpairs to account for the admin queue pair */ if (!pci_is_vf(pci_dev)) { nr_vectors =3D n->params.msix_qsize; - bar_size =3D nvme_mbar_size(n->params.max_ioqpairs + 1, + bar_size =3D nvme_mbar_size(n->num_queues, nr_vectors, &msix_table_offset, &msix_pba_offset); } else { @@ -9552,9 +9603,513 @@ static uint32_t nvme_pci_read_config(PCIDevice *dev= , uint32_t address, int len) return pci_default_read_config(dev, address, len); } =20 +static bool nvme_ctrl_pre_save(void *opaque, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_pre_save_enter(n); + + /* ask SQ processing code not to take new requests */ + qatomic_set(&n->stop_processing_sq, true); + + /* prevent new in-flight IO from appearing */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_cancel(sq->bh); + } + + /* drain all IO */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns; + + ns =3D nvme_ns(n, i); + if (!ns) { + continue; + } + + trace_pci_nvme_pre_save_ns_drain(n, i); + nvme_ns_drain(ns); + } + + /* + * Now, we should take care of AERs. + * + * 1. Save all queued events (n->aer_queue). + * This is done automatically, see nvme_vmstate VMStateDescription. + * Here we only need to print them for debugging purpose. + * 2. Go over outstanding AER requests (n->aer_reqs) and check they are + * all have expected opcode (NVME_ADM_CMD_ASYNC_EV_REQ) and other f= ields. + * + * We must be really careful here, because in case of further QEMU NVM= e changes, + * we may break migration without noticing it, or worse, introduce sil= ent + * data corruption during migration. + */ + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_pre_save_aer(event->result.event_type, event->r= esult.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeRequest *re =3D n->aer_reqs[i]; + + /* + * Can't use assert() here, because we don't want + * to just crash QEMU when user requests a migration. + */ + if (!(re->cmd.opcode =3D=3D NVME_ADM_CMD_ASYNC_EV_REQ)) { + error_setg(errp, "re->cmd.opcode (%u) !=3D NVME_ADM_CMD_ASYNC_= EV_REQ", re->cmd.opcode); + goto err; + } + + if (!(re->ns =3D=3D NULL)) { + error_setg(errp, "re->ns !=3D NULL"); + goto err; + } + + if (!(re->sq =3D=3D &n->admin_sq)) { + error_setg(errp, "re->sq !=3D &n->admin_sq"); + goto err; + } + + if (!(re->aiocb =3D=3D NULL)) { + error_setg(errp, "re->aiocb !=3D NULL"); + goto err; + } + + if (!(re->opaque =3D=3D NULL)) { + error_setg(errp, "re->opaque !=3D NULL"); + goto err; + } + + if (!(re->atomic_write =3D=3D false)) { + error_setg(errp, "re->atomic_write !=3D false"); + goto err; + } + + if (re->sg.flags & NVME_SG_ALLOC) { + error_setg(errp, "unexpected NVME_SG_ALLOC flag in re->sg.flag= s"); + goto err; + } + } + + /* wait when all in-flight IO requests (except NVME_ADM_CMD_ASYNC_EV_R= EQ) are processed */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeRequest *req; + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + trace_pci_nvme_pre_save_sq_out_req_drain_wait(n, i, sq->head, sq->= tail, sq->size); + +wait_out_reqs: + QTAILQ_FOREACH(req, &sq->out_req_list, entry) { + if (req->cmd.opcode !=3D NVME_ADM_CMD_ASYNC_EV_REQ) { + cpu_relax(); + goto wait_out_reqs; + } + } + + trace_pci_nvme_pre_save_sq_out_req_drain_wait_end(n, i, sq->head, = sq->tail); + } + + /* wait when all IO requests completions are written to guest memory */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + trace_pci_nvme_pre_save_cq_req_drain_wait(n, i, cq->head, cq->tail= , cq->size); + + while (!QTAILQ_EMPTY(&cq->req_list)) { + /* + * nvme_post_cqes() can't do its job of cleaning cq->req_list + * when CQ is full, it means that we need to save what we have= in + * cq->req_list and restore it back on VM resume. + * + * Good thing is that this can only happen when guest hasn't + * processed CQ for a long time and at the same time, many SQEs + * are in flight. + * + * For now, let's just block migration in this rare case. + */ + if (nvme_cq_full(cq)) { + error_setg(errp, "no free space in CQ (not supported)"); + goto err; + } + + cpu_relax(); + } + + trace_pci_nvme_pre_save_cq_req_drain_wait_end(n, i, cq->head, cq->= tail); + } + + for (uint32_t nsid =3D 0; nsid <=3D NVME_MAX_NAMESPACES; nsid++) { + NvmeNamespace *ns =3D n->namespaces[nsid]; + + if (!ns) + continue; + + if (ns !=3D &n->namespace) { + error_setg(errp, "only one NVMe namespace is supported for mig= ration"); + goto err; + } + } + + return true; + +err: + /* restore sq processing back to normal */ + qatomic_set(&n->stop_processing_sq, false); + return false; +} + +static bool nvme_ctrl_post_load(void *opaque, int version_id, Error **errp) +{ + NvmeCtrl *n =3D opaque; + int i; + + trace_pci_nvme_post_load_enter(n); + + /* restore CQs first */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeCQueue *cq =3D n->cq[i]; + + if (!cq) + continue; + + cq->ctrl =3D n; + nvme_restore_cq(cq); + trace_pci_nvme_post_load_restore_cq(n, i, cq->head, cq->tail, cq->= size); + + if (i =3D=3D 0) { + /* + * Admin CQ lives in n->admin_cq, we don't need + * memory allocated for it in get_ptrs_array_entry() anymore. + * + * nvme_restore_cq() also takes care of: + * n->cq[0] =3D &n->admin_cq; + * so n->cq[0] remains valid. + */ + g_free(cq); + } + } + + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + sq->ctrl =3D n; + nvme_restore_sq(sq); + trace_pci_nvme_post_load_restore_sq(n, i, sq->head, sq->tail, sq->= size); + + if (i =3D=3D 0) { + /* same as for CQ */ + g_free(sq); + } + } + + if (n->aer_queued) { + NvmeAsyncEvent *event; + + QTAILQ_FOREACH(event, &n->aer_queue, entry) { + trace_pci_nvme_post_load_aer(event->result.event_type, event->= result.event_info, + event->result.log_page); + } + } + + for (i =3D 0; i < n->outstanding_aers; i++) { + NvmeSQueue *sq =3D &n->admin_sq; + NvmeRequest *req_from =3D n->aer_reqs[i]; + NvmeRequest *req; + + /* + * We use nvme_vmstate VMStateDescription to save/restore + * NvmeRequest structures, but tricky thing here is that + * memory for each n->aer_reqs[i] will be allocated separately + * during restore. It doesn't work for us. We need to take + * an existing NvmeRequest structure from SQ's req_list + * and fill it with data from the newly allocated one (req_from). + * Then, we can safely release allocated memory for it. + */ + + /* take an NvmeRequest struct from SQ */ + req =3D QTAILQ_FIRST(&sq->req_list); + QTAILQ_REMOVE(&sq->req_list, req, entry); + QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry); + nvme_req_clear(req); + + /* copy data from the source NvmeRequest */ + req->status =3D req_from->status; + memcpy(&req->cqe, &req_from->cqe, sizeof(NvmeCqe)); + memcpy(&req->cmd, &req_from->cmd, sizeof(NvmeCmd)); + + n->aer_reqs[i] =3D req; + g_free(req_from); + } + + /* + * We need to attach namespaces (currently, only one namespace is + * supported for migration). + * This logic comes from nvme_start_ctrl(). + */ + for (i =3D 1; i <=3D NVME_MAX_NAMESPACES; i++) { + NvmeNamespace *ns =3D nvme_subsys_ns(n->subsys, i); + + if (!ns || (!ns->params.shared && ns->ctrl !=3D n)) { + continue; + } + + if (nvme_csi_supported(n, ns->csi) && !ns->params.detached) { + if (!ns->attached || ns->params.shared) { + nvme_attach_ns(n, ns); + } + } + } + + /* schedule SQ processing */ + for (i =3D 0; i < n->num_queues; i++) { + NvmeSQueue *sq =3D n->sq[i]; + + if (!sq) + continue; + + qemu_bh_schedule(sq->bh); + } + + /* + * We ensured in pre_save() that cq->req_list was empty, + * so we don't need to schedule BH for CQ processing. + */ + + return true; +} + +static const VMStateDescription nvme_vmstate_bar =3D { + .name =3D "nvme-bar", + .minimum_version_id =3D 1, + .version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(cap, NvmeBar), + VMSTATE_UINT32(vs, NvmeBar), + VMSTATE_UINT32(intms, NvmeBar), + VMSTATE_UINT32(intmc, NvmeBar), + VMSTATE_UINT32(cc, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd24, NvmeBar, 4), + VMSTATE_UINT32(csts, NvmeBar), + VMSTATE_UINT32(nssr, NvmeBar), + VMSTATE_UINT32(aqa, NvmeBar), + VMSTATE_UINT64(asq, NvmeBar), + VMSTATE_UINT64(acq, NvmeBar), + VMSTATE_UINT32(cmbloc, NvmeBar), + VMSTATE_UINT32(cmbsz, NvmeBar), + VMSTATE_UINT32(bpinfo, NvmeBar), + VMSTATE_UINT32(bprsel, NvmeBar), + VMSTATE_UINT64(bpmbl, NvmeBar), + VMSTATE_UINT64(cmbmsc, NvmeBar), + VMSTATE_UINT32(cmbsts, NvmeBar), + VMSTATE_UINT8_ARRAY(rsvd92, NvmeBar, 3492), + VMSTATE_UINT32(pmrcap, NvmeBar), + VMSTATE_UINT32(pmrctl, NvmeBar), + VMSTATE_UINT32(pmrsts, NvmeBar), + VMSTATE_UINT32(pmrebs, NvmeBar), + VMSTATE_UINT32(pmrswtp, NvmeBar), + VMSTATE_UINT32(pmrmscl, NvmeBar), + VMSTATE_UINT32(pmrmscu, NvmeBar), + VMSTATE_UINT8_ARRAY(css, NvmeBar, 484), + VMSTATE_END_OF_LIST() + }, +}; + +static const VMStateDescription nvme_vmstate_cqueue =3D { + .name =3D "nvme-cq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(phase, NvmeCQueue), + VMSTATE_UINT16(cqid, NvmeCQueue), + VMSTATE_UINT16(irq_enabled, NvmeCQueue), + VMSTATE_UINT32(head, NvmeCQueue), + VMSTATE_UINT32(tail, NvmeCQueue), + VMSTATE_UINT32(vector, NvmeCQueue), + VMSTATE_UINT32(size, NvmeCQueue), + VMSTATE_UINT64(dma_addr, NvmeCQueue), + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_squeue =3D { + .name =3D "nvme-sq", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(sqid, NvmeSQueue), + VMSTATE_UINT16(cqid, NvmeSQueue), + VMSTATE_UINT32(head, NvmeSQueue), + VMSTATE_UINT32(tail, NvmeSQueue), + VMSTATE_UINT32(size, NvmeSQueue), + VMSTATE_UINT64(dma_addr, NvmeSQueue), + /* db_addr, ei_addr, etc will be recalculated */ + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event_result =3D { + .name =3D "nvme-async-event-result", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(event_type, NvmeAerResult), + VMSTATE_UINT8(event_info, NvmeAerResult), + VMSTATE_UINT8(log_page, NvmeAerResult), + VMSTATE_UINT8(resv, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_async_event =3D { + .name =3D "nvme-async-event", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_STRUCT(result, NvmeAsyncEvent, 0, nvme_vmstate_async_event= _result, NvmeAerResult), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cqe =3D { + .name =3D "nvme-cqe", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(result, NvmeCqe), + VMSTATE_UINT32(dw1, NvmeCqe), + VMSTATE_UINT16(sq_head, NvmeCqe), + VMSTATE_UINT16(sq_id, NvmeCqe), + VMSTATE_UINT16(cid, NvmeCqe), + VMSTATE_UINT16(status, NvmeCqe), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr_sgl =3D { + .name =3D "nvme-request-cmd-dptr-sgl", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(addr, NvmeSglDescriptor), + VMSTATE_UINT32(len, NvmeSglDescriptor), + VMSTATE_UINT8_ARRAY(rsvd, NvmeSglDescriptor, 3), + VMSTATE_UINT8(type, NvmeSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd_dptr =3D { + .name =3D "nvme-request-cmd-dptr", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT64(prp1, NvmeCmdDptr), + VMSTATE_UINT64(prp2, NvmeCmdDptr), + VMSTATE_STRUCT(sgl, NvmeCmdDptr, 0, nvme_vmstate_cmd_dptr_sgl, Nvm= eSglDescriptor), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_cmd =3D { + .name =3D "nvme-request-cmd", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT8(opcode, NvmeCmd), + VMSTATE_UINT8(flags, NvmeCmd), + VMSTATE_UINT16(cid, NvmeCmd), + VMSTATE_UINT32(nsid, NvmeCmd), + VMSTATE_UINT64(res1, NvmeCmd), + VMSTATE_UINT64(mptr, NvmeCmd), + VMSTATE_STRUCT(dptr, NvmeCmd, 0, nvme_vmstate_cmd_dptr, NvmeCmdDpt= r), + VMSTATE_UINT32(cdw10, NvmeCmd), + VMSTATE_UINT32(cdw11, NvmeCmd), + VMSTATE_UINT32(cdw12, NvmeCmd), + VMSTATE_UINT32(cdw13, NvmeCmd), + VMSTATE_UINT32(cdw14, NvmeCmd), + VMSTATE_UINT32(cdw15, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription nvme_vmstate_request =3D { + .name =3D "nvme-request", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT16(status, NvmeRequest), + VMSTATE_STRUCT(cqe, NvmeRequest, 0, nvme_vmstate_cqe, NvmeCqe), + VMSTATE_STRUCT(cmd, NvmeRequest, 0, nvme_vmstate_cmd, NvmeCmd), + VMSTATE_END_OF_LIST() + } +}; + static const VMStateDescription nvme_vmstate =3D { .name =3D "nvme", - .unmigratable =3D 1, + .minimum_version_id =3D 1, + .version_id =3D 1, + .pre_save_errp =3D nvme_ctrl_pre_save, + .post_load_errp =3D nvme_ctrl_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_PCI_DEVICE(parent_obj, NvmeCtrl), + VMSTATE_MSIX(parent_obj, NvmeCtrl), + VMSTATE_STRUCT(bar, NvmeCtrl, 0, nvme_vmstate_bar, NvmeBar), + + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + sq, NvmeCtrl, num_queues, 0, nvme_vmstate_squeue, NvmeSQueue), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT32_ALLOC( + cq, NvmeCtrl, num_queues, 0, nvme_vmstate_cqueue, NvmeCQueue), + + VMSTATE_BOOL(qs_created, NvmeCtrl), + VMSTATE_UINT32(page_size, NvmeCtrl), + VMSTATE_UINT16(page_bits, NvmeCtrl), + VMSTATE_UINT16(max_prp_ents, NvmeCtrl), + VMSTATE_UINT32(max_q_ents, NvmeCtrl), + VMSTATE_UINT8(outstanding_aers, NvmeCtrl), + VMSTATE_UINT32(irq_status, NvmeCtrl), + VMSTATE_INT32(cq_pending, NvmeCtrl), + + VMSTATE_UINT64(host_timestamp, NvmeCtrl), + VMSTATE_UINT64(timestamp_set_qemu_clock_ms, NvmeCtrl), + VMSTATE_UINT64(starttime_ms, NvmeCtrl), + VMSTATE_UINT16(temperature, NvmeCtrl), + VMSTATE_UINT8(smart_critical_warning, NvmeCtrl), + + VMSTATE_UINT32(conf_msix_qsize, NvmeCtrl), + VMSTATE_UINT32(conf_ioqpairs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_dbs, NvmeCtrl), + VMSTATE_UINT64(dbbuf_eis, NvmeCtrl), + VMSTATE_BOOL(dbbuf_enabled, NvmeCtrl), + + VMSTATE_UINT8(aer_mask, NvmeCtrl), + VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT8_ALLOC( + aer_reqs, NvmeCtrl, outstanding_aers, 0, nvme_vmstate_request,= NvmeRequest), + VMSTATE_QTAILQ_V(aer_queue, NvmeCtrl, 1, nvme_vmstate_async_event, + NvmeAsyncEvent, entry), + VMSTATE_INT32(aer_queued, NvmeCtrl), + + VMSTATE_END_OF_LIST() + }, }; =20 static void nvme_class_init(ObjectClass *oc, const void *data) diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 457b6637249..9c5f53c688c 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -638,6 +638,7 @@ typedef struct NvmeCtrl { =20 NvmeNamespace namespace; NvmeNamespace *namespaces[NVME_MAX_NAMESPACES + 1]; + uint32_t num_queues; NvmeSQueue **sq; NvmeCQueue **cq; NvmeSQueue admin_sq; @@ -669,6 +670,7 @@ typedef struct NvmeCtrl { =20 /* Migration-related stuff */ Error *migration_blocker; + bool stop_processing_sq; } NvmeCtrl; =20 typedef enum NvmeResetType { diff --git a/hw/nvme/trace-events b/hw/nvme/trace-events index 6be0bfa1c1f..8e5544e0008 100644 --- a/hw/nvme/trace-events +++ b/hw/nvme/trace-events @@ -7,6 +7,17 @@ pci_nvme_dbbuf_config(uint64_t dbs_addr, uint64_t eis_addr= ) "dbs_addr=3D0x%"PRIx64 pci_nvme_map_addr(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %"PRI= u64"" pci_nvme_map_addr_cmb(uint64_t addr, uint64_t len) "addr 0x%"PRIx64" len %= "PRIu64"" pci_nvme_map_prp(uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t= prp2, int num_prps) "trans_len %"PRIu64" len %"PRIu32" prp1 0x%"PRIx64" pr= p2 0x%"PRIx64" num_prps %d" +pci_nvme_pre_save_enter(void *n) "n=3D%p" +pci_nvme_pre_save_ns_drain(void *n, int i) "n=3D%p i=3D%d" +pci_nvme_pre_save_sq_out_req_drain_wait(void *n, int i, uint32_t head, uin= t32_t tail, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRI= x32" size=3D0x%"PRIx32"" +pci_nvme_pre_save_sq_out_req_drain_wait_end(void *n, int i, uint32_t head,= uint32_t tail) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32"" +pci_nvme_pre_save_cq_req_drain_wait(void *n, int i, uint32_t head, uint32_= t tail, uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32"= size=3D0x%"PRIx32"" +pci_nvme_pre_save_cq_req_drain_wait_end(void *n, int i, uint32_t head, uin= t32_t tail) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32"" +pci_nvme_pre_save_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type 0= x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" +pci_nvme_post_load_enter(void *n) "n=3D%p" +pci_nvme_post_load_restore_cq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_restore_sq(void *n, int i, uint32_t head, uint32_t tail= , uint32_t size) "n=3D%p i=3D%d head=3D0x%"PRIx32" tail=3D0x%"PRIx32" size= =3D0x%"PRIx32"" +pci_nvme_post_load_aer(uint8_t typ, uint8_t info, uint8_t log_page) "type = 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8"" pci_nvme_map_sgl(uint8_t typ, uint64_t len) "type 0x%"PRIx8" len %"PRIu64"" pci_nvme_io_cmd(uint16_t cid, uint32_t nsid, uint16_t sqid, uint8_t opcode= , const char *opname) "cid %"PRIu16" nsid 0x%"PRIx32" sqid %"PRIu16" opc 0x= %"PRIx8" opname '%s'" pci_nvme_admin_cmd(uint16_t cid, uint16_t sqid, uint8_t opcode, const char= *opname) "cid %"PRIu16" sqid %"PRIu16" opc 0x%"PRIx8" opname '%s'" --=20 2.47.3 From nobody Mon Apr 13 12:13:33 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=mihalicyn.com ARC-Seal: i=1; a=rsa-sha256; t=1772809654; cv=none; d=zohomail.com; s=zohoarc; b=Ljib0gSg6xZTwscU/9kHJY8Unm28ASEvJkF2Wlq7BoD+hKRoed6WoCv8zBZPnh/6TiVKPjXqtR+BB+HDT3UQVGuc4MoLlQLjcBGnyXAvmhvOV+D1DBhEmaOYB5E8swG67BLkdhgj2ie94JRdc7oQt8BuC1zpW/Ri6t15C3hmFXE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772809654; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=9BVjzpz+g8zyE8326nn6m7L9ExlEPRzKWbaO0sFI7v0=; b=BH13Q8JjXeuzEZRXayp8+UeKLy/vU9nc8ZIvgTufNEPPurFv1RHVcbjo47CPpkyQk6vGSfTBoUASeuHwYN4dN24ueaDOUWDqw3KMHTU5nMl5zHs2Ys6s+w69XH81RZbxTJw1+cMmye699jvp85d6exUfi8YSuoMfGcmAK2wUxVc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772809654908133.72732492946454; Fri, 6 Mar 2026 07:07:34 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vyWip-00083z-8L; Fri, 06 Mar 2026 10:04:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vyWic-0007q7-Hq for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:04:00 -0500 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vyWia-0003LE-5g for qemu-devel@nongnu.org; Fri, 06 Mar 2026 10:03:58 -0500 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-439c5cce2c6so3208855f8f.3 for ; Fri, 06 Mar 2026 07:03:55 -0800 (PST) Received: from alex-laptop.lan (p200300cf57228c00583f1937c9b9cecc.dip0.t-ipconnect.de. [2003:cf:5722:8c00:583f:1937:c9b9:cecc]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae36785sm4703623f8f.27.2026.03.06.07.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 07:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mihalicyn.com; s=mihalicyn; t=1772809434; x=1773414234; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9BVjzpz+g8zyE8326nn6m7L9ExlEPRzKWbaO0sFI7v0=; b=SnlUqC+oH2MgfSef53a8fVo0G8PfAAs12DJneZ9BzY3v34LLFqAYL8I3NLqASiW2+W f7UEh5h5zE5hWCuT1kmQB7+N2AbTqM8gU0IpJe9nCfBEsvj+wgDe+p9eOp8SUcXPfHca LeDyUtPE4ZXwW1cQKmDT4o2G57197ACdeMeE0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772809434; x=1773414234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9BVjzpz+g8zyE8326nn6m7L9ExlEPRzKWbaO0sFI7v0=; b=GQdK4qTtBoIWF3WYSdS/ccr5aZeN49/+GEd1SuJOm1ZRFd/Z0iMm1EkIrUcUp+he7j jFpvjF3EpgVsjDSnTBM4gUvzHgFvXcG5Pew4qkAI0RmUMWl7RnYf+6kmYtiB2wmNZRL1 YiyjciBGFu2tpu6PG2s2h66q5y74P8jIFAOoiSytxP+zWh9JDbTllhnbAcT8ijxl/vzC j49rVPuH0jfNnYJlcXSNtwpEC38D5T6OKL9n2h36nFMipFESZGT2yn7bPL6CRBDn2tZN xG/LEd6hqI4KhVaHUGmESBAxtI400dAUsfhFEa5m1En/JW503KO/0zqXPKBOYkEpZN04 jv2w== X-Gm-Message-State: AOJu0YyboEA2FrMW0g0fK4JzZe/PlYsUDiFbUNROjKs9pF7PlSuLg+zd Zfpfse4oKEbXjJgizIzCqMVZZcTiWfgMhebihEu5zdMng2V5WZ9OJ9WYQQpxzn+PI8FhCHj+rnr oXup8734= X-Gm-Gg: ATEYQzzm92H7IY3fSe3Lly2iXuoIrttjWQDWHO1PfXllvZfhF7O8I3IcpAZrh9VKGEv qtF0lJnzmHSpH/r1HyO888GdX9O+YqXE1f068ctmSmV0HBwhfupilB4elH7qaByf/0dzrLF2tJ7 Y3pRhaAT4ut7+GR+yTWEtKvBMB7PO2Xj048cxNRWcs55CIYbi4OKdH8I/d/SAHWuNunApFhhzPo u4KfwbEFLBkBePbxgxfij2ljUuaYI/R9MDDWIG9Bb2DabO7SVPd+biz6bxVkmwxQgVV6VFdDyfC BH73oQgxhD8vteXY8Z63XajHxdNdDuj/rfUooBwyVD1rHhBUFmqORb9ad9pcQmm7ry2lqoHhmK0 OMoQRAKrByYKmVg8+gZvs8JPURDXBLVDhB+/zP5MXPsBVPVdiTD45b1XtTBHvNnhLdmeliFI69J 4KtLVmZgVp8kgP2RxALU7HUabD/NtttoeY21cWalhp9mKfaNmBqZe0vgIqY8b6r04EL1pIBUGlF GEPMDnbbmPAyEYOzPt4cmI= X-Received: by 2002:a05:6000:604:b0:439:b715:6f4b with SMTP id ffacd0b85a97d-439da8a78d1mr4395120f8f.57.1772809430264; Fri, 06 Mar 2026 07:03:50 -0800 (PST) From: Alexander Mikhalitsyn To: qemu-devel@nongnu.org Cc: Zhao Liu , Jesper Devantier , Alexander Mikhalitsyn , Klaus Jensen , =?UTF-8?q?St=C3=A9phane=20Graber?= , Paolo Bonzini , qemu-block@nongnu.org, Keith Busch , Peter Xu , Fabiano Rosas , Alexander Mikhalitsyn Subject: [PATCH v3 6/6] tests/functional/x86_64: add migration test for NVMe device Date: Fri, 6 Mar 2026 16:03:42 +0100 Message-ID: <20260306150342.395923-7-alexander@mihalicyn.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260306150342.395923-1-alexander@mihalicyn.com> References: <20260306150342.395923-1-alexander@mihalicyn.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::434; envelope-from=alexander@mihalicyn.com; helo=mail-wr1-x434.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @mihalicyn.com) X-ZM-MESSAGEID: 1772809658829154100 Content-Type: text/plain; charset="utf-8" From: Alexander Mikhalitsyn Introduce a very simple test to ensure that NVMe device migration works fine. Test plan is simple: 1. prepare VM with NVMe device 2. run workload that produces relatively heavy IO on the device 3. migrate VM 4. ensure that workload is alive and finishes without errors Test can be run as simple as: $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build In the future we can extend this approach, and introduce some fio-based tests. And probably, it makes sense to make this test to apply not only to NVMe device, but also virtio-{blk,scsi}, ide, sata and other migratable devices. Signed-off-by: Alexander Mikhalitsyn --- tests/functional/x86_64/meson.build | 1 + .../functional/x86_64/test_nvme_migration.py | 159 ++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100755 tests/functional/x86_64/test_nvme_migration.py diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/= meson.build index 05e4914c772..a3b010d24c6 100644 --- a/tests/functional/x86_64/meson.build +++ b/tests/functional/x86_64/meson.build @@ -30,6 +30,7 @@ tests_x86_64_system_thorough =3D [ 'linux_initrd', 'multiprocess', 'netdev_ethtool', + 'nvme_migration', 'replay', 'reverse_debug', 'tuxrun', diff --git a/tests/functional/x86_64/test_nvme_migration.py b/tests/functio= nal/x86_64/test_nvme_migration.py new file mode 100755 index 00000000000..3788a8e3473 --- /dev/null +++ b/tests/functional/x86_64/test_nvme_migration.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# x86_64 NVMe migration test + +from migration import MigrationTest +from qemu_test import QemuSystemTest, Asset +from qemu_test import wait_for_console_pattern +from qemu_test import exec_command, exec_command_and_wait_for_pattern + + +class X8664NVMeMigrationTest(MigrationTest): + ASSET_KERNEL =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/vmlinuz'), + 'd4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129') + + ASSET_INITRD =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Server/x86_64/os/images/pxeboot/initrd.img'), + '277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b') + + ASSET_DISKIMAGE =3D Asset( + ('https://archives.fedoraproject.org/pub/archive/fedora/linux/rele= ases' + '/31/Cloud/x86_64/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2'), + 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0') + + DEFAULT_KERNEL_PARAMS =3D ('root=3D/dev/nvme0n1p1 console=3DttyS0 net.= ifnames=3D0 ' + 'rd.rescue quiet') + + def wait_for_console_pattern(self, success_message, vm): + wait_for_console_pattern( + self, + success_message, + failure_message=3D"Kernel panic - not syncing", + vm=3Dvm, + ) + + def exec_command_and_check(self, command, vm): + prompt =3D '# ' + exec_command_and_wait_for_pattern(self, + f"{command} && echo OK || echo FAI= L", + 'FAIL', vm=3Dvm) + # Note, that commands we send to the console are echo-ed back, so = if we have a word "FAIL" + # in the command itself, we should expect to see it once. + wait_for_console_pattern(self, 'OK', failure_message=3D"FAIL", vm= =3Dvm) + self.wait_for_console_pattern(prompt, vm) + + def configure_machine(self, vm): + kernel_path =3D self.ASSET_KERNEL.fetch() + initrd_path =3D self.ASSET_INITRD.fetch() + diskimage_path =3D self.ASSET_DISKIMAGE.fetch() + + vm.set_console() + vm.add_args("-cpu", "max") + vm.add_args("-m", "2G") + vm.add_args("-accel", "kvm") + + vm.add_args('-drive', + f'file=3D{diskimage_path},if=3Dnone,id=3Ddrv0,sna= pshot=3Don') + vm.add_args('-device', 'nvme,bus=3Dpcie.0,' + + 'drive=3Ddrv0,id=3Dnvme-disk0,serial=3Dnvmemigrat= etest,bootindex=3D1') + + vm.add_args( + "-kernel", + kernel_path, + "-initrd", + initrd_path, + "-append", + self.DEFAULT_KERNEL_PARAMS + ) + + def launch_source_vm(self, vm): + vm.launch() + + self.wait_for_console_pattern('Entering emergency mode.', vm) + prompt =3D '# ' + self.wait_for_console_pattern(prompt, vm) + + # Synchronize on NVMe driver creating the root device + exec_command_and_wait_for_pattern(self, + "while ! (dmesg -c | grep nvme0n1:) ; do sleep 1 ;= done", + "nvme0n1", vm=3Dvm) + self.wait_for_console_pattern(prompt, vm) + + # prepare system + exec_command_and_wait_for_pattern(self, 'mount /dev/nvme0n1p1 /sys= root', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'chroot /sysroot', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t proc proc /proc', + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs /sys= ', + prompt, vm=3Dvm) + + # Run workload before migration to check if it continues to run pr= operly after migration + # + # Workload is simple: it continuously calculates checksums of all = files in /usr/bin + # to generate some I/O load on the NVMe disk and at the same time = it drops caches to + # make sure that we have some read I/O on the disk as well. + # If there are any issues with the migration of the NVMe device, w= e should see errors + # in dmesg and consequently in the workload log. + exec_command_and_wait_for_pattern(self, + "(while [ ! -f /tmp/test_nvme_migr= ation_workload.stop ]; do \ + rm -f /tmp/test_nvme_migration= _workload.iteration_finished; \ + echo 3 > /proc/sys/vm/drop_cac= hes; \ + find /usr/bin -type f -exec ck= sum {} \\;; \ + touch /tmp/test_nvme_migration= _workload.iteration_finished; \ + done) > /dev/null 2> /tmp/test_nvm= e_migration_workload.errors &", + prompt, vm=3Dvm) + exec_command_and_wait_for_pattern(self, 'echo $! > /tmp/test_nvme_= migration_workload.pid', + prompt, vm=3Dvm) + + # check if process is alive and running + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + def assert_dest_vm(self, vm): + prompt =3D '# ' + + # check if process is alive and running after migration, if not - = fail the test + self.exec_command_and_check("kill -0 $(cat /tmp/test_nvme_migratio= n_workload.pid)", vm) + + # signal workload to stop + exec_command_and_wait_for_pattern(self, 'touch /tmp/test_nvme_migr= ation_workload.stop', + prompt, vm=3Dvm) + + # wait workload to finish, because we want to examine log to see i= f there are any errors + exec_command_and_wait_for_pattern(self, + "while [ ! -f /tmp/test_nvme_migra= tion_workload.iteration_finished ]; do sleep 1; done;", + prompt, vm=3Dvm) + + exec_command_and_wait_for_pattern(self, 'cat /tmp/test_nvme_migrat= ion_workload.errors', + prompt, vm=3Dvm) + + # fail the test if non-empty + self.exec_command_and_check("[ ! -s /tmp/test_nvme_migration_workl= oad.errors ]", vm) + + def test_migration_with_tcp_localhost(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_tcp_localhost() + + def test_migration_with_unix(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_unix() + + def test_migration_with_exec(self): + self.set_machine('q35') + self.require_accelerator("kvm") + + self.migration_with_exec() + + +if __name__ =3D=3D '__main__': + MigrationTest.main() --=20 2.47.3