From nobody Mon Feb 9 01:45:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C131835F8B7 for ; Thu, 29 Jan 2026 21:25:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769721958; cv=none; b=Sh6q7txgI6JkftT29Osu+evlU9vs5BlVZjr3o5/T3f5Cdli2YrNgVhbA4qR8xc8UvaoLd0vghEiKkbaRcmaK1JB7blRAMsDEJ9ZGV9W8yMdxVpzgLyaxqPg9SfJREaGbDDwmn5Hlb6HP9/FIOV8Tm/T5+yOLhHkp4wtX8oNDqdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769721958; c=relaxed/simple; bh=Fptw60PqRkYzWIuTBN2mK+X1wys6jPz6VcpFsR3yJlM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=e5Q5kihq9VKZ4Mgv2I2eDYrZyaEGo5oIC9pCO8QET9gRTLjmG/eKH1fPy5F1OFsAVY1WHdRt3CCkwLrOIlkdkTANYXYIAN7GyaKEiiTP72tKx5bfVi6CSDz+OOwptTXJ918yQxR+NXJW9avnqeVh+yejbImDqhuNm9ftTDUa2LY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aP8+0eiA; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aP8+0eiA" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-34c704d5d15so2642608a91.1 for ; Thu, 29 Jan 2026 13:25:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769721956; x=1770326756; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ibCvH5eZEobsLpnry66xHV5xBXd91tAWSqhhlaOXScs=; b=aP8+0eiArsIVTvtj7o4LSZBOiYFcpm8isKZLT2N+x+NokzE1S91UrUx2UMIGAt30gw 5hDPtgzFsLczo/KKP7qTLwA2Ywm/oA/qL2DPBHoiRyFayDjk8RtnzfjRkaJQvgN+5upp YgxKRwWUSrtL6Ap+r/ohSX5tDHFa3n8CzRb8Oe4e+ab+S3Nd60Sfowe+tboV5/BPWn93 0/rruoBRES59bxrvygKvX62Lm31AJAa2z7Ucftv9W1vSlKuIDI04oix8WwVjHMOAeYqe XqOTUV++UcionZ9h9ekwVJ5IrqdlvwKDp7lCKnb64VYdBfnLJKKa7oIxufYJId0Uqkao fg6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769721956; x=1770326756; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ibCvH5eZEobsLpnry66xHV5xBXd91tAWSqhhlaOXScs=; b=pnfs8arngxJW6DE9tHZAU1G3z+JZjWcN77HA7ydaNNvyRZkiDPVCwQgb8uXqwnTUg7 jg2rWreD1fwFQ18pEXeX7FsXDXh/AXOumXGWu2ga8tYP+aMbe0qrUlSprdWjW08+ztef Gi3q7SwDVuUzZWu+fMR2QF6xrocWx8tmBiToBmsPZIZsqxGx/wOtVQjN/LyZ2JAMU+4Z 8aRAOUw+4RkxhOXr58LMimC6uyonv+BbWufPAtPyhESX0EKbgDpZjd1kgcvBz7epdKIn iL3AZxZ+hfJk2E3yK+nAZx4QbOGJR/VMn0Sh9fMW54Nkki9HKLzJYTza6KRej4sjF8g7 1TXg== X-Forwarded-Encrypted: i=1; AJvYcCUs6mJxjhJfuZ/xjmfI3L0f3nQHPKamGXLJ/CpRX/2lvxNrtKPMCAhI9/68Vh7EuXbndqxGpy94uDxF0yQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyroMJzyfEGGEyEJ5v/U/NfmRh+R7vsai6Fpsm4Pr8FTFU4KMLr xu7fRdpIgoatcIyshLYO6YCMq8Z6ojEelHdN+51ZBlewm3RCAdHgRlKxwBX/mYHwL/T2WGR9y1o mD3wUgcjFFXuLzQ== X-Received: from pjbpv18.prod.google.com ([2002:a17:90b:3c92:b0:33b:ba24:b207]) (user=dmatlack job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3ccd:b0:352:bd7c:ddbd with SMTP id 98e67ed59e1d1-3543b38af32mr802532a91.23.1769721955711; Thu, 29 Jan 2026 13:25:55 -0800 (PST) Date: Thu, 29 Jan 2026 21:24:58 +0000 In-Reply-To: <20260129212510.967611-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260129212510.967611-1-dmatlack@google.com> X-Mailer: git-send-email 2.53.0.rc1.225.gd81095ad13-goog Message-ID: <20260129212510.967611-12-dmatlack@google.com> Subject: [PATCH v2 11/22] docs: liveupdate: Document VFIO device file preservation From: David Matlack To: Alex Williamson Cc: Adithya Jayachandran , Alexander Graf , Alex Mastro , Alistair Popple , Andrew Morton , Ankit Agrawal , Bjorn Helgaas , Chris Li , David Matlack , David Rientjes , Jacob Pan , Jason Gunthorpe , Jason Gunthorpe , Jonathan Corbet , Josh Hilke , Kevin Tian , kexec@lists.infradead.org, kvm@vger.kernel.org, Leon Romanovsky , Leon Romanovsky , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, Lukas Wunner , "=?UTF-8?q?Micha=C5=82=20Winiarski?=" , Mike Rapoport , Parav Pandit , Pasha Tatashin , Pranjal Shrivastava , Pratyush Yadav , Raghavendra Rao Ananta , Rodrigo Vivi , Saeed Mahameed , Samiullah Khawaja , Shuah Khan , "=?UTF-8?q?Thomas=20Hellstr=C3=B6m?=" , Tomita Moeko , Vipin Sharma , Vivek Kasireddy , William Tu , Yi Liu , Zhu Yanjun Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation for preserving VFIO device files across a Live Update, as well as some generic file preservation documentation. This documentation will be extended in the future as new types of files are supported and new dependency/ordering requirements are added. Signed-off-by: David Matlack --- Documentation/userspace-api/liveupdate.rst | 144 +++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/Documentation/userspace-api/liveupdate.rst b/Documentation/use= rspace-api/liveupdate.rst index 41c0473e4f16..dbf1e4aeddd7 100644 --- a/Documentation/userspace-api/liveupdate.rst +++ b/Documentation/userspace-api/liveupdate.rst @@ -14,6 +14,150 @@ ioctl uAPI =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D .. kernel-doc:: include/uapi/linux/liveupdate.h =20 +File Preservation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Files can be preserved across Live Update in sessions. Since only one proc= ess +can open /dev/liveupdate, sessions must be created by a centralized process +(e.g. "luod") and then passed via UDS to lower privilege processes (e.g. V= MMs) +for them to preserve their own files. + +luod:: + + luo_fd =3D open("/dev/liveupdate", ...); + + ... + + // Create a new session with the given name. + struct liveupdate_ioctl_create_session arg =3D { + .size =3D sizeof(arg), + .name =3D SESSION_NAME, + }; + ioctl(luo_fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &arg); + + // Send session_fd to the VMM over UDS. + send_session_fd(..., arg.fd); + +VMM:: + + // Receive the newly created session from luod over UDS + session_fd =3D create_session(SESSION_NAME); + + ... + + // Preserve a file with a unique token value in the session. + struct liveupdate_session_preserve_fd arg =3D { + .size =3D sizeof(arg), + .fd =3D fd, + .token =3D TOKEN, + } + ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &arg); + +Files can be unpreserved with the LIVEUPDATE_SESSION_UNPRESERVE_FD ioctl. = They +are also unpreserved once the last reference to the session is dropped. To +carry preserved files across a Live Update, references must be kept on the +session files through the reboot(LINUX_REBOOT_CMD_KEXEC) syscall. + +While a file is preserved in a session, the kernel holds an extra reference +to it to prevent it from being destroyed. + +Only the following types of files support LIVEUPDATE_SESSION_PRESERVE_FD. = More +types of files are expected to be added in the future. + + - memfd + - VFIO character device files (vfio-pci only) + +File Retrieval +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Files that are preserved in a session retrieved after +reboot(LINUX_REBOOT_CMD_KEXEC). + +luod:: + + luo_fd =3D open("/dev/liveupdate", ...); + + ... + + struct liveupdate_ioctl_retrieve_session arg =3D { + .size =3D sizeof(arg), + .name =3D SESSION_NAME, + }; + ioctl(luo_fd, LIVEUPDATE_IOCTL_RETRIEVE_SESSION, &arg); + + // Send session_fd to VMM over UDS. + send_session_fd(..., arg.fd); + +VMM:: + + // Receive the retrieved session from luod over UDS + session_fd =3D retrieve_session(SESSION_NAME); + + ... + + // Retrieve the file associated with the token from the session. + struct liveupdate_session_retrieve_fd arg =3D { + .size =3D sizeof(arg), + .token =3D TOKEN, + }; + ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg); + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...); + +A session can only be finished once all of the files within it have been +retrieved, and are fully restored from the kernel's perspective. The exact +requirements will vary by file type. + +VFIO Character Device (cdev) Files +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The kernel supports preserving VFIO character device files across Live Upd= ate +within a session:: + + device_fd =3D open("/dev/vfio/devices/X"); + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, { ..., device_fd, ...}= ); + +Attempting to preserve files acquired via VFIO_GROUP_GET_DEVICE_FD will fa= il. + +Since the kernel holds an extra reference to files preserved in sessions, = there +is no way for the underlying PCI device to be unbound from vfio-pci while = it +is being preserved. + +When a VFIO device file is preserved in a session, interrupts must be disa= bled +on the device prior to reboot(LINUX_REBOOT_CMD_KEXEC), or the kexec will f= ail. + +Preserved VFIO device files can be retrieved after a Live Update just like= any +other preserved file:: + + ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg); + device_fd =3D arg.fd; + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...); + +Prior to LIVEUPDATE_SESSION_FINISH, preserved devices must be retrieved fr= om +the session and bound to an iommufd. Attempting to open the device through +its character device (/dev/vfio/devices/X) or VFIO_GROUP_GET_DEVICE_FD will +fail with -EBUSY. + +The eventual goal of these support is to preserve devices running uninterr= upted +across a Live Update. However there are many steps still needed to achieve= this +(see Future Work below). So for now, VFIO will reset and restore the device +back into an idle state during reboot(LINUX_REBOOT_CMD_KEXEC). + +Future work: + + - Preservation of iommufd files + - Preservation of IOMMU driver state + - Preservation of PCI state (BAR resources, device state, bridge state, .= ..) + - Preservation of vfio-pci driver state + See Also =3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.53.0.rc1.225.gd81095ad13-goog