[RFC 10/14] vfio/nvgrace-egm: Clear Memory before handing out to VM

ankita@nvidia.com posted 14 patches 4 weeks, 1 day ago
[RFC 10/14] vfio/nvgrace-egm: Clear Memory before handing out to VM
Posted by ankita@nvidia.com 4 weeks, 1 day ago
From: Ankit Agrawal <ankita@nvidia.com>

The EGM region is invisible to the host Linux kernel and it does not
manage the region. The EGM module manages the EGM memory and thus is
responsible to clear out the region before handing out to the VM.

Clear EGM region on EGM chardev open. It is possible to trigger open
multiple times by tools such as kvmtool. Thus ensure the region is
cleared only on the first open.

Suggested-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 drivers/vfio/pci/nvgrace-gpu/egm.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c
index 7bf6a05aa967..bf1241ed1d60 100644
--- a/drivers/vfio/pci/nvgrace-gpu/egm.c
+++ b/drivers/vfio/pci/nvgrace-gpu/egm.c
@@ -15,6 +15,7 @@ static DEFINE_XARRAY(egm_chardevs);
 struct chardev {
 	struct device device;
 	struct cdev cdev;
+	atomic_t open_count;
 };
 
 static struct nvgrace_egm_dev *
@@ -30,6 +31,26 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file)
 {
 	struct chardev *egm_chardev =
 		container_of(inode->i_cdev, struct chardev, cdev);
+	struct nvgrace_egm_dev *egm_dev =
+		egm_chardev_to_nvgrace_egm_dev(egm_chardev);
+	void *memaddr;
+
+	if (atomic_inc_return(&egm_chardev->open_count) > 1)
+		return 0;
+
+	/*
+	 * nvgrace-egm module is responsible to manage the EGM memory as
+	 * the host kernel has no knowledge of it. Clear the region before
+	 * handing over to userspace.
+	 */
+	memaddr = memremap(egm_dev->egmphys, egm_dev->egmlength, MEMREMAP_WB);
+	if (!memaddr) {
+		atomic_dec(&egm_chardev->open_count);
+		return -EINVAL;
+	}
+
+	memset((u8 *)memaddr, 0, egm_dev->egmlength);
+	memunmap(memaddr);
 
 	file->private_data = egm_chardev;
 
@@ -38,7 +59,11 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file)
 
 static int nvgrace_egm_release(struct inode *inode, struct file *file)
 {
-	file->private_data = NULL;
+	struct chardev *egm_chardev =
+		container_of(inode->i_cdev, struct chardev, cdev);
+
+	if (atomic_dec_and_test(&egm_chardev->open_count))
+		file->private_data = NULL;
 
 	return 0;
 }
@@ -96,6 +121,7 @@ setup_egm_chardev(struct nvgrace_egm_dev *egm_dev)
 	egm_chardev->device.parent = &egm_dev->aux_dev.dev;
 	cdev_init(&egm_chardev->cdev, &file_ops);
 	egm_chardev->cdev.owner = THIS_MODULE;
+	atomic_set(&egm_chardev->open_count, 0);
 
 	ret = dev_set_name(&egm_chardev->device, "egm%lld", egm_dev->egmpxm);
 	if (ret)
-- 
2.34.1
Re: [RFC 10/14] vfio/nvgrace-egm: Clear Memory before handing out to VM
Posted by Jason Gunthorpe 3 weeks, 6 days ago
On Thu, Sep 04, 2025 at 04:08:24AM +0000, ankita@nvidia.com wrote:
> From: Ankit Agrawal <ankita@nvidia.com>
> 
> The EGM region is invisible to the host Linux kernel and it does not
> manage the region. The EGM module manages the EGM memory and thus is
> responsible to clear out the region before handing out to the VM.
> 
> Clear EGM region on EGM chardev open. It is possible to trigger open
> multiple times by tools such as kvmtool. Thus ensure the region is
> cleared only on the first open.

It would be cleaner not to support multi-open, why is kvmtool doing
this?

Jason