From nobody Sun Nov 24 02:51:41 2024 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B4CB21B441 for ; Fri, 8 Nov 2024 15:51:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731081073; cv=none; b=KRnt92wPJQTy0nVnADNd3vhUxeZcL5EcrMOqpc+W7bAv+9aZgYG3quJcD8KP2vAHigkfKiyk3WSkqOMD/6YSTZTIgDrmAZeXIGtpfXQIX9boOjX6n+E0v0FlsME7p1pGam6flmhjtUHTU2j2rJPRhoERhrBRGMy/Dvv8w8lNV0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731081073; c=relaxed/simple; bh=wss1YTUGIt5QGo6tSV9djIgAdvk1YIPYCs+TInKCU54=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HDRhizOy+fxyB1idQ02Yo82iL6tBXIe2k8sYbZy92rAo2pqcZsd/XwRlye9L3iYKabh/KSio43FPTsHWNQNo2x4Q+TbyZBPRUiOClV/dU7M7BOBwNAPMZNfDk9W/gF8n/aGddHXEChtgskGmEUluC9Y51b4ImBzeCC5GiSzU9Mo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LsLrsDTn; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LsLrsDTn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1731081069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+bBieyHX2d1zYDc3Erl++MDH1EG7vt2Iio6NE6nW7HE=; b=LsLrsDTnmcUndwH0ZF03tz/YCik7osE9QPMXdNcskcWTriknlCzrsh9BUn2KgkJWmYtMtG hLIlXlQhcvCXZC1dY+7ILDd97vtIQUWHsSece3nJfsgdAjdFkvG9c77q99JZ4UEk44Ttka rRKNK+B13h/CgZ12sBW2DK5hV4olTh4= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-347-z2_nwBsXMMmZ3Q4pqaaPwA-1; Fri, 08 Nov 2024 10:51:07 -0500 X-MC-Unique: z2_nwBsXMMmZ3Q4pqaaPwA-1 X-Mimecast-MFC-AGG-ID: z2_nwBsXMMmZ3Q4pqaaPwA Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 01C4819560B0; Fri, 8 Nov 2024 15:51:06 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 47440300019E; Fri, 8 Nov 2024 15:51:05 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: michael.roth@amd.com, seanjc@google.com Subject: [PATCH 3/3] KVM: gmem: track preparedness a page at a time Date: Fri, 8 Nov 2024 10:50:56 -0500 Message-ID: <20241108155056.332412-4-pbonzini@redhat.com> In-Reply-To: <20241108155056.332412-1-pbonzini@redhat.com> References: <20241108155056.332412-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Type: text/plain; charset="utf-8" With support for large pages in gmem, it may happen that part of the gmem is mapped with large pages and part with 4k pages. For example, if a conversion happens on a small region within a large page, the large page has to be smashed into small pages even if backed by a large folio. Each of the small pages will have its own state of preparedness, which makes it harder to use the uptodate flag for preparedness. Just switch to a bitmap in the inode's i_private data. This is a bit gnarly because ordinary bitmap operations in Linux are not atomic, but otherwise not too hard. Signed-off-by: Paolo Bonzini --- virt/kvm/guest_memfd.c | 103 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 100 insertions(+), 3 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 416e02a00cae..e08503dfdd8a 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -68,8 +68,13 @@ static struct file *kvm_gmem_create_file(const char *nam= e, const struct file_ope } =20 =20 +#define KVM_GMEM_INODE_SIZE(size) \ + struct_size_t(struct kvm_gmem_inode, prepared, \ + DIV_ROUND_UP(size, PAGE_SIZE * BITS_PER_LONG)) + struct kvm_gmem_inode { unsigned long flags; + unsigned long prepared[]; }; =20 struct kvm_gmem { @@ -107,18 +112,110 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm,= struct kvm_memory_slot *slo return 0; } =20 +/* + * The bitmap of prepared pages has to be accessed atomically, because + * preparation is not protected by any guest. This unfortunately means + * that we cannot use regular bitmap operations. + * + * The logic becomes a bit simpler for set and test, which operate a + * folio at a time and therefore can assume that the range is naturally + * aligned (meaning that either it is smaller than a word, or it is does + * not include fractions of a word). For punch-hole operations however + * there is all the complexity. + */ + +static void bitmap_set_atomic_word(unsigned long *p, unsigned long start, = unsigned long len) +{ + unsigned long mask_to_set =3D + BITMAP_FIRST_WORD_MASK(start) & BITMAP_LAST_WORD_MASK(start + len); + + atomic_long_or(mask_to_set, (atomic_long_t *)p); +} + +static void bitmap_clear_atomic_word(unsigned long *p, unsigned long start= , unsigned long len) +{ + unsigned long mask_to_set =3D + BITMAP_FIRST_WORD_MASK(start) & BITMAP_LAST_WORD_MASK(start + len); + + atomic_long_andnot(mask_to_set, (atomic_long_t *)p); +} + +static bool bitmap_test_allset_word(unsigned long *p, unsigned long start,= unsigned long len) +{ + unsigned long mask_to_set =3D + BITMAP_FIRST_WORD_MASK(start) & BITMAP_LAST_WORD_MASK(start + len); + + return (*p & mask_to_set) =3D=3D mask_to_set; +} + static void kvm_gmem_mark_prepared(struct file *file, pgoff_t index, struc= t folio *folio) { - folio_mark_uptodate(folio); + struct kvm_gmem_inode *i_gmem =3D (struct kvm_gmem_inode *)file->f_inode-= >i_private; + unsigned long *p =3D i_gmem->prepared + BIT_WORD(index); + unsigned long npages =3D folio_nr_pages(folio); + + /* Folios must be naturally aligned */ + WARN_ON_ONCE(index & (npages - 1)); + index &=3D ~(npages - 1); + + /* Clear page before updating bitmap. */ + smp_wmb(); + + if (npages < BITS_PER_LONG) { + bitmap_set_atomic_word(p, index, npages); + } else { + BUILD_BUG_ON(BITS_PER_LONG !=3D 64); + memset64((u64 *)p, ~0, BITS_TO_LONGS(npages)); + } } =20 static void kvm_gmem_mark_range_unprepared(struct inode *inode, pgoff_t in= dex, pgoff_t npages) { + struct kvm_gmem_inode *i_gmem =3D (struct kvm_gmem_inode *)inode->i_priva= te; + unsigned long *p =3D i_gmem->prepared + BIT_WORD(index); + + index &=3D BITS_PER_LONG - 1; + if (index) { + int first_word_count =3D min(npages, BITS_PER_LONG - index); + bitmap_clear_atomic_word(p, index, first_word_count); + npages -=3D first_word_count; + p++; + } + + if (npages > BITS_PER_LONG) { + BUILD_BUG_ON(BITS_PER_LONG !=3D 64); + memset64((u64 *)p, 0, BITS_TO_LONGS(npages)); + p +=3D BIT_WORD(npages); + npages &=3D BITS_PER_LONG - 1; + } + + if (npages) + bitmap_clear_atomic_word(p++, 0, npages); } =20 static bool kvm_gmem_is_prepared(struct file *file, pgoff_t index, struct = folio *folio) { - return folio_test_uptodate(folio); + struct kvm_gmem_inode *i_gmem =3D (struct kvm_gmem_inode *)file->f_inode-= >i_private; + unsigned long *p =3D i_gmem->prepared + BIT_WORD(index); + unsigned long npages =3D folio_nr_pages(folio); + bool ret; + + /* Folios must be naturally aligned */ + WARN_ON_ONCE(index & (npages - 1)); + index &=3D ~(npages - 1); + + if (npages < BITS_PER_LONG) { + ret =3D bitmap_test_allset_word(p, index, npages); + } else { + for (; npages > 0; npages -=3D BITS_PER_LONG) + if (*p++ !=3D ~0) + break; + ret =3D (npages =3D=3D 0); + } + + /* Synchronize with kvm_gmem_mark_prepared(). */ + smp_rmb(); + return ret; } =20 /* @@ -499,7 +596,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t si= ze, u64 flags) struct file *file; int fd, err; =20 - i_gmem =3D kvzalloc(sizeof(struct kvm_gmem_inode), GFP_KERNEL); + i_gmem =3D kvzalloc(KVM_GMEM_INODE_SIZE(size), GFP_KERNEL); if (!i_gmem) return -ENOMEM; i_gmem->flags =3D flags; --=20 2.43.5