Here's v13. Thanks everyone for the comments and fast responses!
We're now at ~3 weeks to soft-close at 7.3-rc5.

v13 is based on 7.3-rc2 and:

+ Hugh's patch to export lru_cache_drain_for_folio()
+ Another series [1], which makes kvm_gmem_get_pfn() NOT return a
  refcounted page to KVM.

Here's everything stitched together for your convenience:

https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v13

Changes in v13:

+ Picked up Reviewed-bys
+ Fixed bug Fuad pointed out in "Ensure pages are not in use before
  conversion", and squashed the lru draining in, adopted Sean's
  suggestion.
+ Fixed documentation bug that Fuad pointed out in
  kernel-parameters.txt.

Here's v13 with tests:

https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-coco-selftests-v13

Tested with both CONFIG_KVM_VM_MEMORY_ATTRIBUTES enabled and disabled:

+ tools/testing/selftests/kvm/guest_memfd_test.c
+ tools/testing/selftests/kvm/pre_fault_memory_test.c
+ tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c

[1] https://lore.kernel.org/all/20260826-gmem-no-return-page-v4-0-3bb9c1ddb4e3@google.com/

v12: https://lore.kernel.org/r/20260830-gmem-inplace-conversion-v12-0-85e5fd25252a@google.com
v11: https://lore.kernel.org/r/20260826-gmem-inplace-conversion-v11-0-0a15d8a799aa@google.com
v10: https://lore.kernel.org/r/20260807-gmem-inplace-conversion-v10-0-2fc18ee6d3ba@google.com
v9: https://lore.kernel.org/r/20260728-gmem-inplace-conversion-v9-0-35f9aec2aed2@google.com
v8: https://lore.kernel.org/r/20260618-gmem-inplace-conversion-v8-0-9d2959357853@google.com
v7: https://lore.kernel.org/r/20260522-gmem-inplace-conversion-v7-0-2f0fae496530@google.com
v6: https://lore.kernel.org/r/20260507-gmem-inplace-conversion-v6-0-91ab5a8b19a4@google.com
RFC v5: https://lore.kernel.org/r/20260428-gmem-inplace-conversion-v5-0-d8608ccfca22@google.com
RFC v4: https://lore.kernel.org/r/20260326-gmem-inplace-conversion-v4-0-e202fe950ffd@google.com
RFC v3: https://lore.kernel.org/r/20260313-gmem-inplace-conversion-v3-0-5fc12a70ec89@google.com
RFC v2: https://lore.kernel.org/r/cover.1770071243.git.ackerleytng@google.com
RFC v1: https://lore.kernel.org/r/cover.1760731772.git.ackerleytng@google.com

Previous versions of this feature, part of other series, are available at:

+ https://lore.kernel.org/all/bd163de3118b626d1005aa88e71ef2fb72f0be0f.1726009989.git.ackerleytng@google.com/
+ https://lore.kernel.org/all/20250117163001.2326672-6-tabba@google.com/
+ https://lore.kernel.org/all/b784326e9ccae6a08388f1bf39db70a2204bdc51.1747264138.git.ackerleytng@google.com/

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (22):
      KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
      KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
      KVM: guest_memfd: Pass mapping type filter to invalidation helper
      KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
      KVM: guest_memfd: Ensure pages are not in use before conversion
      KVM: guest_memfd: Call arch make_shared callback for to-shared conversion
      KVM: guest_memfd: Return early if range already has requested attributes
      KVM: guest_memfd: Zero page while getting pfn
      KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION
      KVM: selftests: Test basic single-page conversion flow
      KVM: selftests: Test conversion flow when INIT_SHARED
      KVM: selftests: Test conversion precision in guest_memfd
      KVM: selftests: Test conversion before allocation
      KVM: selftests: Convert with allocated folios in different layouts
      KVM: selftests: Test that truncation does not change shared/private status
      KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
      KVM: selftests: Test conversion with elevated page refcount
      KVM: selftests: Reset shared memory after hole-punching
      KVM: selftests: Provide function to look up guest_memfd details from gpa
      KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
      KVM: selftests: Set up page size and alignment independently for guest_memfd
      KVM: selftests: Update private_mem_conversions_test for in-place conversions

Michael Roth (1):
      KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE

Sean Christopherson (21):
      KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
      KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
      KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings
      KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES
      KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined
      KVM: Rename memory attribute APIs to prepare for in-place gmem conversion
      KVM: Provide generic interface for checking memory private/shared status
      KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
      KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h
      KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
      KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86
      KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
      KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs
      KVM: selftests: Create gmem fd before "regular" fd when adding memslot
      KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}
      KVM: selftests: Add support for mmap() on guest_memfd in core library
      KVM: selftests: Add selftests global for guest memory attributes capability
      KVM: selftests: Add helpers for calling ioctls on guest_memfd
      KVM: selftests: Test that shared/private status is consistent across processes
      KVM: selftests: Provide common function to set memory attributes
      KVM: selftests: Update private memory exits test to work with per-gmem attributes

 Documentation/admin-guide/kernel-parameters.txt    |  25 +
 Documentation/virt/kvm/api.rst                     | 110 ++++-
 .../virt/kvm/x86/amd-memory-encryption.rst         |  14 +-
 Documentation/virt/kvm/x86/intel-tdx.rst           |   4 +
 arch/x86/include/asm/kvm-x86-ops.h                 |   2 +-
 arch/x86/include/asm/kvm_host.h                    |   9 +-
 arch/x86/kvm/Kconfig                               |  15 +-
 arch/x86/kvm/mmu/mmu.c                             |  28 +-
 arch/x86/kvm/svm/sev.c                             |  13 +-
 arch/x86/kvm/vmx/tdx.c                             |   8 +-
 arch/x86/kvm/x86.c                                 |  20 +-
 include/linux/kvm_host.h                           |  83 ++--
 include/trace/events/kvm.h                         |   6 +-
 include/uapi/linux/kvm.h                           |  16 +
 tools/testing/selftests/kvm/Makefile.kvm           |   1 +
 tools/testing/selftests/kvm/include/kvm_util.h     | 139 +++++-
 tools/testing/selftests/kvm/include/test_util.h    |  34 +-
 tools/testing/selftests/kvm/lib/kvm_util.c         | 222 +++++----
 tools/testing/selftests/kvm/lib/test_util.c        |   7 -
 .../kvm/x86/guest_memfd_conversions_test.c         | 512 +++++++++++++++++++++
 .../kvm/x86/private_mem_conversions_test.c         |  66 ++-
 .../selftests/kvm/x86/private_mem_kvm_exits_test.c |  36 +-
 virt/kvm/Kconfig                                   |   3 -
 virt/kvm/guest_memfd.c                             | 468 +++++++++++++++++--
 virt/kvm/kvm_main.c                                |  92 ++--
 25 files changed, 1642 insertions(+), 291 deletions(-)
---
base-commit: 5e036ce12de91c6fd674dad33b169c6150be2a7a
change-id: 20260225-gmem-inplace-conversion-bd0dbd39753a
prerequisite-message-id: 02876cea-5727-2ca4-bead-73659ea6fec4@google.com
prerequisite-patch-id: 28922a76dee80792da6e445a55cdbb4bedc85374
prerequisite-change-id: 20260818-gmem-no-return-page-614927a29f97:v4
prerequisite-patch-id: 38553271bcb80c17dbae29cef5c20bcf95e804f9
prerequisite-patch-id: 0d4b50926073ba4f5e5d8b2c0cdae8869a278863
prerequisite-patch-id: bf39277ddc12818fe562a006f31d7992e5c7d42a
prerequisite-patch-id: 74fe849c04c187074bd380e664e536a07076292d
prerequisite-patch-id: f92bdb4870dd7c1a2026fca1083e4400c2be11d8

Best regards,
--
Ackerley Tng <ackerleytng@google.com>

On Thu, 10 Sep 2026 16:55:26 -0700, Ackerley Tng wrote:
> Here's v13. Thanks everyone for the comments and fast responses!
> We're now at ~3 weeks to soft-close at 7.3-rc5.
> 
> v13 is based on 7.3-rc2 and:
> 
> + Hugh's patch to export lru_cache_drain_for_folio()
> + Another series [1], which makes kvm_gmem_get_pfn() NOT return a
>   refcounted page to KVM.
> 
> [...]

Applied to kvm-x86 coco.  A *huge* thanks to everyone who helped with this!

[01/44] KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
        https://github.com/kvm-x86/linux/commit/54623a16bbdc
[02/44] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
        https://github.com/kvm-x86/linux/commit/fa7b8b1bb5d0
[03/44] KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings
        https://github.com/kvm-x86/linux/commit/1ae5be958ff7
[04/44] KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES
        https://github.com/kvm-x86/linux/commit/44e17d0a3663
[05/44] KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined
        https://github.com/kvm-x86/linux/commit/c2ae2bc1a311
[06/44] KVM: Rename memory attribute APIs to prepare for in-place gmem conversion
        https://github.com/kvm-x86/linux/commit/a84ef89cbf91
[07/44] KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
        https://github.com/kvm-x86/linux/commit/98fd6e30bd14
[08/44] KVM: Provide generic interface for checking memory private/shared status
        https://github.com/kvm-x86/linux/commit/8ab5719fbcdc
[09/44] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
        https://github.com/kvm-x86/linux/commit/ee1168fb7313
[10/44] KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h
        https://github.com/kvm-x86/linux/commit/bcc640eabbb2
[11/44] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
        https://github.com/kvm-x86/linux/commit/e088c7f86bef
[12/44] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
        https://github.com/kvm-x86/linux/commit/2e646bcfcf90
[13/44] KVM: guest_memfd: Pass mapping type filter to invalidation helper
        https://github.com/kvm-x86/linux/commit/2f7f4e102880
[14/44] KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
        https://github.com/kvm-x86/linux/commit/799c9fadc639
[15/44] KVM: guest_memfd: Ensure pages are not in use before conversion
        https://github.com/kvm-x86/linux/commit/dcde2f853ff4
[16/44] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion
        https://github.com/kvm-x86/linux/commit/6fa1dab756fd
[17/44] KVM: guest_memfd: Return early if range already has requested attributes
        https://github.com/kvm-x86/linux/commit/856852ef2108
[18/44] KVM: guest_memfd: Zero page while getting pfn
        https://github.com/kvm-x86/linux/commit/cb0544296a33
[19/44] KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE
        https://github.com/kvm-x86/linux/commit/edcc3f42981e
[20/44] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION
        https://github.com/kvm-x86/linux/commit/af602a96f401
[21/44] KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86
        https://github.com/kvm-x86/linux/commit/31670c382efc
[22/44] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
        https://github.com/kvm-x86/linux/commit/537ec2b15fdd
[23/44] KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs
        https://github.com/kvm-x86/linux/commit/b5ed1ee2a7a0
[24/44] KVM: selftests: Create gmem fd before "regular" fd when adding memslot
        https://github.com/kvm-x86/linux/commit/b312cab4e48c
[25/44] KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}
        https://github.com/kvm-x86/linux/commit/8d5f49000f8b
[26/44] KVM: selftests: Add support for mmap() on guest_memfd in core library
        https://github.com/kvm-x86/linux/commit/ee455183c92b
[27/44] KVM: selftests: Add selftests global for guest memory attributes capability
        https://github.com/kvm-x86/linux/commit/c3e48c166e35
[28/44] KVM: selftests: Add helpers for calling ioctls on guest_memfd
        https://github.com/kvm-x86/linux/commit/0bbbd2b02bf0
[29/44] KVM: selftests: Test basic single-page conversion flow
        https://github.com/kvm-x86/linux/commit/eaef4ed5fabc
[30/44] KVM: selftests: Test conversion flow when INIT_SHARED
        https://github.com/kvm-x86/linux/commit/e5fa04dc920e
[31/44] KVM: selftests: Test conversion precision in guest_memfd
        https://github.com/kvm-x86/linux/commit/492080a3d4e5
[32/44] KVM: selftests: Test conversion before allocation
        https://github.com/kvm-x86/linux/commit/97137c140360
[33/44] KVM: selftests: Convert with allocated folios in different layouts
        https://github.com/kvm-x86/linux/commit/5362c5fb88f6
[34/44] KVM: selftests: Test that truncation does not change shared/private status
        https://github.com/kvm-x86/linux/commit/bfb9b5a286ea
[35/44] KVM: selftests: Test that shared/private status is consistent across processes
        https://github.com/kvm-x86/linux/commit/8b72eaa766b6
[36/44] KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
        https://github.com/kvm-x86/linux/commit/e8a40b9a604e
[37/44] KVM: selftests: Test conversion with elevated page refcount
        https://github.com/kvm-x86/linux/commit/0db3f14f40c4
[38/44] KVM: selftests: Reset shared memory after hole-punching
        https://github.com/kvm-x86/linux/commit/c235250e8bc1
[39/44] KVM: selftests: Provide function to look up guest_memfd details from gpa
        https://github.com/kvm-x86/linux/commit/ee6f4c381903
[40/44] KVM: selftests: Provide common function to set memory attributes
        https://github.com/kvm-x86/linux/commit/f49dd8ba330e
[41/44] KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
        https://github.com/kvm-x86/linux/commit/ef72e71da099
[42/44] KVM: selftests: Set up page size and alignment independently for guest_memfd
        https://github.com/kvm-x86/linux/commit/710169986d9b
[43/44] KVM: selftests: Update private_mem_conversions_test for in-place conversions
        https://github.com/kvm-x86/linux/commit/e24e534ff850
[44/44] KVM: selftests: Update private memory exits test to work with per-gmem attributes
        https://github.com/kvm-x86/linux/commit/37e0791600e5

--
https://github.com/kvm-x86/linux/tree/next

Sean Christopherson <seanjc@google.com> writes:

> On Thu, 10 Sep 2026 16:55:26 -0700, Ackerley Tng wrote:
>> Here's v13. Thanks everyone for the comments and fast responses!
>> We're now at ~3 weeks to soft-close at 7.3-rc5.
>>
>> v13 is based on 7.3-rc2 and:
>>
>> + Hugh's patch to export lru_cache_drain_for_folio()
>> + Another series [1], which makes kvm_gmem_get_pfn() NOT return a
>>   refcounted page to KVM.
>>
>> [...]
>
> Applied to kvm-x86 coco.  A *huge* thanks to everyone who helped with this!
>

Thank you so much everyone!

>
> [...snip...]
>

On 9/25/26 00:31, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
>> On Thu, 10 Sep 2026 16:55:26 -0700, Ackerley Tng wrote:
>>> Here's v13. Thanks everyone for the comments and fast responses!
>>> We're now at ~3 weeks to soft-close at 7.3-rc5.
>>>
>>> v13 is based on 7.3-rc2 and:
>>>
>>> + Hugh's patch to export lru_cache_drain_for_folio()
>>> + Another series [1], which makes kvm_gmem_get_pfn() NOT return a
>>>   refcounted page to KVM.
>>>
>>> [...]
>>
>> Applied to kvm-x86 coco.  A *huge* thanks to everyone who helped with this!
>>
> 
> Thank you so much everyone!

Nice, a big milestone!

Well, let's cross fingers that it will end up upstream! ;)

-- 
Cheers,

David

On Thu, Sep 10, 2026 at 04:55:26PM -0700, Ackerley Tng via B4 Relay wrote:
> Here's v13. Thanks everyone for the comments and fast responses!
> We're now at ~3 weeks to soft-close at 7.3-rc5.
> 
> v13 is based on 7.3-rc2 and:
> 
> + Hugh's patch to export lru_cache_drain_for_folio()
> + Another series [1], which makes kvm_gmem_get_pfn() NOT return a
>   refcounted page to KVM.
> 
> Here's everything stitched together for your convenience:
> 
> https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v13
The following configurations were tested and all booted TDs successfully:
(1) gmem_in_place_conversion=1, in-place conversions + in-place-add init mem
(2) gmem_in_place_conversion=1, in-place conversions + out-of-place-add init mem
(3) gmem_in_place_conversion=0, out-of-place conversions + out-of-place-add init mem

Tested-by: Yan Zhao <yan.y.zhao@intel.com>