From nobody Sun Feb 8 20:28:23 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612870041; cv=none; d=zohomail.com; s=zohoarc; b=Yj6xwfWzBzu2XEjnCDkV/4Q4mBSgvqadM9Y7Li0aQND2nV9BoXLawIgoUphVVod9SpSdgZoFJ3JwSGtS4Fu/ecJAUEDsm+k+Wa2z3TW6Pu8ojJJavxsVydIvBpDualC1orr/aKi6WsY/yd0NHiwzStU6HcXvE2LGkqTTjZ1yd70= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612870041; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=obSi4SgeCJ+YBCdSEH0c51FwoSt0SiGpC9OCfOXmi1g=; b=OFi1FiDv/YJHaZ2+7Zzv1gbwN4HdhH9Tka6I+4hdmY15CDye6zvcNcSnTAn24ZLjdL2C8qGyVH5rfYoexvWn0rF8O1RC4qVi2r6h0piaPZcwP5Nb5sBO1TA3pQ/zI590wUGymUQawEA3CCqnzSpiHSpYyOQPwIiIM6vqFvFO0b4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1612870040840350.1509039210889; Tue, 9 Feb 2021 03:27:20 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-540-1csNxMYxMg-AoPeAKib5gg-1; Tue, 09 Feb 2021 06:27:17 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6F8FACC623; Tue, 9 Feb 2021 11:27:12 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3B57819726; Tue, 9 Feb 2021 11:27:12 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 0140558084; Tue, 9 Feb 2021 11:27:12 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 119BQlpu005993 for ; Tue, 9 Feb 2021 06:26:47 -0500 Received: by smtp.corp.redhat.com (Postfix) id 9C1256267D; Tue, 9 Feb 2021 11:26:47 +0000 (UTC) Received: from localhost.localdomain (unknown [10.40.195.5]) by smtp.corp.redhat.com (Postfix) with ESMTP id E646E61F49; Tue, 9 Feb 2021 11:26:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612870039; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=obSi4SgeCJ+YBCdSEH0c51FwoSt0SiGpC9OCfOXmi1g=; b=ZHNO1Crd1MaDHh1DFJgMqlmLc7ZziWZgn5Imnq/pxhv+7GXuLsQNnVTEEHRIjd3oyd7q1u dgtmhwDthSrNKQepIIB2pxQJLpOXrVSxLjcJ38Tnb4zG+9Yh0gyYw7XhKt6kULQ93mkxKf VUoVzQqfWe/WPZemmOu0jgjmRNoYoes= X-MC-Unique: 1csNxMYxMg-AoPeAKib5gg-1 From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH v2 13/13] kbase: Document virtio-mem Date: Tue, 9 Feb 2021 12:26:18 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Cc: david@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This commit adds new memorydevices.rst page which should serve all models of memory devices. Yet, I'm documenting virtio-mem quirks only. Signed-off-by: Michal Privoznik --- docs/kbase/index.rst | 4 + docs/kbase/memorydevices.rst | 160 +++++++++++++++++++++++++++++++++++ docs/kbase/meson.build | 1 + 3 files changed, 165 insertions(+) create mode 100644 docs/kbase/memorydevices.rst diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 532804fe05..45450bf33b 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -46,6 +46,10 @@ Usage `PCI topology <../pci-addresses.html>`__ Addressing schemes for PCI devices =20 +`Memory devices `__ + Memory devices and their use + + Internals / Debugging --------------------- =20 diff --git a/docs/kbase/memorydevices.rst b/docs/kbase/memorydevices.rst new file mode 100644 index 0000000000..23adf54e16 --- /dev/null +++ b/docs/kbase/memorydevices.rst @@ -0,0 +1,160 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. contents:: + +Basics +=3D=3D=3D=3D=3D=3D + +Memory devices can be divided into two families: DIMMs and NVDIMMs. The fo= rmer +is typical RAM memory: it's volatile and thus its contents doesn't survive +reboots nor guest shut downs and power ons. The latter retains its contents +across reboots or power outages. + +In Libvirt, there are two models for DIMMs: + +* ``dimm`` model: + + :: + + + + 523264 + 0 + +
+ + +* ``virtio-mem`` model: + + :: + + + + 1048576 + 0 + 2048 + 524288 + +
+ + +Then there are two models for NVDIMMs: + +* ``nvidmm`` model: + + :: + + + + /tmp/nvdimm + + + 523264 + 0 + +
+ + +* ``virtio-pmem`` model: + + :: + + + + /tmp/virtio_pmem + + + 524288 + +
+ + + +Please not that (maybe somewhat surprisingly) virtio models go onto PCI bus +instead of DIMM slots. + +Furthermore, DIMMs can have ```` element which configures backend= for +devices. For NVDIMMs the element is mandatory and reflects where the conte= nts +is saved. + +See https://libvirt.org/formatdomain.html#elementsMemory + +``virtio-mem`` model +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The ``virtio-mem`` model can be viewed as revised memory balloon. It offers +memory hotplug and hotunplug solution (without the actual hotplug of the +device). It solves problems that memory balloon can't solve on its own and= thus +is more flexible than DIMM + balloon solution. ``virtio-mem`` is NUMA awar= e, +and thus memory can be inflated/deflated only for a subset of guest NUMA n= odes. +Also, it works with chunks that are either exposed to guest or taken back = from +it. + +See https://virtio-mem.gitlab.io/ + +Under the hood, ``virtio-mem`` device is split into chunks of equal size w= hich +are then exposed to the guest. Either all of them or only a portion depend= ing +on user's request. Therefore there are three important sizes for +``virtio-mem``. All are to be found under ```` element: + +#. The maximum size the device can ever offer, exposed under ```` +#. The size a single block, exposed under ```` +#. The current size exposed to the guest, exposed under ```` + +For instance, the following example the maximum size is 4GiB, the block si= ze is +2MiB and only 1GiB should be exposed to the guest: + + :: + + + + 4194304 + 2048 + 1048576 + + + +Please note that ```` must be an integer multiple of ```` +size or zero (memory completely deflated) and has to be less or equal to +```` (memory completely inflated). Furthermore, QEMU recommends the +```` size to be as big as a Transparent Huge Page (usually 2MiB). + +To change the size exposed to the guest, users should pass memory device X= ML +with nothing but ```` changed into the +``virDomainUpdateDeviceFlags()`` API. For user's convenience this can be d= one +via virsh too: + + :: + + # virsh update-memory-device $dom --requested size 2GiB + +If there are two or more ```` devices then ``--alias`` shall be u= sed +to tell virsh which memory device should be updated. + +For running guests there is fourth size that can be found under ````: + + :: + + 2097152 + +The ```` reflects the actual size consumed by the guest. In gener= al it +can differ from ````. Reasons include guest kernel missing +``virtio-mem`` module and thus being unable to take offered memory, or gue= st +kernel being unable to free memory and allow deflation. Since ``= `` +only reports size to users, the element is never parsed. It is formatted o= nly +into live XML. + +Since changing actual allocation requires cooperation with guest kernel, +requests for change are not instant. Therefore, libvirt emits +``VIR_DOMAIN_EVENT_ID_MEMORY_DEVICE_SIZE_CHANGE`` event whenever actual +allocation changed. + +Please not that using ``virtio-mem`` with memory balloon is not possible, +currently. The real reason is that libvirt's memory accounting isn't ready= and +mixing these two would be confusing to users. Libvirt exposes current valu= e of +memory balloon under ```` but if it were to account for +```` too then it would be impossible to learn true size of the +balloon. Also it might result in mistakenly trying to deflate ``virtio-mem= `` +via ``setmem`` command. diff --git a/docs/kbase/meson.build b/docs/kbase/meson.build index 7b4e7abbd3..7f6adba915 100644 --- a/docs/kbase/meson.build +++ b/docs/kbase/meson.build @@ -9,6 +9,7 @@ docs_kbase_files =3D [ 'locking-lockd', 'locking', 'locking-sanlock', + 'memorydevices', 'migrationinternals', 'qemu-passthrough-security', 'rpm-deployment', --=20 2.26.2