From nobody Sun Apr 12 05:57:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@amazon.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=amazon.com ARC-Seal: i=1; a=rsa-sha256; t=1772030920; cv=none; d=zohomail.com; s=zohoarc; b=K0nX+gJoRAmZDXSswER0J4khTvUCUJnlEzw0rQ1sYT4cK9YWb4b6TwbvdBwNpLkbUfvO1vpHNBZ7hjEFfcgmUJ+Se/u72gPe9JXyWdQi5+PPp43cIvbKqRyA4WNXSOE8vJ/aCpZPDhtDiFl3xXuSbt7ET48sjGuwRONF5ksUX8M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772030920; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+aizdLXyc6doicL/j8QKWc/fhiSn6T+aU4ZL0eEP5CQ=; b=UrYEtL/IfAStAWEzKK2qIHOqhJ5xtVV8RahlF8cilop8V71QM/3GlGITDg/oDZvTFCaG2+QiE5EXecQnj7aNg+IDZ7zDAwcvw+mp86pnUrsTyadKtS/o33FqMb/XH+8NXQZgnK9ox5JA88COnVX92GjuFn401filKGr35Iu65so= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@amazon.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1772030920431441.11837629176216; Wed, 25 Feb 2026 06:48:40 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vvGBh-0001mI-1R; Wed, 25 Feb 2026 09:48:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vvGAa-00019j-12; Wed, 25 Feb 2026 09:47:23 -0500 Received: from pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com ([50.112.246.219]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vvGAX-00079s-1S; Wed, 25 Feb 2026 09:47:18 -0500 Received: from ip-10-5-12-219.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.12.219]) by internal-pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Feb 2026 14:47:15 +0000 Received: from EX19MTAUWA001.ant.amazon.com [205.251.233.182:28167] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.36.66:2525] with esmtp (Farcaster) id 2e12e5cc-0052-430e-b0f8-27005418b99b; Wed, 25 Feb 2026 14:47:14 +0000 (UTC) Received: from EX19D020UWC004.ant.amazon.com (10.13.138.149) by EX19MTAUWA001.ant.amazon.com (10.250.64.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.37; Wed, 25 Feb 2026 14:47:12 +0000 Received: from ip-10-253-83-51.amazon.com (172.19.99.218) by EX19D020UWC004.ant.amazon.com (10.13.138.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.37; Wed, 25 Feb 2026 14:47:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1772030837; x=1803566837; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+aizdLXyc6doicL/j8QKWc/fhiSn6T+aU4ZL0eEP5CQ=; b=QQbA4AKXfGv3Sj8bnMovwH6fK6kQP5o5Uyyo5IrxCPO3P22zfvsr0zuc 8dv8+2ew892fTiQ/8LfSWiB0K2EC65HIqA855ZcaqrMX7jZjW5lYYoV5y sbEzt3Vl0hWPWz9f5xn+d3kkvB5WIAuRMaaAZLLZxxN2kJ8GK0d6gV2Xq mkrY5UA3Q19EzECbrS2bOke/twU5L3OIn8Uz69xNWZDPIj3tVQzqFxENR XpRoENKwTOd/OW2K+wQ2iCKs/vDHyyxALPJ8Ur6dMZutbIc64UAMbZbTj aoTUtasZGzg+ShsZuNQaTgbg5ne0pGwz0R/FfrM1Zf6IoQ9AgzKDiHn46 w==; X-CSE-ConnectionGUID: 8EGA8BPBRQSuKKq1t6eCuw== X-CSE-MsgGUID: Ko1KZaXzQ7uQ3aBLoj31Dg== X-IronPort-AV: E=Sophos;i="6.21,310,1763424000"; d="scan'208";a="13594551" X-Farcaster-Flow-ID: 2e12e5cc-0052-430e-b0f8-27005418b99b From: Alexander Graf To: CC: , Peter Maydell , "Thomas Huth" , , , , , Cornelia Huck , , Dorjoy Chowdhury , Pierrick Bouvier , Paolo Bonzini , Tyler Fanelli , , Subject: [PATCH v2 11/11] docs: Add Nitro Enclaves documentation Date: Wed, 25 Feb 2026 14:45:30 +0000 Message-ID: <20260225144532.84673-12-graf@amazon.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20260225144532.84673-1-graf@amazon.com> References: <20260225144532.84673-1-graf@amazon.com> MIME-Version: 1.0 X-Originating-IP: [172.19.99.218] X-ClientProxiedBy: EX19D039UWB003.ant.amazon.com (10.13.138.93) To EX19D020UWC004.ant.amazon.com (10.13.138.149) Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=50.112.246.219; envelope-from=prvs=509822c20=graf@amazon.de; helo=pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com X-Spam_score_int: -1 X-Spam_score: -0.2 X-Spam_bar: / X-Spam_report: (-0.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.734, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.78, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, UNPARSEABLE_RELAY=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @amazon.com) X-ZM-MESSAGEID: 1772030921406158500 Content-Type: text/plain; charset="utf-8" Now that all pieces are in place to spawn Nitro Enclaves using a special purpose accelerator and machine model, document how to use it. Signed-off-by: Alexander Graf --- v1 -> v2: - explain the nitro bus, its notification flow and topology - give example for qom-get of the enclave cid - document the accel enclave-cid property --- MAINTAINERS | 1 + docs/system/confidential-guest-support.rst | 1 + docs/system/index.rst | 1 + docs/system/nitro.rst | 133 +++++++++++++++++++++ 4 files changed, 136 insertions(+) create mode 100644 docs/system/nitro.rst diff --git a/MAINTAINERS b/MAINTAINERS index 53ce075e9a..5e9e429530 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3027,6 +3027,7 @@ M: Alexander Graf S: Maintained F: hw/nitro/ F: include/hw/nitro/ +F: docs/system/nitro.rst =20 Subsystems ---------- diff --git a/docs/system/confidential-guest-support.rst b/docs/system/confi= dential-guest-support.rst index 66129fbab6..562a7c3c28 100644 --- a/docs/system/confidential-guest-support.rst +++ b/docs/system/confidential-guest-support.rst @@ -41,5 +41,6 @@ Currently supported confidential guest mechanisms are: * Intel Trust Domain Extension (TDX) (see :doc:`i386/tdx`) * POWER Protected Execution Facility (PEF) (see :ref:`power-papr-protected= -execution-facility-pef`) * s390x Protected Virtualization (PV) (see :doc:`s390x/protvirt`) +* AWS Nitro Enclaves (see :doc:`nitro`) =20 Other mechanisms may be supported in future. diff --git a/docs/system/index.rst b/docs/system/index.rst index 427b020483..d297a95282 100644 --- a/docs/system/index.rst +++ b/docs/system/index.rst @@ -39,5 +39,6 @@ or Hypervisor.Framework. multi-process confidential-guest-support igvm + nitro vm-templating sriov diff --git a/docs/system/nitro.rst b/docs/system/nitro.rst new file mode 100644 index 0000000000..5907d6153e --- /dev/null +++ b/docs/system/nitro.rst @@ -0,0 +1,133 @@ +AWS Nitro Enclaves +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +`AWS Nitro Enclaves `_ +are isolated compute environments that run alongside EC2 instances. +They are created by partitioning CPU and memory resources from a parent +instance and launching a signed Enclave Image Format (EIF) file inside +a confidential VM managed by the Nitro Hypervisor. + +QEMU supports launching Nitro Enclaves on EC2 instances that have +enclave support enabled, using the ``nitro`` accelerator and the +``nitro`` machine type. + +Prerequisites +------------- + +* An EC2 instance with Nitro Enclaves enabled +* The ``nitro_enclaves`` kernel module loaded (provides ``/dev/nitro_encla= ves``) +* CPU cores allocated to the Nitro Enclaves pool via ``nitro-enclaves-allo= cator`` +* Huge pages allocated for Nitro Enclaves via ``nitro-enclaves-allocator`` + +Quick Start +----------- + +Launch a Nitro Enclave from a pre-built EIF file:: + + $ qemu-system-x86_64 -accel nitro,debug-mode=3Don -M nitro -nographic \ + -smp 2 -m 512M -kernel enclave.eif + +Launch an enclave from individual kernel and initrd files:: + + $ qemu-system-x86_64 -accel nitro,debug-mode=3Don -M nitro -nographic \ + -smp 2 -m 512M -kernel vmlinuz -initrd initrd.cpio \ + -append "console=3DttyS0" + +The same commands work with ``qemu-system-aarch64`` on Graviton based EC2 +instances. + +Accelerator +----------- + +The ``nitro`` accelerator (``-accel nitro``) drives the +``/dev/nitro_enclaves`` device to create and manage a Nitro Enclave. +It handles: + +* Creating the enclave VM slot +* Donating memory regions (must be huge page backed) +* Adding vCPUs (must be full physical cores) +* Starting the enclave +* Notifying vsock bus devices of the enclave CID + +Accelerator options: + +``debug-mode=3Don|off`` + Enable debug mode. When enabled, the Nitro Hypervisor exposes the + enclave's serial console output via a vsock port that the machine + model automatically connects to. In debug mode, PCR values are zero. + Default is ``off``. + +Machine +------- + +The ``nitro`` machine (``-M nitro``) is a minimal, architecture-independent +machine that provides only what a Nitro Enclave needs: + +* RAM (huge page backed via memfd) +* vCPUs (defaults to ``host`` CPU type) +* A Nitro vsock bus with: + + - A heartbeat device (vsock server on port 9000) + - A serial console bridge (vsock client, debug mode only) + +Communication to the Nitro Enclave is limited to virtio-vsock. The Enclave +is allocated a CID at launch at which it is reachable. A specific CID can +be requested with ``-accel nitro,enclave-cid=3D`` (0 lets the hypervisor +choose). The assigned CID is readable from the vsock bridge device:: + + (qemu) qom-get /machine/peripheral/nitro-vsock enclave-cid + +EIF Image Format +^^^^^^^^^^^^^^^^ + +Nitro Enclaves boot from EIF (Enclave Image Format) files. When +``-kernel`` points to an EIF file (detected by the ``.eif`` magic +bytes), it is loaded directly into guest memory. + +When ``-kernel`` points to a regular kernel image (e.g. a bzImage or +Image), the machine automatically assembles a minimal EIF on the fly +from ``-kernel``, ``-initrd``, and ``-append``. This allows standard +direct kernel boot without external EIF tooling. + +CPU Requirements +^^^^^^^^^^^^^^^^ + +Nitro Enclaves require full physical CPU cores. On hyperthreaded +systems, this means ``-smp`` must be a multiple of the threads per +core (typically 2). + +Nitro Enclaves can only consume cores that are donated to the Nitro Enclave +CPU pool. You can configure the CPU pool using the ``nitro-enclaves-alloca= tor`` +tool or manually by writing to the nitro_enclaves cpu pool parameter. To +allocate vCPUs 1, 2 and 3, you can call:: + + $ echo 1,2,3 | sudo tee /sys/module/nitro_enclaves/parameters/ne_cpus + +Beware that on x86-64 systems, hyperthread siblings are not consecutive +and must be added in pairs to the pool. Consult tools like ``lstopo`` +or ``lscpu`` for details about your instance's CPU topology. + +Memory Requirements +^^^^^^^^^^^^^^^^^^^ + +Enclave memory must be huge page backed. The machine automatically +creates a memfd memory backend with huge pages enabled. To make the +huge page allocation work, ensure that huge pages are reserved in +the system. To reserve 1 GiB of memory on a 4 KiB PAGE_SIZE system, +you can call:: + + $ echo 512 | sudo tee /proc/sys/vm/nr_hugepages + +Emulated Nitro Enclaves +----------------------- + +In addition to the native Nitro Enclaves invocation, you can also use +the emulated nitro-enclave machine target (see :doc:`i386/nitro-enclave`) +which implements the x86 Nitro Enclave device model. While -M nitro +delegates virtual machine device emulation to the Nitro Hypervisor, -M +nitro-enclave implements all devices itself, which means it also works +on non-EC2 instances. + +If you require NSM based attestation backed by valid AWS certificates, +you must use -M nitro. The -M nitro-enclave model does not provide +you with an AWS signed attestation document. --=20 2.47.1 Amazon Web Services Development Center Germany GmbH Tamara-Danz-Str. 13 10243 Berlin Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B Sitz: Berlin Ust-ID: DE 365 538 597