From nobody Sun Apr 19 05:32:06 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6D9DC433EF for ; Tue, 5 Jul 2022 15:45:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232349AbiGEPpI (ORCPT ); Tue, 5 Jul 2022 11:45:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbiGEPpA (ORCPT ); Tue, 5 Jul 2022 11:45:00 -0400 Received: from na01-obe.outbound.protection.outlook.com (mail-eastus2azon11021016.outbound.protection.outlook.com [52.101.57.16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F15F416582; Tue, 5 Jul 2022 08:44:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n7MRc3krSPMoVjrFOtvGaGA/m5DiKGvqKzUMocU/t4Mv/rDthiOaweNk0qzVkZWupYEGRGkvIx+6kdOVnuF9fAgtoUjlQ2HJ20OGtQTQADuBuTNRZCO7ApfM07BlZQQCvLCkOwHu8N3ic1Lr5AWizgkKMZgoJhbVfBYJkj7BcPs/i1T2NyQEqthwyXEhXqMBKR3cd01rYDxwgQ51CMJ6ZANBx//TZXA+UwB8TlSLPf0mz06R7I+7Uwg4qxlFtBWasW/8LTuBYxava7yhr/nPbHK6gc6CxhgfOarV0ddchFOJZW5uXp4CuMPaeRt9VbKtVUKS0iFhhxccMnhHbsLYeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0estcobQMwuwBu15qB655d3pqGrdqiDGURVJH7AEyj4=; b=ZezcF9I4/m0xLlpYD8CAljOF6TMX6lw/Ws0fuyAbbt6ZF15o/64QCyq4KPDFZ43D+Y48rMiLj+lKOKuiRSpeBzBpg54D9odwBCLgN/pXuiqyAoBo62sDKeVXsZ6Pq1rYoTDm2JJym0+aRUk8OYy0o/g3pH2rO8wo2+YD/wLasN2MwPrqUwBXUtSGcxNvNp+hZKl7QyBUIBiQehR9GALInLragSyqJXLaLguc3XJaQCh1Ohevn9W8sG1IcV/KElVYW0Ze9U7gkULaTvGpj1m+WVkwIWUlc55qyugPpK/bzSjRNwhFzLPC57DlpMD1DLOJcxl0/vfpm/fxwwUlp0psjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0estcobQMwuwBu15qB655d3pqGrdqiDGURVJH7AEyj4=; b=I9k9uD4RRdN0u81K+1ocj6TBWMGuGHbDTels5PTzt8U6zbTmhtT1TKH8gRcRWSdBEsx3PrtGjPl9FEjNtGA9wQATx2OD3qSePi2D/7VH7N+ENvlZW9Rl1u4pBXvOI0Rp7AIK+TZRuWncTrKHtiSjGBmDZVv1CxnxF40ZEQP0FuE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1322.namprd21.prod.outlook.com (2603:10b6:5:175::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.3; Tue, 5 Jul 2022 15:44:56 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Tue, 5 Jul 2022 15:44:56 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH 1/3] Documentation: hyperv: Add overview of Hyper-V enlightenments Date: Tue, 5 Jul 2022 08:43:40 -0700 Message-Id: <1657035822-47950-2-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> References: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MWHPR14CA0014.namprd14.prod.outlook.com (2603:10b6:300:ae::24) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 702575af-9a7d-4fd5-22b9-08da5e9d4fa5 X-MS-TrafficTypeDiagnostic: DM6PR21MB1322:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iaqy1Qgn3HDvArACT8mwK+IbN0H7a304rngf9Rn8CML/pO/YztPuAGIOtfg6lPxhlIvQsq8gsJlH4dVTjIKPzTV9R7QQZN3LTRXmoSmC+176a8sMaBWU94w8lDWEUtHCv4Oh0BNBGGJqcNTI4s17ixadPZwYs7VyeWLNUhuCv7xs7FtH5mJL4di+HLQtf+eUrMlO1IXmDuPaNbAjc9hU05uInXHMr2253/8ZbcynZjUieJRDYA39PFBXpzcwf5hJ1Bo3Zw4NXvluKXEH9mXPihoME5+H9oZVtG+CWJB69zpcuCRHvySqOEAMqHtFw366cf/Bac6C8Rx2cn5N00i+nph4i3fppJroATibU216RC1P5mHDh9QjhFXWWKrnA5bWuCfXmuLDNuz+3NhFZfzJE5c8Y6A0/YAr/nzON4HqTmY6cstaz0lc/9mt3N5VXNT+QR70C5i+OGYvy6VKQPosc0asW3UchxbVarDH8Eo+znZZs0gnLZ/QI0q2E2fya+3gPTOzj6MqqRJhZmwAsEDvbRkpaPD0t5TDiSJSeL498O5LNiK1iIX5gPTm0vYZDVmMTBK5iqb6YucTj4xa8IhJUcVFlXPXIO8w8ZLuBs0I9rZcbs+or7bn6CEyDcxuBVRHIvfEWyf1wgv+MGNFMJqQBuKW+E+Y8aHXasCuD7MOOg1T0wbydZaKI9d4wPNJRvPCn8dKQLZYHoVKeBJhXnZnttTa+ZN1HExQiNjknGmilB33MKkoA/HpFerKYTcUKi6raIKf28YRRLyVHYxb2g5CjCNXT+sq3cJgSSEnJ+sn9xnPMmR2PVD/knV9GBRO+O9AFtTTJQldbITOJcelKAAumbEEXCgT0NOSdquHsig57IAYYaGkuyT7nYzNgg6GRYpZ2Wy85/WhWb5fBDGKu0YRyQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(136003)(376002)(39860400002)(396003)(366004)(451199009)(26005)(41300700001)(6506007)(2616005)(2906002)(6666004)(107886003)(66476007)(186003)(52116002)(83380400001)(30864003)(6512007)(478600001)(6486002)(966005)(86362001)(5660300002)(8936002)(4326008)(36756003)(38100700002)(38350700002)(82960400001)(66556008)(316002)(82950400001)(10290500003)(8676002)(66946007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JSB+/hhhximwy9xPP4TM33dod/B4iwxxcMMnw8JFA6eAQTrLQBT/pa5phuRU?= =?us-ascii?Q?lqbXuNPMyFONcHeVgUXsmuG3pvLv/JeEkwTQdlP0kbibdjMWB/PPAPs6AwfO?= =?us-ascii?Q?x6DKxyorcMjNilRN1g7RGp3yIJVLLrtjqOVrW8WPnRV6xpW4TUKishUaSHI6?= =?us-ascii?Q?uVBa7r5OI3vSVEAkFdrH/p4bM2zbAySK/l/ub+q7BlucvAkOf8CPiYOMLgrj?= =?us-ascii?Q?GbOl1ZSAOBjwGi7WJQIhSZ/0FvAOh3pTM0KsEx6cmzdNu1rtmdP4N+BKgEvm?= =?us-ascii?Q?sYzfxd2JIYyVMwv3TxJAO2YXoYIzlsvk4bGzK2yz8kXai2eUAWFMzQcOiB/H?= =?us-ascii?Q?IRw9TQOf3hYjH9/urM1UYVZNeNquvurmtXQQg713w4ontrPoA80ctntlO6IM?= =?us-ascii?Q?XjTbZNII0QI7E65TSVfLY0ptGjVrzYcbAmaszpOcxymO30MvcrgaI3ImgwY3?= =?us-ascii?Q?1lPTyJMiWIxIW1TKJ3MIZpVE9foA/BEZRmpgbRnGnY5nY057SIn+wj2IWDv8?= =?us-ascii?Q?sQSjsCj/Gs6EMQFCRM9lFFtiTsviBvsEOqXT9eOE05bCoxV6Ww3MlS/G3AXE?= =?us-ascii?Q?cXLDM6w8SpA81te+5JP1dTv/gHkm/HRdR93TBLccZtQxxPIso9G3myPImlYV?= =?us-ascii?Q?eGcbl6q7vjooLGOQ/1R2GIW48Jrlh+M19j4Y6k3+9xpGoDZ5gTznZ2hoaaiq?= =?us-ascii?Q?22OeDF2rTB5eXUdk0I1zQ29KjG8ow+Ai5ptcs/d86r/w4s0XVYyGmTGUIMWR?= =?us-ascii?Q?4WT0GDMqarZaun7C2xqRpdQoKHbQJMy30G+GMZab+AxbHSqXws057SUMGXVb?= =?us-ascii?Q?y2T5OwjFisYxbab91ZRT+2wV4Oj/DiWVqJH34nIGfpB3FT8KkAvvG41jamGp?= =?us-ascii?Q?KYHhsd0o7tDkES3KE+ZLysdh4zeIP7BNGIRAG7hKUQ0HvSc7wtQT6PTbw30P?= =?us-ascii?Q?47kLDo8QhNxt9tHfsUJMKBMguV+DD2o7L3x94CIxmScmkl2rVp02SMiqKTso?= =?us-ascii?Q?dI7UbAjWxvUro+Lk13zgJjRZvakTGv76Bhu8x8/MDWVd3kGqQ2baD9pTHpDs?= =?us-ascii?Q?ZA1uioNe+oQyyVPxEnT7bCHbwzhLuGKxn4yQkQVK40+2W0U2ckGpJKA315lo?= =?us-ascii?Q?4craaEHSWQlCd17rnQDqhkmfAAKoAoXWgQMi/U7uFUc6Tyz5vNXV9pg5183s?= =?us-ascii?Q?rowcMB1taZO2pcWqYL2mCfSXgj7K9gh6O0uf+wWSJue9EUt4h8vr9R0ODPM0?= =?us-ascii?Q?ixorPsjvJYJjDCKdXCADTPGgBLRPpf8iyo9Wlm4/oHjjIkL/bSuemxFLJR5R?= =?us-ascii?Q?xYu9QbxlIv3t6lwlD+6xwgnWpaSIk17QUihXqdEk8ctCDJCqk1XQMm9iy4Ss?= =?us-ascii?Q?Xk2ukfEy/o5Q3wuUT+JjqFkqkfU9J+5X71u2JPNpr4GzvEZpJgZsYmFd8hF6?= =?us-ascii?Q?K5+TOhH46guWIYe/tKVOsIlQL2B/LHBMyP3MeExHT2sriktxN3RYtwriki+1?= =?us-ascii?Q?VrJA0Q5TU0M2yC62/+P59SFLMksQw3ZIIWhEomgKqolLZj/ecOSjWo/UWVyI?= =?us-ascii?Q?6SYzD0/JvLX4MpSHeheoB+q6d+1DeSEf45yXryXejqc2gSR3Za2dkS4RUpdF?= =?us-ascii?Q?NA=3D=3D?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 702575af-9a7d-4fd5-22b9-08da5e9d4fa5 X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2022 15:44:56.6695 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Y/T4vfCWpXx98ZrhFWK02+HiYYFBw2MYNspr8V+vpYCVsIkG4j2mcsERGA/n6RNVga2KWfmAuGPjHjUgnKo9WQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1322 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an initial documentation topic for Linux enlightenments to run as a guest on Microsoft's Hyper-V hypervisor, linked under the "virt" documentation area. Update the virt doc index.rst and the MAINTAINERS file. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/index.rst | 10 ++ Documentation/virt/hyperv/overview.rst | 207 +++++++++++++++++++++++++++++= ++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 1 + 4 files changed, 219 insertions(+) create mode 100644 Documentation/virt/hyperv/index.rst create mode 100644 Documentation/virt/hyperv/overview.rst diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst new file mode 100644 index 0000000..991bee4 --- /dev/null +++ b/Documentation/virt/hyperv/index.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Hyper-V Enlightenments +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. toctree:: + :maxdepth: 1 + + overview diff --git a/Documentation/virt/hyperv/overview.rst b/Documentation/virt/hy= perv/overview.rst new file mode 100644 index 0000000..cd49333 --- /dev/null +++ b/Documentation/virt/hyperv/overview.rst @@ -0,0 +1,207 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D +The Linux kernel contains a variety of code for running as a fully +enlightened guest on Microsoft's Hyper-V hypervisor. Hyper-V +consists primarily of a bare-metal hypervisor plus a virtual machine +management service running in the parent partition (roughly +equivalent to KVM and QEMU, for example). Guest VMs run in child +partitions. In this documentation, references to Hyper-V usually +encompass both the hypervisor and the VMM service without making a +distinction about which functionality is provided by which +component. + +Hyper-V runs on x86/x64 and arm64 architectures, and Linux guests +are supported on both. The functionality and behavior of Hyper-V is +generally the same on both architectures unless noted otherwise. + +Linux Guest Communication with Hyper-V +-------------------------------------- +Linux guests communicate with Hyper-V in four different ways: + +* Implicit traps: As defined by the x86/x64 or arm64 architecture, + some guest actions trap to Hyper-V. Hyper-V emulates the action and + returns control to the guest. This behavior is generally invisible + to the Linux kernel. + +* Explicit hypercalls: Linux makes an explicit function call to + Hyper-V, passing parameters. Hyper-V performs the requested action + and returns control to the caller. Parameters are passed in + processor registers or in memory shared between the Linux guest and + Hyper-V. On x86/x64, hypercalls use a Hyper-V specific calling + sequence. On arm64, hypercalls use the ARM standard SMCCC calling + sequence. + +* Synthetic register access: Hyper-V implements a variety of + synthetic registers. On x86/x64 these registers appear as MSRs in + the guest, and the Linux kernel can read or write these MSRs using + the normal mechanisms defined by the x86/x64 architecture. On + arm64, these synthetic registers must be accessed using explicit + hypercalls. + +* VMbus: VMbus is a higher-level software construct that is built on + the other 3 mechanisms. It is a message passing interface between + the Hyper-V host and the Linux guest. It uses memory that is shared + between Hyper-V and the guest, along with various signaling + mechanisms. + +The first three communication mechanisms are documented in the +`Hyper-V Top Level Functional Spec (TLFS)`_. The TLFS describes +general Hyper-V functionality and provides details on the hypercalls +and synthetic registers. The TLFS is currently written for the +x86/x64 architecture only. + +.. _Hyper-V Top Level Functional Spec (TLFS): https://docs.microsoft.com/e= n-us/virtualization/hyper-v-on-windows/tlfs/tlfs + +VMbus is not documented. This documentation provides a high-level +overview of VMbus and how it works, but the details can be discerned +only from the code. + +Sharing Memory +-------------- +Many aspects are communication between Hyper-V and Linux are based +on sharing memory. Such sharing is generally accomplished as +follows: + +* Linux allocates memory from its physical address space using + standard Linux mechanisms. + +* Linux tells Hyper-V the guest physical address (GPA) of the + allocated memory. Many shared areas are kept to 1 page so that a + single GPA is sufficient. Larger shared areas require a list of + GPAs, which usually do not need to be contiguous in the guest + physical address space. How Hyper-V is told about the GPA or list + of GPAs varies. In some cases, a single GPA is written to a + synthetic register. In other cases, a GPA or list of GPAs is sent + in a VMbus message. + +* Hyper-V translates the GPAs into "real" physical memory addresses, + and creates a virtual mapping that it can use to access the memory. + +* Linux can later revoke sharing it has previously established by + telling Hyper-V to set the shared GPA to zero. + +Hyper-V operates with a page size of 4 Kbytes. GPAs communicated to +Hyper-V may be in the form of page numbers, and always describe a +range of 4 Kbytes. Since the Linux guest page size on x86/x64 is +also 4 Kbytes, the mapping from guest page to Hyper-V page is 1-to-1. +On arm64, Hyper-V supports guests with 4/16/64 Kbyte pages as +defined by the arm64 architecture. If Linux is using 16 or 64 +Kbyte pages, Linux code must be careful to communicate with Hyper-V +only in terms of 4 Kbyte pages. HV_HYP_PAGE_SIZE and related macros +are used in code that communicates with Hyper-V so that it works +correctly in all configurations. + +As described in the TLFS, a few memory pages shared between Hyper-V +and the Linux guest are "overlay" pages. With overlay pages, Linux +uses the usual approach of allocating guest memory and telling +Hyper-V the GPA of the allocated memory. But Hyper-V then replaces +that physical memory page with a page it has allocated, and the +original physical memory page is no longer accessible in the guest +VM. Linux may access the memory normally as if it were the memory +that it originally allocated. The "overlay" behavior is visible +only because the contents of the page (as seen by Linux) change at +the time that Linux originally establishes the sharing and the +overlay page is inserted. Similarly, the contents change if Linux +revokes the sharing, in which case Hyper-V removes the overlay page, +and the guest page originally allocated by Linux becomes visible +again. + +Before Linux does a kexec to a kdump kernel or any other kernel, +memory shared with Hyper-V should be revoked. Hyper-V could modify +a shared page or remove an overlay page after the new kernel is +using the page for a different purpose, corrupting the new kernel. +Hyper-V does not provide a single "set everything" operation to +guest VMs, so Linux code must individually revoke all sharing before +doing kexec. See hv_kexec_handler() and hv_crash_handler(). But +the crash/panic path still has holes in cleanup because some shared +pages are set using per-CPU synthetic registers and there's no +mechanism to revoke the shared pages for CPUs other than the CPU +running the panic path. + +CPU Management +-------------- +Hyper-V does not have a ability to hot-add or hot-remove a CPU +from a running VM. However, Windows Server 2019 Hyper-V and +earlier versions may provide guests with ACPI tables that indicate +more CPUs than are actually present in the VM. As is normal, Linux +treats these additional CPUs as potential hot-add CPUs, and reports +them as such even though Hyper-V will never actually hot-add them. +Starting in Windows Server 2022 Hyper-V, the ACPI tables reflect +only the CPUs actually present in the VM, so Linux does not report +any hot-add CPUs. + +A Linux guest CPU may be taken offline using the normal Linux +mechanisms, provided no VMbus channel interrupts are assigned to +the CPU. See the section on VMbus Interrupts for more details +on how VMbus channel interrupts can be re-assigned to permit +taking a CPU offline. + +32-bit and 64-bit +----------------- +On x86/x64, Hyper-V supports 32-bit and 64-bit guests, and Linux +will build and run in either version. While the 32-bit version is +expected to work, it is used rarely and may suffer from undetected +regressions. + +On arm64, Hyper-V supports only 64-bit guests. + +Endian-ness +----------- +All communication between Hyper-V and guest VMs uses Little-Endian +format on both x86/x64 and arm64. Big-endian format on arm64 is not +supported by Hyper-V, and Linux code does not use endian-ness macros +when accessing data shared with Hyper-V. + +Versioning +---------- +Current Linux kernels operate correctly with older versions of +Hyper-V back to Windows Server 2012 Hyper-V. Support for running +on the original Hyper-V release in Windows Server 2008/2008 R2 +has been removed. + +A Linux guest on Hyper-V outputs in dmesg the version of Hyper-V +it is running on. This version is in the form of a Windows build +number and is for display purposes only. Linux code does not +test this version number at runtime to determine available features +and functionality. Hyper-V indicates feature/function availability +via flags in synthetic MSRs that Hyper-V provides to the guest, +and the guest code tests these flags. + +VMbus has its own protocol version that is negotiated during the +initial VMbus connection from the guest to Hyper-V. This version +number is also output to dmesg during boot. This version number +is checked in a few places in the code to determine if specific +functionality is present. + +Furthermore, each synthetic device on VMbus also has a protocol +version that is separate from the VMbus protocol version. Device +drivers for these synthetic devices typically negotiate the device +protocol version, and may test that protocol version to determine +if specific device functionality is present. + +Code Packaging +-------------- +Hyper-V related code appears in the Linux kernel code tree in three +main areas: + +1. drivers/hv + +2. arch/x86/hyperv and arch/arm64/hyperv + +3. individual device driver areas such as drivers/scsi, drivers/net, + drivers/clocksource, etc. + +A few miscellaneous files appear elsewhere. See the full list under +"Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIVER FOR HYPERV +SYNTHETIC VIDEO DEVICE" in the MAINTAINERS file. + +The code in #1 and #2 is built only when CONFIG_HYPERV is set. +Similarly, the code for most Hyper-V related drivers is built only +when CONFIG_HYPERV is set. + +Most Hyper-V related code in #1 and #3 can be built as a module. +The architecture specific code in #2 must be built-in. Also, +drivers/hv/hv_common.c is low-level code that is common across +architectures and must be built-in. diff --git a/Documentation/virt/index.rst b/Documentation/virt/index.rst index 492f092..2f1cffa 100644 --- a/Documentation/virt/index.rst +++ b/Documentation/virt/index.rst @@ -14,6 +14,7 @@ Linux Virtualization Support ne_overview acrn/index coco/sev-guest + hyperv/index =20 .. only:: html and subproject =20 diff --git a/MAINTAINERS b/MAINTAINERS index a6d3bd9..fd2e5ce 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9172,6 +9172,7 @@ S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git F: Documentation/ABI/stable/sysfs-bus-vmbus F: Documentation/ABI/testing/debugfs-hyperv +F: Documentation/virt/hyperv F: Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst F: arch/arm64/hyperv F: arch/arm64/include/asm/hyperv-tlfs.h --=20 1.8.3.1 From nobody Sun Apr 19 05:32:06 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F872C433EF for ; Tue, 5 Jul 2022 15:45:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232360AbiGEPpO (ORCPT ); Tue, 5 Jul 2022 11:45:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232223AbiGEPpC (ORCPT ); Tue, 5 Jul 2022 11:45:02 -0400 Received: from na01-obe.outbound.protection.outlook.com (mail-eastus2azon11021016.outbound.protection.outlook.com [52.101.57.16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 439BF11A1D; Tue, 5 Jul 2022 08:45:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HyouXjCPCn7jtDSoXVpwW4VdNYf+HGPJZWMvfDspZuXfIs0NWMRo3qjSiiLIzUuFvPCJy4zL3p1b2NlfYJnlFkI+OCEOFpmXdJhvaL2wLFOx/6CaoALiSBHEpX6PBkgi0ApQ53Rb/HEjsqzBTOKfLYHjG6y9HmEtdfucwGyQbZJRDq1F6y3e9HgnBCTRk+kyYt5ZupeTZrAbAQ4kTV50RmwHgOKvnzfPNxkK2cC5PCrQ39PGl7WLqExfU9uemebJo8QpMHXej9NB1twzfar0BNiTVJ1CyWO6ClSSk/qpqa7gLzyRtVy6cVK5MU+75lJFBfZEMuBGTMbWLQvt9usE4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xsPeFnFqp/QHKTTfaOcuAGPTm3tbONgQZt9ZEvaVM98=; b=leB86EjX0VsZ2/g9MkhjFAN9G2+6iZwMTB8idYpbz7gKfuZ73Nvz8DvrrLo/YQuViWP7qlTRJYLyzppMqZQy3qp85pJB4qPCFyYp7wxkAGYabOD5iSCatAH6qMgpZ7RGrWivmdttk4xv6imspH/1cqZ6eTrnQRJaYmC7AUQVFaAKp0YIMfZOieH7mTOkRoRnGFHHT6kBxatOxR3NC2ighSZiuDdzQiwl3hPja68ELA+nfdZ2cF1T52piWCE9YI8oSRWodBtYwCpUmv/hRgm1mT6/WrI9z+FewwzUvvGJJ9C6jrSp29ZcN4aQYXLKTe5Vgs+/gezmuaok+DCEFfZM2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xsPeFnFqp/QHKTTfaOcuAGPTm3tbONgQZt9ZEvaVM98=; b=icTVqkECjDC0T6FE0d8ZhH2mlMIjxDzjVZoH/WBDuIyHx7POZ5V9sqkP9HvwVuwiJxeswS32diDq3Ki0Q3OFDrF8P8GYT+BNWISKkAsX1HKdvhvBtI8jMexyapQYy4umRyT7htI/Oh+l9WQlx48BKiOlW2E7oH6Gu7nruzSJW6w= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1322.namprd21.prod.outlook.com (2603:10b6:5:175::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.3; Tue, 5 Jul 2022 15:44:57 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Tue, 5 Jul 2022 15:44:57 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH 2/3] Documentation: hyperv: Add overview of VMbus Date: Tue, 5 Jul 2022 08:43:41 -0700 Message-Id: <1657035822-47950-3-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> References: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MWHPR14CA0014.namprd14.prod.outlook.com (2603:10b6:300:ae::24) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1da7b165-c81c-4ea6-2fa4-08da5e9d5015 X-MS-TrafficTypeDiagnostic: DM6PR21MB1322:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aFw4sn97/SRuJUuqjwAkXBGtOheCmofL/bGmpDGZRWEmK5HshAuCxa8wt+sywWO2N4xs49tKyTTNLqNA4S86Q3LaCR/3yQn1mdrksBQor9QgYJ3jgcNyjCrJzG8Ou2Vv8/pLju1/UdDB97AztXNjNyd8Msc1eK3LSXxRwiy20NXp0rw8PjkJDn6+Uq6z0mtJeGO1hM1C5plxHs6xazO3N9H0hp2bq0EeSwCJdX0i0Is0QCnmlkDPR20dT4/4Tbk47M5tEl/OO08AnFoSBOdHm06FOqT9OubbYJEHuyVxfMmC4H/Zjrrd/XBxbgYrxdXmtkZj/gSVAafRvxRL6S89HsH9x7KfUKGIb3+EH1LAFSCSFfU4kTA7U/3Bu805dvwTKddBtwgP61KAuZAeBjHAANaEJeYNFmnVakzfiPz10ODLqKu4NRlEJ3Hk+mi7ZpzE5r4WsIdALZw5QgXTVFsZVuw/48uS4z/RIeAte2QOeciS/E8npw4Hgm5IWcKf4KpUX/5NjKTnG7fbPWS9SdfYkMbmKIrppvPdTKWpjarrxySJnT/GMCelXvdNX7btmWWKZs87476mOyP6nDBwrPTy6Rae9Nwm/A9SFzFYZbYRWUxaMEngjaEZJ8ZYNcNftR8CEQbRcIuA5E1apUcn7EULsKKLIzPo0mMJrl289jxUwENFcuLPgvKolvMie8Nxy214eBugptVxKYje1+/Sm5o0jgqlzvXovRNWh20XrQwocui7By10PIFvIhMdEzz6I+TRVZHLCIhPTXzf4u/i0ZrQynvJnNWETP44pWB8Ga2+n1hjLvWx8qDwI8LzwlV8j1JnDbAirdVIk2Ytt6i5ReXR8aWI+JnZkcuEletKnUqmz4k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(136003)(376002)(39860400002)(396003)(366004)(451199009)(26005)(41300700001)(6506007)(2616005)(2906002)(6666004)(107886003)(66476007)(186003)(52116002)(83380400001)(30864003)(6512007)(478600001)(6486002)(86362001)(5660300002)(8936002)(4326008)(36756003)(38100700002)(38350700002)(82960400001)(66556008)(316002)(82950400001)(10290500003)(8676002)(18074004)(66946007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?G83/AQeH3Bt7i7j4tKl47C5f9T5Lay19ofmpa7JnAt9aVKys+wov4zkXa3Rh?= =?us-ascii?Q?BK5s40+0BOEV+QmnSgeoA6ApNwwMw4IuBRN/DHJfO/HQRtFwy0vCPGj/LIrb?= =?us-ascii?Q?7yE/0NwhF5apWsWXH1XrKZCj+3IxMYt1mvAyDiB2ZjjNEExe2BFclNoluWhn?= =?us-ascii?Q?TMlJuWewlPUQ9ZxneIdrJMmIzdO5osUU/s2lu7SEbF2o8inSbRagO8S5Y5+N?= =?us-ascii?Q?CjCrV4f753cbtodI5WR+5+Zi8tHQoY0VHOCyFJjYtc9EPqZKihIhTek4ium2?= =?us-ascii?Q?lhkybG3eG9lsgN2sh4LXRgd5qtAz6/Y1TIcnBqRTI08bSPJtaJYF8r7ECGeU?= =?us-ascii?Q?ZWf2IbmhdeV/EaYOKSvve0CSn7vThwjTwlzkjjvKsaa1CsBc7zTNBdlK0AT8?= =?us-ascii?Q?BF2JMzBYbISqkFbCCnJRntl4HWUFi4FHx5HtBs0OULKs8urfePax/wuumH7x?= =?us-ascii?Q?o3nV4iFOB9KWxMLa9u6Nwes0FehYhAdsSgZ8CpbYJovijvXchhapU1kAW95m?= =?us-ascii?Q?70tiD+BeHKmEPJ19wEAO6thXzH1CNDFhK2l3KYa3Ghs20ElUMm993RZnO6sB?= =?us-ascii?Q?WdlgC0Zs+RtH4j39zkD5cA6qIl30MlhHLT1ryer70LvK9lWaEYKvo7bcVhiX?= =?us-ascii?Q?2yQYH56YnoAhvffG+H5IMnJl3mHJ+bdvW942CpBhiFTqEHg/0Z7zuqFo2lWz?= =?us-ascii?Q?MifMfgD5dLr+RyrzS3d621lkH+/g/54g21zqNHLAwW5rVoxQGkCzHRDNPhUW?= =?us-ascii?Q?MBsKxjmXs3EZWAAL2VerV+yC0+ZA56/XbbowjS+kW6VlyqgtXqKqaZ+P8soM?= =?us-ascii?Q?V2i1aN+rVrH5G9yFPlDJZsdjQNROtx2pXNnotOTyeP0kn6+aT4kGQA+MPVr9?= =?us-ascii?Q?gXZ7iLceV4rTvNZVCr/okgFBV6DGyCDsEChNchlrhx33UwBqR28WH4d+c6wR?= =?us-ascii?Q?Ff9InFEZwDVdImELxO2uf//wzeIbED4M8ydY8QDauSFoxKF+lt/HQWRIaP5r?= =?us-ascii?Q?WMOr+wpOHQG0WB8FnYgCpJMhl37xwgpXv6IbkbT5TQI0+5GPeVZw7azjHiU2?= =?us-ascii?Q?7QEuv02DJE74I8g+9ij9vAZVND49tn+9V/m2AW5oGQmY1nYguSi5MQ8llWkw?= =?us-ascii?Q?xAaUysdYKw8lF9U8dRh/AzKsZL+8kLvRveSWk84CUJ6dPCTYmE91m1qqlm3e?= =?us-ascii?Q?rn0n0vV+ofCVeLhRzdhwAz7e+iFOeqFcNqYPc+r9qFu9DlBwRj34+mAgU4iz?= =?us-ascii?Q?PTRHa+yOpEL694bIj6X/zrOGKeR3arOYCpenOhYLLTKKiL4WhkuDuCyP/eKD?= =?us-ascii?Q?jj2Deeld+VqTKuO2py30iQZtvx2QjBJZqHxga/P0huZnnHN1MG1WneexLF2N?= =?us-ascii?Q?SYluKI+G0GlKJNnZxWGWaJ35R+0Nuq730gZegAWpPXKdUnchEGCT0nugwIb3?= =?us-ascii?Q?gkRPDv+Y//szCXhyHadQmUULQWjdLnagcBfqqXKegGIPv3AKAQel7Lkft6nr?= =?us-ascii?Q?eIfI5xIJxWMzQ6gwco95XYmaYCds1ooPuJvvfvdSI9tLjvJRCl7zTwyervGM?= =?us-ascii?Q?ziMWCA5yL49yn8IdyfeFQ6K7bsfPzq+zxEPC3ZVOeOnvlmj66AizmmZE+uWY?= =?us-ascii?Q?WA=3D=3D?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1da7b165-c81c-4ea6-2fa4-08da5e9d5015 X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2022 15:44:57.4340 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XeC7MfUmObtgHjkN1ZvRjazb3V+sweBNG7XjkaL6TzLxZMCsbNQFx22p+/iMPXvijlN/XywmxI42/OXQ5TzR/w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1322 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation topic for using VMbus when running as a guest on Hyper-V. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/index.rst | 1 + Documentation/virt/hyperv/vmbus.rst | 303 ++++++++++++++++++++++++++++++++= ++++ 2 files changed, 304 insertions(+) create mode 100644 Documentation/virt/hyperv/vmbus.rst diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst index 991bee4..caa43ab 100644 --- a/Documentation/virt/hyperv/index.rst +++ b/Documentation/virt/hyperv/index.rst @@ -8,3 +8,4 @@ Hyper-V Enlightenments :maxdepth: 1 =20 overview + vmbus diff --git a/Documentation/virt/hyperv/vmbus.rst b/Documentation/virt/hyper= v/vmbus.rst new file mode 100644 index 0000000..a9de881 --- /dev/null +++ b/Documentation/virt/hyperv/vmbus.rst @@ -0,0 +1,303 @@ +.. SPDX-License-Identifier: GPL-2.0 + +VMbus +=3D=3D=3D=3D=3D +VMbus is a software construct provided by Hyper-V to guest VMs. It +consists of a control path and common facilities used by synthetic +devices that Hyper-V presents to guest VMs. The control path is +used to offer synthetic devices to the guest VM and, in some cases, +to rescind those devices. The common facilities include software +channels for communicating between the device driver in the guest VM +and the synthetic device implementation that is part of Hyper-V, and +signaling primitives to allow Hyper-V and the guest to interrupt +each other. + +VMbus is modeled in Linux as a bus, with the expected /sys/bus/vmbus +entry in a running Linux guest. The VMbus driver (drivers/hv/vmbus_drv.c) +establishes the VMbus control path with the Hyper-V host, then +registers itself as a Linux bus driver. It implements the standard +bus functions for adding and removing devices to/from the bus. + +Most synthetic devices offered by Hyper-V have a corresponding Linux +device driver. These devices include: + +* SCSI controller +* NIC +* Graphics frame buffer +* Keyboard +* Mouse +* PCI device pass-thru +* Heartbeat +* Time Sync +* Shutdown +* Memory balloon +* Key/Value Pair (KVP) exchange with Hyper-V +* Hyper-V online backup (a.k.a. VSS) + +Guest VMs may have multiple instances of the synthetic SCSI +controller, synthetic NIC, and PCI pass-thru devices. Other +synthetic devices are limited to a single instance per VM. Not +listed above are a small number of synthetic devices offered by +Hyper-V that are used only by Windows guests and for which Linux +does not have a driver. + +Hyper-V uses the terms "VSP" and "VSC" in describing synthetic +devices. "VSP" refers to the Hyper-V code that implements a +particular synthetic device, while "VSC" refers to the driver for +the device in the guest VM. For example, the Linux driver for the +synthetic NIC is referred to as "netvsc" and the Linux driver for +the synthetic SCSI controller is "storvsc". These drivers contain +functions with names like "storvsc_connect_to_vsp". + +VMbus channels +-------------- +An instance of a synthetic device uses VMbus channels to communicate +between the VSP and the VSC. Channels are bi-directional and used +for passing messages. Most synthetic devices use a single channel, +but the synthetic SCSI controller and synthetic NIC may use multiple +channels to achieve higher performance and greater parallelism. + +Each channel consists of two ring buffers. These are classic ring +buffers from a university data structures textbook. If the read +and writes pointers are equal, the ring buffer is considered to be +empty, so a full ring buffer always has at least one byte unused. +The "in" ring buffer is for messages from the Hyper-V host to the +guest, and the "out" ring buffer is for messages from the guest to +the Hyper-V host. In Linux, the "in" and "out" designations are as +viewed by the guest side. The ring buffers are memory that is +shared between the guest and the host, and they follow the standard +paradigm where the memory is allocated by the guest, with the list +of GPAs that make up the ring buffer communicated to the host. Each +ring buffer consists of a header page (4 Kbytes) with the read and +write indices and some control flags, followed by the memory for the +actual ring. The size of the ring is determined by the VSC in the +guest and is specific to each synthetic device. The list of GPAs +making up the ring is communicated to the Hyper-V host over the +VMbus control path as a GPA Descriptor List (GPADL). See function +vmbus_establish_gpadl(). + +Each ring buffer is mapped into contiguous Linux kernel virtual +space in three parts: 1) the 4 Kbyte header page, 2) the memory +that makes up the ring itself, and 3) a second mapping of the memory +that makes up the ring itself. Because (2) and (3) are contiguous +in kernel virtual space, the code that copies data to and from the +ring buffer need not be concerned with ring buffer wrap-around. +Once a copy operation has completed, the read or write index may +need to be reset to point back into the first mapping, but the +actual data copy does not need to be broken into two parts. This +approach also allows complex data structures to be easily accessed +directly in the ring without handling wrap-around. + +On arm64 with page sizes > 4 Kbytes, the header page must still be +passed to Hyper-V as a 4 Kbyte area. But the memory for the actual +ring must be aligned to PAGE_SIZE and have a size that is a multiple +of PAGE_SIZE so that the duplicate mapping trick can be done. Hence +a portion of the header page is unused and not communicated to +Hyper-V. This case is handled by vmbus_establish_gpadl(). + +Hyper-V enforces a limit on the aggregate amount of guest memory +that can be shared with the host via GPADLs. This limit ensures +that a rogue guest can't force the consumption of excessive host +resources. For Windows Server 2019 and later, this limit is +approximately 1280 Mbytes. For versions prior to Windows Server +2019, the limit is approximately 384 Mbytes. + +VMbus messages +-------------- +All VMbus messages have a standard header that includes the message +length, the offset of the message payload, some flags, and a +transactionID. The portion of the message after the header is +unique to each VSP/VSC pair. + +Messages follow one of two patterns: + +* Unidirectional: Either side sends a message and does not + expect a response message +* Request/response: One side (usually the guest) sends a message + and expects a response + +The transactionID (a.k.a. "requestID") is for matching requests & +responses. Some synthetic devices allow multiple requests to be in- +flight simultaneously, so the guest specifies a transactionID when +sending a request. Hyper-V sends back the same transactionID in the +matching response. + +Messages passed between the VSP and VSC are control messages. For +example, a message sent from the storvsc driver might be "execute +this SCSI command". If a message also implies some data transfer +between the guest and the Hyper-V host, the actual data to be +transferred may be embedded with the control message, or it may be +specified as a separate data buffer that the Hyper-V host will +access as a DMA operation. The former case is used when the size of +the data is small and the cost of copying the data to and from the +ring buffer is minimal. For example, time sync messages from the +Hyper-V host to the guest contain the actual time value. When the +data is larger, a separate data buffer is used. In this case, the +control message contains a list of GPAs that describe the data +buffer. For example, the storvsc driver uses this approach to +specify the data buffers to/from which disk I/O is done. + +Three functions exist to send VMbus messages: + +1. vmbus_sendpacket(): Control-only messages and messages with + embedded data -- no GPAs +2. vmbus_sendpacket_pagebuffer(): Message with list of GPAs + identifying data to transfer. An offset and length is + associated with each GPA so that multiple discontinuous areas + of guest memory can be targeted. +3. vmbus_sendpacket_mpb_desc(): Message with list of GPAs + identifying data to transfer. A single offset and length is + associated with a list of GPAs. The GPAs must describe a + single logical area of guest memory to be targeted. + +Historically, Linux guests have trusted Hyper-V to send well-formed +and valid messages, and Linux drivers for synthetic devices did not +fully validate messages. With the introduction of processor +technologies that fully encrypt guest memory and that allow the +guest to not trust the hypervisor (AMD SNP-SEV, Intel TDX), trusting +the Hyper-V host is no longer a valid assumption. The drivers for +VMbus synthetic devices are being updated to fully validate any +values read from memory that is shared with Hyper-V, which includes +messages from VMbus devices. To facilitate such validation, +messages read by the guest from the "in" ring buffer are copied to a +temporary buffer that is not shared with Hyper-V. Validation is +performed in this temporary buffer without the risk of Hyper-V +maliciously modifying the message after it is validated but before +it is used. + +VMbus interrupts +---------------- +VMbus provides a mechanism for the guest to interrupt the host when +the guest has queued new messages in a ring buffer. The host +expects that the guest will send an interrupt only when an "out" +ring buffer transitions from empty to non-empty. If the guest sends +interrupts at other times, the host deems such interrupts to be +unnecessary. If a guest sends an excessive number of unnecessary +interrupts, the host may throttle that guest by suspending its +execution for a few seconds to prevent a denial-of-service attack. + +Similarly, the host will interrupt the guest when it sends a new +message on the VMbus control path, or when a VMbus channel "in" ring +buffer transitions from empty to non-empty. Each CPU in the guest +may receive VMbus interrupts, so they are best modeled as per-CPU +interrupts in Linux. This model works well on arm64 where a single +per-CPU IRQ is allocated for VMbus. Since x86/x64 lacks support for +per-CPU IRQs, an x86 interrupt vector is statically allocated (see +HYPERVISOR_CALLBACK_VECTOR) across all CPUs and explicitly coded to +call the VMbus interrupt service routine. These interrupts are +visible in /proc/interrupts on the "HYP" line. + +The guest CPU that a VMbus channel will interrupt is selected by the +guest when the channel is created, and the host is informed of that +selection. VMbus devices are broadly grouped into two categories: + +1. "Slow" devices that need only one VMbus channel. The devices + (such as keyboard, mouse, heartbeat, and timesync) generate + relatively few interrupts. Their VMbus channels are all + assigned to interrupt the VMBUS_CONNECT_CPU, which is always + CPU 0. + +2. "High speed" devices that may use multiple VMbus channels for + higher parallelism and performance. These devices include the + synthetic SCSI controller and synthetic NIC. Their VMbus + channels interrupts are assigned to CPUs that are spread out + among the available CPUs in the VM so that interrupts on + multiple channels can be processed in parallel. + +The assignment of VMbus channel interrupts to CPUs is done in the +function init_vp_index(). This assignment is done outside of the +normal Linux interrupt affinity mechanism, so the interrupts are +neither "unmanaged" nor "managed" interrupts. + +The CPU that a VMbus channel will interrupt can be seen in +/sys/bus/vmbus/devices// channels//cpu. +When running on later versions of Hyper-V, the CPU can be changed +by writing a new value to this sysfs entry. Because the interrupt +assignment is done outside of the normal Linux affinity mechanism, +there are no entries in /proc/irq corresponding to individual +VMbus channel interrupts. + +An online CPU in a Linux guest may not be taken offline if it has +VMbus channel interrupts assigned to it. Any such channel +interrupts must first be manually reassigned to another CPU as +described above. When no channel interrupts are assigned to the +CPU, it can be taken offline. + +When a guest CPU receives a VMbus interrupt from the host, the +function vmbus_isr() handles the interrupt. It first checks for +channel interrupts by calling vmbus_chan_sched(), which looks at a +bitmap setup by the host to determine which channels have pending +interrupts on this CPU. If multiple channels have pending +interrupts for this CPU, they are processed sequentially. When all +channel interrupts have been processed, vmbus_isr() checks for and +processes any message received on the VMbus control path. + +The VMbus channel interrupt handling code is designed to work +correctly even if an interrupt is received on a CPU other than the +CPU assigned to the channel. Specifically, the code does not use +CPU-based exclusion for correctness. In normal operation, Hyper-V +will interrupt the assigned CPU. But when the CPU assigned to a +channel is being changed via sysfs, the guest doesn't know exactly +when Hyper-V will make the transition. The code must work correctly +even if there is a time lag before Hyper-V starts interrupting the +new CPU. See comments in target_cpu_store(). + +VMbus device creation/deletion +------------------------------ +Hyper-V and the Linux guest have a separate message-passing path +that is used for synthetic device creation and deletion. This +path does not use a VMbus channel. See vmbus_post_msg() and +vmbus_on_msg_dpc(). + +The first step is for the guest to connect to the generic +Hyper-V VMbus mechanism. As part of establishing this connection, +the guest and Hyper-V agree on a VMbus protocol version they will +use. This negotiation allows newer Linux kernels to run on older +Hyper-V versions, and vice versa. + +The guest then tells Hyper-v to "send offers". Hyper-V sends an +offer message to the guest for each synthetic device that the VM +is configured to have. Each VMbus device type has a fixed GUID +known as the "class ID", and each VMbus device instance is also +identified by a GUID. The offer message from Hyper-V contains +both GUIDs to uniquely (within the VM) identify the device. +There is one offer message for each device instance, so a VM with +two synthetic NICs will get two offers messages with the NIC +class ID. The ordering of offer messages can vary from boot-to-boot +and must not be assumed to be consistent in Linux code. Offer +message may also arrive long after Linux has initially booted +because Hyper-V supports adding devices, such as synthetic NICs, +to running VMs. A new offer message is processed by +vmbus_process_offer(), which indirectly invokes vmbus_add_channel_work(). + +Upon receipt of an offer message, the guest identifies the device +type based on the class ID, and invokes the correct driver to set up +the device. Driver/device matching is performed using the standard +Linux mechanism. + +The device driver probe function opens the primary VMbus channel to +the corresponding VSP. It allocates guest memory for the channel +ring buffers and shares the ring buffer with the Hyper-V host by +giving the host a list of GPAs for the ring buffer memory. See +vmbus_establish_gpadl(). + +Once the ring buffer is set up, the device driver and VSP exchange +setup messages via the primary channel. These messages may include +negotiating the device protocol version to be used between the Linux +VSC and the VSP on the Hyper-V host. The setup messages may also +include creating additional VMbus channels, which are somewhat +mis-named as "sub-channels" since they are functionally +equivalent to the primary channel once they are created. + +Finally, the device driver may create entries in /dev as with +any device driver. + +The Hyper-V host can send a "rescind" message to the guest to +remove a device that was previously offered. Linux drivers must +handle such a rescind message at any time. Rescinding a device +invokes the device driver "remove" function to cleanly shut +down the device and remove it. Once a synthetic device is +rescinded, neither Hyper-V nor Linux retains any state about +its previous existence. Such a device might be re-added later, +in which case it is treated as an entirely new device. See +vmbus_onoffer_rescind(). --=20 1.8.3.1 From nobody Sun Apr 19 05:32:06 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52540C43334 for ; Tue, 5 Jul 2022 15:45:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232390AbiGEPpU (ORCPT ); Tue, 5 Jul 2022 11:45:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232238AbiGEPpD (ORCPT ); Tue, 5 Jul 2022 11:45:03 -0400 Received: from na01-obe.outbound.protection.outlook.com (mail-eastus2azon11021016.outbound.protection.outlook.com [52.101.57.16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E352D15FF6; Tue, 5 Jul 2022 08:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WGcW1iZjci/xkw/8vjy3lPc//sEoyb4fx4AANjCUe3djUX6sln7W9Y16oKWLW4xasBakOkejc10Nzyguz8Zw5vLjStZGVVkC68hCD0UT0O4X7D59FxU7gr6J1PH9eHucE4WQdVXluxcTt48wwnr0gRlku322a9xE6sQN1IckVc8Wyy2mdrQKqYqJZhgzkFyhVtgb8oLWc4MhkZx04oFABW4wPPL/zLix3WYwAXxjUS45VYF3rpyxV+YjT63+Z5iY26w/xKGg4Cm1GcBJU5cUylmPT5ROdSLWEh7f+I9zwa87fznCxOCxx0UEUNFLocRU/Ud5xcnnxYpBjze+raJ6oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dRGms1+lb5F45Tnp5jmqex2n50PDIorw+aTDag81nWo=; b=hM6YtkXkbKLyczQyEjtz9+JyjIgqrr6rJZf0bhN+jrma6czK0lnDjzKZB1EAVQi7JgTfJK5ZpGmNQgrg8hnBefNpzCa/VxP17QUEHd+oI5H9mHB7fQdGvYRVKAfnv+GcOUu/6akmZK+89Eyp8GwwpmSX44uAgvOIocTAA/1YFRTZJFxdbin0ATtP7DcKRsuBDnb7/8WbyuAQcPOGfyqRxnF2Drp6gaG1+M26LLajjEshDvT2Agd/gd8C6Tn9z/9zu2QDbJuW0o3dqtbw0Sv0TKeNrsAkQOBOh/zqjefCXTAncaXpKl/pgJpxpOBUMANe7MLSyVgCl22cIsgs4ZnFRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dRGms1+lb5F45Tnp5jmqex2n50PDIorw+aTDag81nWo=; b=e+Hkqyxpwce7Sz6kjR1F1thHkU3DRhJNOoeVG9vmX7gJlyONJeUojO93JMVut9MiU9pCUtJW9Vh5gdzppqN9kIImVkesutz+/xheF0iOOoUiI7c2DVh6cW/OAG8MI+FiIcvg6b697862DF3QXfsTGXDeNYNeciqk1jlpOAU+0Rs= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1322.namprd21.prod.outlook.com (2603:10b6:5:175::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.3; Tue, 5 Jul 2022 15:44:58 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Tue, 5 Jul 2022 15:44:58 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH 3/3] Documentation: hyperv: Add overview of clocks and timers Date: Tue, 5 Jul 2022 08:43:42 -0700 Message-Id: <1657035822-47950-4-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> References: <1657035822-47950-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MWHPR14CA0014.namprd14.prod.outlook.com (2603:10b6:300:ae::24) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4e5fbbb0-2f61-4899-7a52-08da5e9d5089 X-MS-TrafficTypeDiagnostic: DM6PR21MB1322:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ujwNx1WL8IoISK8nRXY3W+qNElLMJIbdJklwbYWllR+dqXr2blaTCseaUnmKKltxHonXRyvhuZIFUwTgWp2Kvxbkx7djyP+XJycy63/g2L+5ZDbyAuo4VTuegxSjC7PNVRhZvINHrrUZi0koNwk3fDp+M/bU3ql3BTWhPzliJwBzW7ngQpkVPb5XzUhk9gV5EZdryNpUdtsum/ndRJainx5Z/T8aial4bnrLs/mlHpe7uOXKi+7GCKp4zJf5PF31MYeIKOshVwNByydxGi7muh0hDQwl+22WQ18v14QBGHIV0i2YZYcvqOZr8PUxzXAZ6xs20hpPT54pGz6NQ91Z30zNTMzRsMfsqQTOsbiXMmhmjUt+3vDupCXrODVwuRzVA7FsCjLdKXSmxlY4MUK8GHNLqmwIoSv33o2j1KKzYntEtpWfsn6be1BnhNAB5vUcDELa/fKounqI5dUaj3HpyhGBbgwHKKkHIlFlrudFIUmS6wjcYhEa0AP757eYFYBRl3OcE3uGBA5vQEW8sBv2kK0JSTp+DqZEJBaONOCLVfqMZxAmcFyKKDutlK8jbc4srEbGPgBEJdKpnoJPrcsqhaB4K7SQcq6EseDhHP4SJiMVtlMzxEowN/LgqkygqSoM3px0/DegHpRxqn50FpnSrDjgjOHTw+97DMjfyacPLgO264oeSWDk4FeGrXH4Q6sTJ72uDYEAQenkYevCdbibcbE3XfaP/1o4GfW2urZB9dsI2J88gR3i7BnOPpYwXA/8Oadz+wDvv+7eLVb2Pq2q6XUFr2W3061czkFs3PFczBgBrDnvSysuSvP+x2vvjSpytcYa34FCyhu9b9AisK6f9cKRdvDoirb75FJ++GRJAI4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(136003)(376002)(39860400002)(396003)(366004)(451199009)(26005)(41300700001)(6506007)(2616005)(2906002)(6666004)(107886003)(66476007)(186003)(52116002)(83380400001)(6512007)(478600001)(6486002)(86362001)(5660300002)(8936002)(4326008)(36756003)(38100700002)(38350700002)(82960400001)(66556008)(316002)(82950400001)(10290500003)(8676002)(66946007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YhS23irVvOu9Q1ayWhzIN4OsGhSVimvGtOXFWoBTF0lV/eXTrvf4TYzTAPvn?= =?us-ascii?Q?In64R3aFoTEncO17mXeDNvZhVuUYwdmK59Ug5/NWxH+zpGV21KeI6HrwqSKL?= =?us-ascii?Q?Jca+KX32DAJNSL8TqUOw3A7XAaMUenAK5uYj7lqpj/mRAYV1rM85DL0ZDuyb?= =?us-ascii?Q?WZfoxfoI7phprDI3wUudIVA1EfQw9Naplu3myjjYT1lozgvVxInDHpHLucXn?= =?us-ascii?Q?sW9f9v6sTQbxqe3dZ2usAkyOu5yMuBXpr5v7OUnuqgiqfcunzy1qQcDL78XF?= =?us-ascii?Q?bruNl1pTC9bQYscmZVjRTM0bSt2gbNRnrOCsGXhyZbJi8mW1BoRjI8hbSriN?= =?us-ascii?Q?xfmeecRj/gOKoX+lYC+dhjbc8FwGOr9Wdkfq4gE4CzG0NtoTbpu92PmRyASz?= =?us-ascii?Q?VXWFEzF7wlfCMvNuAQD/fM81FKXCq6NzqX8mFjF/nV3Zf5NJDuG4+t8qZKH+?= =?us-ascii?Q?C02kmn0vbO3hPizB59yjZ43UiU6bHtZ5l5v16XCw0SuimmyvRW5AYwRcF7iC?= =?us-ascii?Q?x1tZcye8ICMckkYiA8LZU4cjN3prJOHkiEliWB2ypIIr1THP2IOn6DH2Bax8?= =?us-ascii?Q?USVpzpjVakuv/VinN8uXp50606k+sAB7qsymmB79Np3exP5aocuZuOdFw5V0?= =?us-ascii?Q?2DUB5V4470iy2CD275y9XIPW7cPHmo6vDfLAVHYZDC4XJc/uFvMKFB7DoQPb?= =?us-ascii?Q?9SSlhEBM6iABksIY1K6D8u2pbcrfxON7hhRSqswui31Y3cJeTGnUI12dOGkk?= =?us-ascii?Q?1ZccCbRLPkzh2kp55ar/guEUf9EcPo82uKRMgaRDFPzvXMiWzZj9Wdtbne4g?= =?us-ascii?Q?WCJT7WHaapb2ou0V6WTGETZIWEDEjYz0O5C5zPqbxcQ1kSVIClDW5/AXIiz7?= =?us-ascii?Q?//4BXZ0pe+2ZKp4EOobafG3SEjFlNYCeA6LRmkWxM+Qnp/ENQwff4QTYizQJ?= =?us-ascii?Q?LrA0a9jiuRYz8A1T9rI1KRJFS0dQeXEXhnkxdyg9gsKs6GFcm5bwQ1+GFG0w?= =?us-ascii?Q?wpBbjH2j8ILmyRl39PvvhAiSchxPrcHbdAEKdJsRcGMVopqRF/Ns8JtQu1R6?= =?us-ascii?Q?yymNGaadvxHwv+jOam8F7GAT+GD/f/HkEPMd5UYIphf953vh5ZlBtitc4zNq?= =?us-ascii?Q?lkr2c3hY3JInDCuC/VxtnpMF3yhynQkT8ZZV0Q2CF4OkSvU4zjEuFHu1GcEl?= =?us-ascii?Q?jpeaMClliUxI6oKRYl46Eq/Sm9Y8Enh2D1kxYI7lGpjez1EWtwwXbdVAKwE/?= =?us-ascii?Q?IfHYGj1cnlRTbhSJYRpAt+sd4kf+qpcHC2/UA00sf8SRqMEVyqNIC2kywggx?= =?us-ascii?Q?t05rdsEzBIZeGOBtOMgHsHWKi12Aervgmh5f+0HCfLmbhNVsC9Yw05HgMqdn?= =?us-ascii?Q?Sg/zZG6tYmCkbLAEUdm+D8UPsy9E64hWOpbwRY5TlgbUilSLzVO7bxV+fcWj?= =?us-ascii?Q?RxVQNZzw/2sBpB4HKNd91J20DNjk+4cdcUWn68FfsqVDxzrgOUKRDknLeF5/?= =?us-ascii?Q?ix7C7DEZJavfgsvptj1KICojWGPdC7lu9MOeLfmNsGejEI9MRa+iTc6VK8EW?= =?us-ascii?Q?rdnNYNmMl3k0K4RnxhZcCE8qV/f040UH0y+nycSxeuRxa957Ezec8ZhgD2Xo?= =?us-ascii?Q?BA=3D=3D?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e5fbbb0-2f61-4899-7a52-08da5e9d5089 X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2022 15:44:58.1972 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RnwjcRDxlx5539Re+Sth64DgMuqzk7eIGYx817B/nG37EKrbbgk0upXRUDOCq21BYybw5bftVGchovinX8uUlw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1322 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation topic for clocks and timers when running as a guest on Hyper-V. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/clocks.rst | 73 ++++++++++++++++++++++++++++++++= ++++ Documentation/virt/hyperv/index.rst | 1 + 2 files changed, 74 insertions(+) create mode 100644 Documentation/virt/hyperv/clocks.rst diff --git a/Documentation/virt/hyperv/clocks.rst b/Documentation/virt/hype= rv/clocks.rst new file mode 100644 index 0000000..e4ba2890 --- /dev/null +++ b/Documentation/virt/hyperv/clocks.rst @@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Clocks and Timers +----------------- + +arm64 +~~~~~ +On arm64, Hyper-V virtualizes the ARMv8 architectural system counter +and timer. Guest VMs use this virtualized hardware as the Linux +clocksource and clockevents via the standard arm_arch_timer.c +driver, just as they would on bare metal. Linux vDSO support for the +architectural system counter is functional in guest VMs on Hyper-V. +While Hyper-V also provides a synthetic system clock and four synthetic +per-CPU timers as described in the TLFS, they are not used by the +Linux kernel in a Hyper-V guest on arm64. However, older versions +of Hyper-V for arm64 only partially virtualize the ARMv8 +architectural timer, such that the timer does not generate +interrupts in the VM. Because of this limitation, running current +Linux kernel versions on these older Hyper-V versions requires an +out-of-tree patch to use the Hyper-V synthetic clocks/timers instead. + +x86/x64 +~~~~~~~ +On x86/x64, Hyper-V provides guest VMs with a synthetic system clock +and four synthetic per-CPU timers as described in the TLFS. Hyper-V +also provides access to the virtualized TSC via the RDTSC and +related instructions. These TSC instructions do not trap to +the hypervisor and so provide excellent performance in a VM. +Hyper-V performs TSC calibration, and provides the TSC frequency +to the guest VM via a synthetic MSR. Hyper-V initialization code +in Linux reads this MSR to get the frequency, so it skips TSC +calibration and sets tsc_reliable. Hyper-V provides virtualized +versions of the PIT (in Hyper-V Generation 1 VMs only), local +APIC timer, and RTC. Hyper-V does not provide a virtualized HPET in +guest VMs. + +The Hyper-V synthetic system clock can be read via a synthetic MSR, +but this access traps to the hypervisor. As a faster alternative, +the guest can configure a memory page to be shared between the guest +and the hypervisor. Hyper-V populates this memory page with a +64-bit scale value and offset value. To read the synthetic clock +value, the guest reads the TSC and then applies the scale and offset +as described in the Hyper-V TLFS. The resulting value advances +at a constant 10 MHz frequency. In the case of a live migration +to a host with a different TSC frequency, Hyper-V adjusts the +scale and offset values in the shared page so that the 10 MHz +frequency is maintained. + +Starting with Windows Server 2022 Hyper-V, Hyper-V uses hardware +support for TSC frequency scaling to enable live migration of VMs +across Hyper-V hosts where the TSC frequency may be different. +When a Linux guest detects that this Hyper-V functionality is +available, it prefers to use Linux's standard TSC-based clocksource. +Otherwise, it uses the clocksource for the Hyper-V synthetic system +clock implemented via the shared page (identified as +"hyperv_clocksource_tsc_page"). + +The Hyper-V synthetic system clock is available to user space via +vDSO, and gettimeofday() and related system calls can execute +entirely in user space. The vDSO is implemented by mapping the +shared page with scale and offset values into user space. User +space code performs the same algorithm of reading the TSC and +appying the scale and offset to get the constant 10 MHz clock. + +Linux clockevents are based on Hyper-V synthetic timer 0. While +Hyper-V offers 4 synthetic timers for each CPU, Linux only uses +timer 0. Interrupts from stimer0 are recorded on the "HVS" line in +/proc/interrupts. Clockevents based on the virtualized PIT and +local APIC timer also work, but the Hyper-V synthetic timer is +preferred. + +The driver for the Hyper-V synthetic system clock and timers is +drivers/clocksource/hyperv_timer.c. diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst index caa43ab..4a7a1b7 100644 --- a/Documentation/virt/hyperv/index.rst +++ b/Documentation/virt/hyperv/index.rst @@ -9,3 +9,4 @@ Hyper-V Enlightenments =20 overview vmbus + clocks --=20 1.8.3.1