From nobody Sat Apr 18 19:03:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F6AC433EF for ; Mon, 11 Jul 2022 17:49:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230519AbiGKRtA (ORCPT ); Mon, 11 Jul 2022 13:49:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230169AbiGKRsz (ORCPT ); Mon, 11 Jul 2022 13:48:55 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2090.outbound.protection.outlook.com [40.107.244.90]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04C1513D07; Mon, 11 Jul 2022 10:48:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k3XLXF75rLyMWFqw36X8ZZcQT87VCvTpvi+jKSiOQNWWIKyZC0QdurCqDkHg23ycesFBfCz2AfLrJvEoUl0KUfQDg/HhJ8s15wSuiZWA8ULxY29KbA46D7yeudX2PRSJmElVt8qwXGXaXsKZbVLLUOEsaI4mSAuQW0JVizR0gUkH6Ljfhqbdgl+iwa1XF1s+SKKRrH4D7eYzbsUJEH0Yj8uJYvQqrK1Fa3osxgr/wcIvipKdmsGGCO37dTT3YAl5jeMRSS2jqzf5uqLThxpsqioyZ7WrVaEyQbdDRGDtQNAYtxRhzZ8fNg/dKehH0UXdpJJaeMD4OVkmP2DfCA+R/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0estcobQMwuwBu15qB655d3pqGrdqiDGURVJH7AEyj4=; b=Jw1M4++yOjMXR5f69ZLLBPrBsagfEXr07L6LfFUZv8cv20q7dosdVgzvbJu/Rqk7gnhhUN0Rl/nLuValP+8oKXDkXHHn1peld81qhFqxyQzEq+bLpn/JZBiP6rD8wrKWzJn0SJH4Ka2xHO3JL+IC2tyynWm4n+ooYlTCakBmEJoV5+zsLfKkY+/6yMUr6jnCRc1va996gXv7uKDZAwcCJHbHhndmfjuF7Elh5apYszuzrSBLM8yGfIFoIYaWUiFGoY6ZS325EMUtocJ7KFnER3ptu3nRjVuaS7WQDcI5v4SBBdDuo4ZQwwlrmBtb5ctDOjpM9SsLK2xjoSfbn60Nvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0estcobQMwuwBu15qB655d3pqGrdqiDGURVJH7AEyj4=; b=RHGNHswTjM6I6RMSQsv18XftqNZO5PrK8kdR3BglHaHlNNlF5u8sGQFBP0fJ0d6h19ZFEOO4Z2/CrDRs39sBiRXycJObKMzjzuhFLNbntVt36JQNv0khgD+cTThzHndp5t3HfavNLVy8iq4DMVGo/G7jiADBq09NL72Iejhdteg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1321.namprd21.prod.outlook.com (2603:10b6:5:175::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.7; Mon, 11 Jul 2022 17:48:52 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Mon, 11 Jul 2022 17:48:52 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH v2 1/3] Documentation: hyperv: Add overview of Hyper-V enlightenments Date: Mon, 11 Jul 2022 10:48:22 -0700 Message-Id: <1657561704-12631-2-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> References: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MW2PR2101CA0013.namprd21.prod.outlook.com (2603:10b6:302:1::26) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5fc01d33-0179-4e1a-15cd-08da63659e19 X-MS-TrafficTypeDiagnostic: DM6PR21MB1321:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YJfyKAGkFrsmzd1SqJGauHdQtqCuzKELYHQ4jvHjNiLpIKV45NQx9CmK6IaYHnkjIl5HCAxApGr2Z/JDvs99Iv+LQVhBxeWrypvV9ehcxBC60Ui79mJHHl7W/GqqxtcsCpJmxO6WhJRNb5Uza63Mc1JtPGqFC0pRnfs9Ts2Jl0O9mxK7AxL9kLcUsHFueF9dDn2aMznC9yEGZrVEd+VTy9mOjhHEB2FlVNIUquiutSJfcZUNUhbmGg4yP7+CxKcySx02hHWg4blZzFCqk1+1LR5cKADfyuUEpyhA0aWlt4Mw8oixxJbfEsGZJuMp8BCexiHIbPhphfvlbirymh6SyKjhmGbZe6tWjSVjCVhXE7TvonL8NQUBx2+e/1yEk/PtuUrj+ZzuKDYNYd86tGh8gJbTQw0WKUrgdJErkKST4IGse3OEV61hm/Qqt0gzPeMycLdObSa5/ALaeEtOyO8Ou4lTp21//CQ4249cP8Z3/9QGycyhkgRPgxn83vfuTd1Xk2W/7btkAchXBDxgZZh3iF/hqA5NZnJGAxTRjbh8gaE14ghtBaUWrsn6LjGJ+gawBQaZgD9vdrgrKko+W1dyak5NI8xF+IngvVcag55iMREoyOYy1v6/FG2OOMRS1foaAjDM4qhuH5JU1T7upWe4H+LpJjPlAcyQShMSzVnnBIFA/n9yKtkvHji+m2iW6qoPDW3dKrZ9DBfQD4A2lkBAVkmUBuMqBj8bU4S7rJLYsvyIcKffZJHbUiylTdTf/7w0JRPB09YdqF4KkmEJ5jv/ve1sYtTyhDlOQPmS+mXoeSQB9AedmDcRlvPsSPeQ72yBesgkoiVe1j+tFEChIYo++GsKXJyo7QFSKG38jxHEHbols0y+f0s47vl2+v3q4pMzh+jehTTQXOJha6nLyRtiJg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(376002)(39860400002)(136003)(396003)(366004)(451199009)(26005)(82950400001)(8936002)(5660300002)(2906002)(36756003)(6506007)(82960400001)(66476007)(10290500003)(478600001)(86362001)(4326008)(66946007)(8676002)(6486002)(966005)(66556008)(30864003)(41300700001)(2616005)(38100700002)(83380400001)(107886003)(6512007)(186003)(38350700002)(6666004)(316002)(52116002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?mk1RIbuHmzF5vUQvGXWk7KXHupDH8MBG/dCGDjjDDJ5bW7DpmPy6dIROg1wz?= =?us-ascii?Q?ZE1F3L8jLEtZ+Ra3GhfymGIbSFz574+ngPwjBskqo5efQhZqeHhj8vRoeZHF?= =?us-ascii?Q?PsgIKVjBJBYxKcxug115JQsLCuFad2yGfRIfSKfQ7uHvvTrGQubN9/bngRNk?= =?us-ascii?Q?zG4ztrzMddv2i5PT0ZbqX1DHVlvZcjpDW01JoGxG8UPNj7mj71iqJVUF1Ks2?= =?us-ascii?Q?eq047+GcgaiSyHfwT9kM0fnWL33G/WsA3uVa81EY1jl0uw+2NCzscxoTIfAy?= =?us-ascii?Q?N+Nzcqd85Lbvn8ORPM5w12FAnKPFNPPcxqqgWw2q953XL1HbF7JS1ZMI3U3H?= =?us-ascii?Q?NZNJZ5YZUQTDyjUK6D2S3IWca5HfLQL3+RixnrS/tN0P47ErQll3Dkp7Vfi9?= =?us-ascii?Q?WysDoxeWk6bCw8a5h/6LWmqVQhuFeFKRFk+jpLUaEBVDY0tEM8u5+keqtDre?= =?us-ascii?Q?Y9zbtR7P8tFvV65iouCm++Jepzi2uVv9rGpA9GxoAuh/fPiZPYba56IHUFqu?= =?us-ascii?Q?vO6ffG8hxIj+MhCelU7hEcxKYAzMDFzWqUkYA94hm9TmdLbr1iUu6jMAI6ns?= =?us-ascii?Q?5du/6jz3Ba3mSUMt4v03sOSN1YuafL3cJU0Rhiy072ntmMmELDL0vrpx2BQ9?= =?us-ascii?Q?T3i3vl0KCDuEiJEke+WGHP8No54EDJm+d9vWG60DGnKZIxH61nbVRQO5y75P?= =?us-ascii?Q?g+MayRg2vSgSwXYlQFnZXjmHJNxcZyIY2sH5zpFK6aEnuQyOP7APm0V2KTD7?= =?us-ascii?Q?De32b4cr3kttwNg7HkTph/11v146Akq5uu/P9q69av88/hRXcF6T6rnemGRC?= =?us-ascii?Q?owIEosVEcp70bek4WfjW0mLJ6qaKmCJTT19o6XddT7rGopx3f7YsOfjNNHYP?= =?us-ascii?Q?kvh90QN/YAlT7E0u5q5wCiSp2ftFa5Zp4MCxhUCwUJj2vP2EezP/tG2g+7oc?= =?us-ascii?Q?wucQqlSj731Hupy4Um/iOPSjZ9eAK0ZL740/kmnahyOeSoA1CDmBJoiN97v4?= =?us-ascii?Q?MVvDzutjsFXb0Hxnysre3QBXaPj61e3BXZNgyb+qfxw76Q7acwR8BaM0AcXc?= =?us-ascii?Q?L09JrEsUjmMVzxnKq/3+dUN6VSg4U6OTJL2gn2YKgeEahCePnVIR04/SqHcJ?= =?us-ascii?Q?ycvIXTFkeOzPt5QX1I6Da1sjgnDY66kzglsd1hn8rPaXP6JmMRRz1/RMrQyf?= =?us-ascii?Q?RvVfbl1IKLV/eLfDf/rItJDGMEyYyrpKpZu2DYCBUQ9EAg1fmhJF8m4i4Tis?= =?us-ascii?Q?LvxyZgRzjnwLZs4Bp44i1qOzVxToJh9wZ+yRSkscmPPTTsdeCw5h2j6h+Fdl?= =?us-ascii?Q?P5KgR4HZ6lPiuwTcYJGF1Wb91tQ86/4o72bR2Z9TdFh70ilfRWP5RUSNRgmy?= =?us-ascii?Q?6C+nPvChSttBEpF4mgijqAbNgIGX2G66QYwTdb4rdBskqoO58RQxZ1ElAuoW?= =?us-ascii?Q?S9vvS0MrzIQz8L78aXcVoRhwqhOGS2VQg1Bc1ruuVRwo19drJHTYf3cenS9r?= =?us-ascii?Q?cI1jgnHGl9vhpCgbeshgGYyLQnrodTj/cFwLRTWyOtxpanz7vTMIawc0y3OB?= =?us-ascii?Q?bQhMQjpwNpD+CT7xaVSoihAXBQ1Zy9AGiGMcA0mf?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5fc01d33-0179-4e1a-15cd-08da63659e19 X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2022 17:48:52.2731 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LD71kjfOpfabHcvU7UuNLDkUNSqGhLfbjvysZsrmZygAZ8qvvAuMOGW19kRbr9d/SsZecpV3LQd392k7bluGkA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1321 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an initial documentation topic for Linux enlightenments to run as a guest on Microsoft's Hyper-V hypervisor, linked under the "virt" documentation area. Update the virt doc index.rst and the MAINTAINERS file. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/index.rst | 10 ++ Documentation/virt/hyperv/overview.rst | 207 +++++++++++++++++++++++++++++= ++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 1 + 4 files changed, 219 insertions(+) create mode 100644 Documentation/virt/hyperv/index.rst create mode 100644 Documentation/virt/hyperv/overview.rst diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst new file mode 100644 index 0000000..991bee4 --- /dev/null +++ b/Documentation/virt/hyperv/index.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Hyper-V Enlightenments +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. toctree:: + :maxdepth: 1 + + overview diff --git a/Documentation/virt/hyperv/overview.rst b/Documentation/virt/hy= perv/overview.rst new file mode 100644 index 0000000..cd49333 --- /dev/null +++ b/Documentation/virt/hyperv/overview.rst @@ -0,0 +1,207 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D +The Linux kernel contains a variety of code for running as a fully +enlightened guest on Microsoft's Hyper-V hypervisor. Hyper-V +consists primarily of a bare-metal hypervisor plus a virtual machine +management service running in the parent partition (roughly +equivalent to KVM and QEMU, for example). Guest VMs run in child +partitions. In this documentation, references to Hyper-V usually +encompass both the hypervisor and the VMM service without making a +distinction about which functionality is provided by which +component. + +Hyper-V runs on x86/x64 and arm64 architectures, and Linux guests +are supported on both. The functionality and behavior of Hyper-V is +generally the same on both architectures unless noted otherwise. + +Linux Guest Communication with Hyper-V +-------------------------------------- +Linux guests communicate with Hyper-V in four different ways: + +* Implicit traps: As defined by the x86/x64 or arm64 architecture, + some guest actions trap to Hyper-V. Hyper-V emulates the action and + returns control to the guest. This behavior is generally invisible + to the Linux kernel. + +* Explicit hypercalls: Linux makes an explicit function call to + Hyper-V, passing parameters. Hyper-V performs the requested action + and returns control to the caller. Parameters are passed in + processor registers or in memory shared between the Linux guest and + Hyper-V. On x86/x64, hypercalls use a Hyper-V specific calling + sequence. On arm64, hypercalls use the ARM standard SMCCC calling + sequence. + +* Synthetic register access: Hyper-V implements a variety of + synthetic registers. On x86/x64 these registers appear as MSRs in + the guest, and the Linux kernel can read or write these MSRs using + the normal mechanisms defined by the x86/x64 architecture. On + arm64, these synthetic registers must be accessed using explicit + hypercalls. + +* VMbus: VMbus is a higher-level software construct that is built on + the other 3 mechanisms. It is a message passing interface between + the Hyper-V host and the Linux guest. It uses memory that is shared + between Hyper-V and the guest, along with various signaling + mechanisms. + +The first three communication mechanisms are documented in the +`Hyper-V Top Level Functional Spec (TLFS)`_. The TLFS describes +general Hyper-V functionality and provides details on the hypercalls +and synthetic registers. The TLFS is currently written for the +x86/x64 architecture only. + +.. _Hyper-V Top Level Functional Spec (TLFS): https://docs.microsoft.com/e= n-us/virtualization/hyper-v-on-windows/tlfs/tlfs + +VMbus is not documented. This documentation provides a high-level +overview of VMbus and how it works, but the details can be discerned +only from the code. + +Sharing Memory +-------------- +Many aspects are communication between Hyper-V and Linux are based +on sharing memory. Such sharing is generally accomplished as +follows: + +* Linux allocates memory from its physical address space using + standard Linux mechanisms. + +* Linux tells Hyper-V the guest physical address (GPA) of the + allocated memory. Many shared areas are kept to 1 page so that a + single GPA is sufficient. Larger shared areas require a list of + GPAs, which usually do not need to be contiguous in the guest + physical address space. How Hyper-V is told about the GPA or list + of GPAs varies. In some cases, a single GPA is written to a + synthetic register. In other cases, a GPA or list of GPAs is sent + in a VMbus message. + +* Hyper-V translates the GPAs into "real" physical memory addresses, + and creates a virtual mapping that it can use to access the memory. + +* Linux can later revoke sharing it has previously established by + telling Hyper-V to set the shared GPA to zero. + +Hyper-V operates with a page size of 4 Kbytes. GPAs communicated to +Hyper-V may be in the form of page numbers, and always describe a +range of 4 Kbytes. Since the Linux guest page size on x86/x64 is +also 4 Kbytes, the mapping from guest page to Hyper-V page is 1-to-1. +On arm64, Hyper-V supports guests with 4/16/64 Kbyte pages as +defined by the arm64 architecture. If Linux is using 16 or 64 +Kbyte pages, Linux code must be careful to communicate with Hyper-V +only in terms of 4 Kbyte pages. HV_HYP_PAGE_SIZE and related macros +are used in code that communicates with Hyper-V so that it works +correctly in all configurations. + +As described in the TLFS, a few memory pages shared between Hyper-V +and the Linux guest are "overlay" pages. With overlay pages, Linux +uses the usual approach of allocating guest memory and telling +Hyper-V the GPA of the allocated memory. But Hyper-V then replaces +that physical memory page with a page it has allocated, and the +original physical memory page is no longer accessible in the guest +VM. Linux may access the memory normally as if it were the memory +that it originally allocated. The "overlay" behavior is visible +only because the contents of the page (as seen by Linux) change at +the time that Linux originally establishes the sharing and the +overlay page is inserted. Similarly, the contents change if Linux +revokes the sharing, in which case Hyper-V removes the overlay page, +and the guest page originally allocated by Linux becomes visible +again. + +Before Linux does a kexec to a kdump kernel or any other kernel, +memory shared with Hyper-V should be revoked. Hyper-V could modify +a shared page or remove an overlay page after the new kernel is +using the page for a different purpose, corrupting the new kernel. +Hyper-V does not provide a single "set everything" operation to +guest VMs, so Linux code must individually revoke all sharing before +doing kexec. See hv_kexec_handler() and hv_crash_handler(). But +the crash/panic path still has holes in cleanup because some shared +pages are set using per-CPU synthetic registers and there's no +mechanism to revoke the shared pages for CPUs other than the CPU +running the panic path. + +CPU Management +-------------- +Hyper-V does not have a ability to hot-add or hot-remove a CPU +from a running VM. However, Windows Server 2019 Hyper-V and +earlier versions may provide guests with ACPI tables that indicate +more CPUs than are actually present in the VM. As is normal, Linux +treats these additional CPUs as potential hot-add CPUs, and reports +them as such even though Hyper-V will never actually hot-add them. +Starting in Windows Server 2022 Hyper-V, the ACPI tables reflect +only the CPUs actually present in the VM, so Linux does not report +any hot-add CPUs. + +A Linux guest CPU may be taken offline using the normal Linux +mechanisms, provided no VMbus channel interrupts are assigned to +the CPU. See the section on VMbus Interrupts for more details +on how VMbus channel interrupts can be re-assigned to permit +taking a CPU offline. + +32-bit and 64-bit +----------------- +On x86/x64, Hyper-V supports 32-bit and 64-bit guests, and Linux +will build and run in either version. While the 32-bit version is +expected to work, it is used rarely and may suffer from undetected +regressions. + +On arm64, Hyper-V supports only 64-bit guests. + +Endian-ness +----------- +All communication between Hyper-V and guest VMs uses Little-Endian +format on both x86/x64 and arm64. Big-endian format on arm64 is not +supported by Hyper-V, and Linux code does not use endian-ness macros +when accessing data shared with Hyper-V. + +Versioning +---------- +Current Linux kernels operate correctly with older versions of +Hyper-V back to Windows Server 2012 Hyper-V. Support for running +on the original Hyper-V release in Windows Server 2008/2008 R2 +has been removed. + +A Linux guest on Hyper-V outputs in dmesg the version of Hyper-V +it is running on. This version is in the form of a Windows build +number and is for display purposes only. Linux code does not +test this version number at runtime to determine available features +and functionality. Hyper-V indicates feature/function availability +via flags in synthetic MSRs that Hyper-V provides to the guest, +and the guest code tests these flags. + +VMbus has its own protocol version that is negotiated during the +initial VMbus connection from the guest to Hyper-V. This version +number is also output to dmesg during boot. This version number +is checked in a few places in the code to determine if specific +functionality is present. + +Furthermore, each synthetic device on VMbus also has a protocol +version that is separate from the VMbus protocol version. Device +drivers for these synthetic devices typically negotiate the device +protocol version, and may test that protocol version to determine +if specific device functionality is present. + +Code Packaging +-------------- +Hyper-V related code appears in the Linux kernel code tree in three +main areas: + +1. drivers/hv + +2. arch/x86/hyperv and arch/arm64/hyperv + +3. individual device driver areas such as drivers/scsi, drivers/net, + drivers/clocksource, etc. + +A few miscellaneous files appear elsewhere. See the full list under +"Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIVER FOR HYPERV +SYNTHETIC VIDEO DEVICE" in the MAINTAINERS file. + +The code in #1 and #2 is built only when CONFIG_HYPERV is set. +Similarly, the code for most Hyper-V related drivers is built only +when CONFIG_HYPERV is set. + +Most Hyper-V related code in #1 and #3 can be built as a module. +The architecture specific code in #2 must be built-in. Also, +drivers/hv/hv_common.c is low-level code that is common across +architectures and must be built-in. diff --git a/Documentation/virt/index.rst b/Documentation/virt/index.rst index 492f092..2f1cffa 100644 --- a/Documentation/virt/index.rst +++ b/Documentation/virt/index.rst @@ -14,6 +14,7 @@ Linux Virtualization Support ne_overview acrn/index coco/sev-guest + hyperv/index =20 .. only:: html and subproject =20 diff --git a/MAINTAINERS b/MAINTAINERS index a6d3bd9..fd2e5ce 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9172,6 +9172,7 @@ S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git F: Documentation/ABI/stable/sysfs-bus-vmbus F: Documentation/ABI/testing/debugfs-hyperv +F: Documentation/virt/hyperv F: Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst F: arch/arm64/hyperv F: arch/arm64/include/asm/hyperv-tlfs.h --=20 1.8.3.1 From nobody Sat Apr 18 19:03:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E602C433EF for ; Mon, 11 Jul 2022 17:49:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229887AbiGKRtE (ORCPT ); Mon, 11 Jul 2022 13:49:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230184AbiGKRs4 (ORCPT ); Mon, 11 Jul 2022 13:48:56 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2090.outbound.protection.outlook.com [40.107.244.90]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C58713F5E; Mon, 11 Jul 2022 10:48:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a1pqpRu2ViyyOZrJGb/buEAkTHSkS46iL2UAiV3ZDqR+IqxtSmZA6mEsvq683QJAsq99sCmpVTtepxvm/ZE0Yq24O3oHbKBattU6j2hGCpHh8CbY8iPixGTSl7O6AsE+c9TxyaPlhnpT8Tav7KhKr7SaF3huvO0Ni1Dei/Ub/gtiqsheWwB07h99X++jYes1PMxbm+TXir7G/2G+XQO2MZoMZMLhiEdp5KQ92ByW8pS9IX0IDZjtLIQR9yddGcCWFc/lm0hcuLkhos9TBSi3am0XNGbw+VIxTFYdPZbpfZCXnlCDBACzanJq+u5xIbwf4iHu0ghLFNT5I3Zpr99u0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=USUqWX22yjhg2T5jVcHObqc8/i1/up2CIJq40BN7Bf0=; b=KomQzSVu7mcaPxlRV6JcdnXOmZp1cmTsdhxsHofaXgYjwvUK2RiWVgzmm8g+EFV+xujZoVFc+CTABAuSN8gbYa9cufXf/9vo5udlmsWjbaB4Irw6JdsmzseMpTJ3YptKXroX96GZypX8gZ50niuOyo5D48fAIig9xSsSpdJdBGZAzYVvXfncyZU/hB3Y0yfi5ZoKoyTrSImDFnXxqAOptOb9ZGMK0FLfV0MHXSkUcq7rFYEEMRt2H1XhjxFQJ7BppMnjyIFxr7IUub+xWoJvGOWH5lc5BNNKGUinW5FOmkgPBk9OvExD8YWODgP8mvcXfbm1qnOhxvmYHAavGjKQfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=USUqWX22yjhg2T5jVcHObqc8/i1/up2CIJq40BN7Bf0=; b=BrxVNVtPEzdiv/BX44Edc6cREW9Bo2fWufmBuF5FYjWel/XEEHXnby8G+k7+cw4vBBj+Rgs80zoIg8H5uVoeVQFgQAP+czlHw8sh3sPVj44g57cIDzhZm1rwJgZidBzhz+xynDNgghaxUFamwfO2OjHHlrTy5Zkhgx5cDiIvVoQ= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1321.namprd21.prod.outlook.com (2603:10b6:5:175::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.7; Mon, 11 Jul 2022 17:48:53 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Mon, 11 Jul 2022 17:48:53 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH v2 2/3] Documentation: hyperv: Add overview of VMbus Date: Mon, 11 Jul 2022 10:48:23 -0700 Message-Id: <1657561704-12631-3-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> References: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MW2PR2101CA0013.namprd21.prod.outlook.com (2603:10b6:302:1::26) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 973a7343-07a1-4415-8514-08da63659e8b X-MS-TrafficTypeDiagnostic: DM6PR21MB1321:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3/gk9G0Hhx+MVyQp5AMVXdDY0izccUwx/2r6WbrF81knoSjM83rHJMVJid+/LySP9mIt+f+3407C2+3NjQKRg7wpPNVv1qSUqUw3ZQgRIlHewvh6EpGQ5W1pCpyDweVtm9xYQ2yVaJZSh4NvLXZYkj63MkQfYzAxmLVA/0AlTmRQXQBQGEfXxn/TNUw8jTgdG/drUeN6oBKbtC7k9K55pNQ3BtyPArKOoSLDER1m5pJK/n1gZ2iMZj2WBFHa0MjRaF4rEXz5MXDGTHFRxOOfp1c6lFgpOAVTLNjM1go3ALDNMZH4+W9N4DUmG5N8Ux6zedDtjCiCWhzHGw4d8r2SEpSWUJ3luwKySjxnNAqduhRTA+FKiL4ikJ+npKKi1Nf1sUrSBbcfkIKwRhe0lK2L1PCQ1Kcxr17yUnoLtYGlcMPJHkhO43fUaIR2yRI5eZ1+pfxyTEOJaUT9kuFUmBAW3ZelftlZTXqzcNzYBrf6xbccg9dPtm4Ca2rK4WSQWad3PSxKd6cs2wC4Ws3sC8jN+N1c2aKtlx40zvRg+PaRiQyOy+takQ7ExGwjJO7ZjvABBwAjpWyd0EDDWgLcJfG3laPIz2bzdaYCiZcph9lLZb8ruK9/lpRXXiqLZQr4Y60cGje4w3FCkioRSY8eQJ6K1ZEBqEaz3wWHgVvUSiQQk35z/756jNLeCd6LCD0RnR94GWXN5W0oRuyf8l5BAJGj5CK3YURiGkHix7o/spxWpADOnOtdEq5RdnYR3p889NhC2miycmvwmYsxuPihaoNLmtIIHUTF1s/d8Yz2sMQshobjMUSFWHlMb5rCBAQDbhJyruRCv0sh+Aq5ZLINbdeQrNwNHT9UEIG3FvTA+AY9vCo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(376002)(39860400002)(136003)(396003)(366004)(451199009)(26005)(82950400001)(8936002)(5660300002)(2906002)(36756003)(6506007)(82960400001)(66476007)(18074004)(10290500003)(478600001)(86362001)(4326008)(66946007)(8676002)(6486002)(66556008)(30864003)(41300700001)(2616005)(38100700002)(83380400001)(107886003)(6512007)(186003)(38350700002)(6666004)(316002)(52116002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3FyWIZj+NbJ9rZ/cM3B5TErmX0nC3RALmrq9XYyKGL8lJxpW636tDeH2j3TF?= =?us-ascii?Q?m0AvOqsOtenElo1ma8wGoB/9GLSDPDuG1WA9tQl6yXmWubyQ1waBw3IowESy?= =?us-ascii?Q?OmtrBRISOH7Fk0cFZJx1H4f6pLGoGkrZF3T+om3jhhxsstvq/jPJeYjHfo6e?= =?us-ascii?Q?Re2ulCTW/9YKLagLREOa54W0E3yKBBRGDss/QB74EYvuQzX1O/IUkzbqXIid?= =?us-ascii?Q?qVvo/o+5U1nUprUGVaMWQz2qjhDmud1TzEkAxt8LgthEA+YlM7J67r9mXury?= =?us-ascii?Q?0SweAoGM8FT/3Ggn8DsY4aMNmSbyuVyidrmMwd218LWc1TOUAhIkxOpbc1v6?= =?us-ascii?Q?Bk9zxbQ1p+Y/Q3vivj7ab18xTqXi3V3mguk8Hb+xQyUkhs0oHil55ovTPlAC?= =?us-ascii?Q?OFhvVleAtmIjVGowd6nKiSG84NS6foL1dNHXazZrIZQ4hxpDx8NRSo84B5y3?= =?us-ascii?Q?3H+gV1SOMfArBieyPShM/3JkzropUMZWOQvztBsP33fhqn3cm9kzXKmnQM6j?= =?us-ascii?Q?I9tVjpfIBShA3DofJR8FxBJlP32J9F5b2SF3V4qIoUnW7uMvS9tocnUpoRbB?= =?us-ascii?Q?VBER0YV5Fk16iT1YrkDZMvCP18rB6137lIW5fQNQ4BUI/AtKpL4sTD27fc55?= =?us-ascii?Q?Wf8QGuA7WUh3fD81WNzfn8cxvURoHaxMUB4alzNf7MS+i3gRZCBIl9vXsgY7?= =?us-ascii?Q?Iy0kH1tyojKOOHvw3h6KvY7UrzCjUSdqUDAiW7xMldrd7hD9eAwDv6N+ZRdq?= =?us-ascii?Q?U52SwAxo2I+sOuyLRwKo0OmMW56x/2rmQQsBC6wB2WVILKZV4gVYocAjVWov?= =?us-ascii?Q?HnqpuNIg3praJtMXmHusNuEgvwO1MLWrEvtoYc9kgjGq8VdTVEQSXxZRSNMK?= =?us-ascii?Q?eh0vlaA7PTz9PRtE7AaUznIypZCuQwsp/OZx1vQe6KsXuPG+/lTwMeau42/l?= =?us-ascii?Q?GIj4/IJLDSLqOib0T8gCFkZLx/Xy/VJHh/glC8q28xMqS/c0XXRWsnyNc9WW?= =?us-ascii?Q?PNfV32yMzEJ0JXQ1rxbuIgslCaNi6cD9Okoz86FPBjY472KuqPhWu3PR6M8Y?= =?us-ascii?Q?7zKOdTqvIStm4MIGOsO4hIewg9TgPyZdPRojGxHyx6bJjtBr55XuM3gST6ti?= =?us-ascii?Q?aPomibC3M2mgvzk4gPVv8BiuRxKTsaU7t3sjleYqQQbcQDHfN7onrQSrjWTX?= =?us-ascii?Q?OIWj93nIHTu40fzJ+znL3VRSaCZJlpv+t2seyZ7lKWRVWzjyEqgGbOGo3Ovw?= =?us-ascii?Q?6f5BIQYyiQjP04IxPVS31UPKFiaLth3pfpJexW7e0kCsttAKnqXwi96XCF16?= =?us-ascii?Q?liAAlWUzOJG0PRrL6TbqG4ptDR3izAasDw9RjhwqU1ST6etfy5lD0KFMVWpV?= =?us-ascii?Q?wsNuAxNl8daXkJA/Jo9WslNfRTowiGyIJ0JgMq4E1rsW+hhAaT7TV4sCnetv?= =?us-ascii?Q?w32z+MJuMkY9rYWzNUxDwJ8a6uz8i9I0ufbphG7Gxm83nEyQeUrKUGjBxg1/?= =?us-ascii?Q?Na90+80G012NBbzWhRammz9Kw9wxi2Dr/cq+22TFDjZ7MWOHSSx17dFrp4n/?= =?us-ascii?Q?BbwodfINKL29nTRZIrZch9VmsHUx/2OiTIGddCNs?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 973a7343-07a1-4415-8514-08da63659e8b X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2022 17:48:53.0543 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5QSToKTh/nSga6szNicLlGAdnSb2601VPLFOEiGGMKvgo1gWM7bcO18LAGLyhfGJCrrM8yLekd3tLmjJSnCyyA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1321 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation topic for using VMbus when running as a guest on Hyper-V. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/index.rst | 1 + Documentation/virt/hyperv/vmbus.rst | 303 ++++++++++++++++++++++++++++++++= ++++ 2 files changed, 304 insertions(+) create mode 100644 Documentation/virt/hyperv/vmbus.rst diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst index 991bee4..caa43ab 100644 --- a/Documentation/virt/hyperv/index.rst +++ b/Documentation/virt/hyperv/index.rst @@ -8,3 +8,4 @@ Hyper-V Enlightenments :maxdepth: 1 =20 overview + vmbus diff --git a/Documentation/virt/hyperv/vmbus.rst b/Documentation/virt/hyper= v/vmbus.rst new file mode 100644 index 0000000..a9de881 --- /dev/null +++ b/Documentation/virt/hyperv/vmbus.rst @@ -0,0 +1,303 @@ +.. SPDX-License-Identifier: GPL-2.0 + +VMbus +=3D=3D=3D=3D=3D +VMbus is a software construct provided by Hyper-V to guest VMs. It +consists of a control path and common facilities used by synthetic +devices that Hyper-V presents to guest VMs. The control path is +used to offer synthetic devices to the guest VM and, in some cases, +to rescind those devices. The common facilities include software +channels for communicating between the device driver in the guest VM +and the synthetic device implementation that is part of Hyper-V, and +signaling primitives to allow Hyper-V and the guest to interrupt +each other. + +VMbus is modeled in Linux as a bus, with the expected /sys/bus/vmbus +entry in a running Linux guest. The VMbus driver (drivers/hv/vmbus_drv.c) +establishes the VMbus control path with the Hyper-V host, then +registers itself as a Linux bus driver. It implements the standard +bus functions for adding and removing devices to/from the bus. + +Most synthetic devices offered by Hyper-V have a corresponding Linux +device driver. These devices include: + +* SCSI controller +* NIC +* Graphics frame buffer +* Keyboard +* Mouse +* PCI device pass-thru +* Heartbeat +* Time Sync +* Shutdown +* Memory balloon +* Key/Value Pair (KVP) exchange with Hyper-V +* Hyper-V online backup (a.k.a. VSS) + +Guest VMs may have multiple instances of the synthetic SCSI +controller, synthetic NIC, and PCI pass-thru devices. Other +synthetic devices are limited to a single instance per VM. Not +listed above are a small number of synthetic devices offered by +Hyper-V that are used only by Windows guests and for which Linux +does not have a driver. + +Hyper-V uses the terms "VSP" and "VSC" in describing synthetic +devices. "VSP" refers to the Hyper-V code that implements a +particular synthetic device, while "VSC" refers to the driver for +the device in the guest VM. For example, the Linux driver for the +synthetic NIC is referred to as "netvsc" and the Linux driver for +the synthetic SCSI controller is "storvsc". These drivers contain +functions with names like "storvsc_connect_to_vsp". + +VMbus channels +-------------- +An instance of a synthetic device uses VMbus channels to communicate +between the VSP and the VSC. Channels are bi-directional and used +for passing messages. Most synthetic devices use a single channel, +but the synthetic SCSI controller and synthetic NIC may use multiple +channels to achieve higher performance and greater parallelism. + +Each channel consists of two ring buffers. These are classic ring +buffers from a university data structures textbook. If the read +and writes pointers are equal, the ring buffer is considered to be +empty, so a full ring buffer always has at least one byte unused. +The "in" ring buffer is for messages from the Hyper-V host to the +guest, and the "out" ring buffer is for messages from the guest to +the Hyper-V host. In Linux, the "in" and "out" designations are as +viewed by the guest side. The ring buffers are memory that is +shared between the guest and the host, and they follow the standard +paradigm where the memory is allocated by the guest, with the list +of GPAs that make up the ring buffer communicated to the host. Each +ring buffer consists of a header page (4 Kbytes) with the read and +write indices and some control flags, followed by the memory for the +actual ring. The size of the ring is determined by the VSC in the +guest and is specific to each synthetic device. The list of GPAs +making up the ring is communicated to the Hyper-V host over the +VMbus control path as a GPA Descriptor List (GPADL). See function +vmbus_establish_gpadl(). + +Each ring buffer is mapped into contiguous Linux kernel virtual +space in three parts: 1) the 4 Kbyte header page, 2) the memory +that makes up the ring itself, and 3) a second mapping of the memory +that makes up the ring itself. Because (2) and (3) are contiguous +in kernel virtual space, the code that copies data to and from the +ring buffer need not be concerned with ring buffer wrap-around. +Once a copy operation has completed, the read or write index may +need to be reset to point back into the first mapping, but the +actual data copy does not need to be broken into two parts. This +approach also allows complex data structures to be easily accessed +directly in the ring without handling wrap-around. + +On arm64 with page sizes > 4 Kbytes, the header page must still be +passed to Hyper-V as a 4 Kbyte area. But the memory for the actual +ring must be aligned to PAGE_SIZE and have a size that is a multiple +of PAGE_SIZE so that the duplicate mapping trick can be done. Hence +a portion of the header page is unused and not communicated to +Hyper-V. This case is handled by vmbus_establish_gpadl(). + +Hyper-V enforces a limit on the aggregate amount of guest memory +that can be shared with the host via GPADLs. This limit ensures +that a rogue guest can't force the consumption of excessive host +resources. For Windows Server 2019 and later, this limit is +approximately 1280 Mbytes. For versions prior to Windows Server +2019, the limit is approximately 384 Mbytes. + +VMbus messages +-------------- +All VMbus messages have a standard header that includes the message +length, the offset of the message payload, some flags, and a +transactionID. The portion of the message after the header is +unique to each VSP/VSC pair. + +Messages follow one of two patterns: + +* Unidirectional: Either side sends a message and does not + expect a response message +* Request/response: One side (usually the guest) sends a message + and expects a response + +The transactionID (a.k.a. "requestID") is for matching requests & +responses. Some synthetic devices allow multiple requests to be in- +flight simultaneously, so the guest specifies a transactionID when +sending a request. Hyper-V sends back the same transactionID in the +matching response. + +Messages passed between the VSP and VSC are control messages. For +example, a message sent from the storvsc driver might be "execute +this SCSI command". If a message also implies some data transfer +between the guest and the Hyper-V host, the actual data to be +transferred may be embedded with the control message, or it may be +specified as a separate data buffer that the Hyper-V host will +access as a DMA operation. The former case is used when the size of +the data is small and the cost of copying the data to and from the +ring buffer is minimal. For example, time sync messages from the +Hyper-V host to the guest contain the actual time value. When the +data is larger, a separate data buffer is used. In this case, the +control message contains a list of GPAs that describe the data +buffer. For example, the storvsc driver uses this approach to +specify the data buffers to/from which disk I/O is done. + +Three functions exist to send VMbus messages: + +1. vmbus_sendpacket(): Control-only messages and messages with + embedded data -- no GPAs +2. vmbus_sendpacket_pagebuffer(): Message with list of GPAs + identifying data to transfer. An offset and length is + associated with each GPA so that multiple discontinuous areas + of guest memory can be targeted. +3. vmbus_sendpacket_mpb_desc(): Message with list of GPAs + identifying data to transfer. A single offset and length is + associated with a list of GPAs. The GPAs must describe a + single logical area of guest memory to be targeted. + +Historically, Linux guests have trusted Hyper-V to send well-formed +and valid messages, and Linux drivers for synthetic devices did not +fully validate messages. With the introduction of processor +technologies that fully encrypt guest memory and that allow the +guest to not trust the hypervisor (AMD SNP-SEV, Intel TDX), trusting +the Hyper-V host is no longer a valid assumption. The drivers for +VMbus synthetic devices are being updated to fully validate any +values read from memory that is shared with Hyper-V, which includes +messages from VMbus devices. To facilitate such validation, +messages read by the guest from the "in" ring buffer are copied to a +temporary buffer that is not shared with Hyper-V. Validation is +performed in this temporary buffer without the risk of Hyper-V +maliciously modifying the message after it is validated but before +it is used. + +VMbus interrupts +---------------- +VMbus provides a mechanism for the guest to interrupt the host when +the guest has queued new messages in a ring buffer. The host +expects that the guest will send an interrupt only when an "out" +ring buffer transitions from empty to non-empty. If the guest sends +interrupts at other times, the host deems such interrupts to be +unnecessary. If a guest sends an excessive number of unnecessary +interrupts, the host may throttle that guest by suspending its +execution for a few seconds to prevent a denial-of-service attack. + +Similarly, the host will interrupt the guest when it sends a new +message on the VMbus control path, or when a VMbus channel "in" ring +buffer transitions from empty to non-empty. Each CPU in the guest +may receive VMbus interrupts, so they are best modeled as per-CPU +interrupts in Linux. This model works well on arm64 where a single +per-CPU IRQ is allocated for VMbus. Since x86/x64 lacks support for +per-CPU IRQs, an x86 interrupt vector is statically allocated (see +HYPERVISOR_CALLBACK_VECTOR) across all CPUs and explicitly coded to +call the VMbus interrupt service routine. These interrupts are +visible in /proc/interrupts on the "HYP" line. + +The guest CPU that a VMbus channel will interrupt is selected by the +guest when the channel is created, and the host is informed of that +selection. VMbus devices are broadly grouped into two categories: + +1. "Slow" devices that need only one VMbus channel. The devices + (such as keyboard, mouse, heartbeat, and timesync) generate + relatively few interrupts. Their VMbus channels are all + assigned to interrupt the VMBUS_CONNECT_CPU, which is always + CPU 0. + +2. "High speed" devices that may use multiple VMbus channels for + higher parallelism and performance. These devices include the + synthetic SCSI controller and synthetic NIC. Their VMbus + channels interrupts are assigned to CPUs that are spread out + among the available CPUs in the VM so that interrupts on + multiple channels can be processed in parallel. + +The assignment of VMbus channel interrupts to CPUs is done in the +function init_vp_index(). This assignment is done outside of the +normal Linux interrupt affinity mechanism, so the interrupts are +neither "unmanaged" nor "managed" interrupts. + +The CPU that a VMbus channel will interrupt can be seen in +/sys/bus/vmbus/devices// channels//cpu. +When running on later versions of Hyper-V, the CPU can be changed +by writing a new value to this sysfs entry. Because the interrupt +assignment is done outside of the normal Linux affinity mechanism, +there are no entries in /proc/irq corresponding to individual +VMbus channel interrupts. + +An online CPU in a Linux guest may not be taken offline if it has +VMbus channel interrupts assigned to it. Any such channel +interrupts must first be manually reassigned to another CPU as +described above. When no channel interrupts are assigned to the +CPU, it can be taken offline. + +When a guest CPU receives a VMbus interrupt from the host, the +function vmbus_isr() handles the interrupt. It first checks for +channel interrupts by calling vmbus_chan_sched(), which looks at a +bitmap setup by the host to determine which channels have pending +interrupts on this CPU. If multiple channels have pending +interrupts for this CPU, they are processed sequentially. When all +channel interrupts have been processed, vmbus_isr() checks for and +processes any message received on the VMbus control path. + +The VMbus channel interrupt handling code is designed to work +correctly even if an interrupt is received on a CPU other than the +CPU assigned to the channel. Specifically, the code does not use +CPU-based exclusion for correctness. In normal operation, Hyper-V +will interrupt the assigned CPU. But when the CPU assigned to a +channel is being changed via sysfs, the guest doesn't know exactly +when Hyper-V will make the transition. The code must work correctly +even if there is a time lag before Hyper-V starts interrupting the +new CPU. See comments in target_cpu_store(). + +VMbus device creation/deletion +------------------------------ +Hyper-V and the Linux guest have a separate message-passing path +that is used for synthetic device creation and deletion. This +path does not use a VMbus channel. See vmbus_post_msg() and +vmbus_on_msg_dpc(). + +The first step is for the guest to connect to the generic +Hyper-V VMbus mechanism. As part of establishing this connection, +the guest and Hyper-V agree on a VMbus protocol version they will +use. This negotiation allows newer Linux kernels to run on older +Hyper-V versions, and vice versa. + +The guest then tells Hyper-V to "send offers". Hyper-V sends an +offer message to the guest for each synthetic device that the VM +is configured to have. Each VMbus device type has a fixed GUID +known as the "class ID", and each VMbus device instance is also +identified by a GUID. The offer message from Hyper-V contains +both GUIDs to uniquely (within the VM) identify the device. +There is one offer message for each device instance, so a VM with +two synthetic NICs will get two offers messages with the NIC +class ID. The ordering of offer messages can vary from boot-to-boot +and must not be assumed to be consistent in Linux code. Offer +messages may also arrive long after Linux has initially booted +because Hyper-V supports adding devices, such as synthetic NICs, +to running VMs. A new offer message is processed by +vmbus_process_offer(), which indirectly invokes vmbus_add_channel_work(). + +Upon receipt of an offer message, the guest identifies the device +type based on the class ID, and invokes the correct driver to set up +the device. Driver/device matching is performed using the standard +Linux mechanism. + +The device driver probe function opens the primary VMbus channel to +the corresponding VSP. It allocates guest memory for the channel +ring buffers and shares the ring buffer with the Hyper-V host by +giving the host a list of GPAs for the ring buffer memory. See +vmbus_establish_gpadl(). + +Once the ring buffer is set up, the device driver and VSP exchange +setup messages via the primary channel. These messages may include +negotiating the device protocol version to be used between the Linux +VSC and the VSP on the Hyper-V host. The setup messages may also +include creating additional VMbus channels, which are somewhat +mis-named as "sub-channels" since they are functionally +equivalent to the primary channel once they are created. + +Finally, the device driver may create entries in /dev as with +any device driver. + +The Hyper-V host can send a "rescind" message to the guest to +remove a device that was previously offered. Linux drivers must +handle such a rescind message at any time. Rescinding a device +invokes the device driver "remove" function to cleanly shut +down the device and remove it. Once a synthetic device is +rescinded, neither Hyper-V nor Linux retains any state about +its previous existence. Such a device might be re-added later, +in which case it is treated as an entirely new device. See +vmbus_onoffer_rescind(). --=20 1.8.3.1 From nobody Sat Apr 18 19:03:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C2CEC433EF for ; Mon, 11 Jul 2022 17:49:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231389AbiGKRtI (ORCPT ); Mon, 11 Jul 2022 13:49:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230327AbiGKRs5 (ORCPT ); Mon, 11 Jul 2022 13:48:57 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2090.outbound.protection.outlook.com [40.107.244.90]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE55313F1E; Mon, 11 Jul 2022 10:48:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TIywPVsFO5pUsknAwerEEpwN4V4Dh7zjSGmoQVVdrIzc1pzpmrAszg0LnrxO1e/EqI4PbFhBPu54W4SeBa/qDVuwVULJnykKR7QkIAHASgTpeympEQ5qQcuY7kS66t6N4t5LC/YC4t8+DltyCKZA54irvFEoXJcemC2tFFzn0Rt226iBO5oPxawdiCMmuU94OHj16PRGmh7xWkznqGLPYzbNZDe8h4SYnC8whMAkrkOd8N4ZDxuGBMyKD9v2PWn/dy+immDw+H+B75KJuU5Z2Xbi4cUhL/o3ajJt/yvo9iPkuoY6cGAau5iZDwot/srHt2uiyw5rgRtnhsBPZvSpZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iFtmH13/jDA3heSMON/hN8D0/8/ZJBChjjes3eOKhvA=; b=PyNOjvjnKQlPH4Pbpik7V6grZMYtLTUnKHvBhmz9SLNL26ZUU8Zh51Zh0WyVyrgzDDL8Ojgr2+k6QeWOWrrEZFeDZOgqP5PBYXwgDiAk5ijj0etBcvlJL5SjuiA0oRga3W8/WvLcDluLENqQ2nyIQhQ8eEm9AOhJvHOA89AkCEQlZyaI1UGnNi+nqGqiuYbLjEc3b1IovWyl9li2w//jGZZBeITgcap4c2hIWvI+WQT6n+C3NvqUElWGSDRfW5xxLCDslchpDcmLVYSD3+vFIe9KxVaeJTB8RDVcc9uIHhAUyeKaVXBsdx+tMZVOrVuLZg/gJ53+4iFAg4l6FJfgIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iFtmH13/jDA3heSMON/hN8D0/8/ZJBChjjes3eOKhvA=; b=JYM41o0AGLj1ytMZAxGORV2UKppIgctVYifnLGppzbAODkBNHyRN4G8WOOp+lDgUlhHq9TFNJMAX5dVT9SR/4KE7gy2SmaPu7LDm1/WIVDwZmCBo/Xfxbvh7z4qsAST5RisH4PhkGNTF5HpyAzL6lsmfmu5gTfgtZ44JYcTTBi8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) by DM6PR21MB1321.namprd21.prod.outlook.com (2603:10b6:5:175::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.7; Mon, 11 Jul 2022 17:48:53 +0000 Received: from DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6]) by DM6PR21MB1370.namprd21.prod.outlook.com ([fe80::684f:525a:aab1:bba6%4]) with mapi id 15.20.5417.010; Mon, 11 Jul 2022 17:48:53 +0000 From: Michael Kelley To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-hyperv@vger.kernel.org Cc: mikelley@microsoft.com Subject: [PATCH v2 3/3] Documentation: hyperv: Add overview of clocks and timers Date: Mon, 11 Jul 2022 10:48:24 -0700 Message-Id: <1657561704-12631-4-git-send-email-mikelley@microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> References: <1657561704-12631-1-git-send-email-mikelley@microsoft.com> X-ClientProxiedBy: MW2PR2101CA0013.namprd21.prod.outlook.com (2603:10b6:302:1::26) To DM6PR21MB1370.namprd21.prod.outlook.com (2603:10b6:5:16b::28) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 05373536-fd54-4737-47c6-08da63659f05 X-MS-TrafficTypeDiagnostic: DM6PR21MB1321:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zBamVFkqnbUHYNjVgAyZMcZfcvpxtPV69ErwbgKGshBcldpGP1bP8wWcoWISRapzJmInddQ+L8+084EcENIEsskKLKWJGToJY+rpiddLyCV1GSIhZ7y7bIqyLXzbXeqXeMPm5dTCwheqorJQQXmr95hnyr3wj6WVSkAYh/0pitEsytvwkaPdDN8QgOHRX6a0nwlWUzaGdsYoFpusBlPV7R08+vCn/uvwFIfXHoz3iMzoWys+syLt+nTl6NCBJnYc8RqUKXSjANE6bq1zOzzVyWUjbYu7peZNbZ3vGTEvKfGnR9IviUB6RqJidxdHDU9KL6eSeR4g3xCdqQh3Q6gPqfaWKcUmZLH82l7/QQ41CDGxn3eq61RhKAmtJaMj28vFq9pgoOTGwpZ1bEyDHuFr6eaccWHAvONTvkh1d3PI9HajiJwbmmxq3AbDiQ6kz0b37DJX5znadQCSvyW02RGQXs5ri3zaC7PLsOooUX9Ik0v+BrF+g0elvu/pX45Iw+YHuS7pSXXjyHzuHozAUXM7GdbHlgy+l6VpRevv+njta54pivE9/evKuh4fRn4OXSZ9mvzmkHQWbXprpkIB6O78k7NYdEjHlu/icDUM1CpYsl+wFVG9ezZgvyVsjZMOCeQbmxOEUmoWYn5M+0TIFZWHMLIo/58HoLXKG0YDBfuTc3L7v2Rl8bcqc9GZ+Acm0S2GUe8TMfaLveeY5iVe+4WuElyC4D3quSvhAg6KBANvsn9mQJv8QclHqXl5djxWDEQ2YUIqKqpok4d5FYCk65ULz4Qo0fNablQvADW2RMcNw7w/PgQZWlUHgdDbQS2rpjthXYJCH7d7SBM7957+kn6LcszWXl0YWnzk3eGVeZb4MRg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1370.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(376002)(39860400002)(136003)(396003)(366004)(451199009)(26005)(82950400001)(8936002)(5660300002)(2906002)(36756003)(6506007)(82960400001)(66476007)(10290500003)(478600001)(86362001)(4326008)(66946007)(8676002)(6486002)(66556008)(41300700001)(2616005)(38100700002)(83380400001)(107886003)(6512007)(186003)(38350700002)(6666004)(316002)(52116002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9ffaIQhFF6llqRwumtkRn1qhBTCfTkuS4gVADjmX1WmFtJMZwHuuhU7RbPgh?= =?us-ascii?Q?uUy/FMRKMIwPtA1MTjjqY8slqKWfFsnSkou4U/aQKqFxgekwce7v46KvtYrC?= =?us-ascii?Q?Q9br35DTwZ1c5HUFgcFuNaEqGCvLdaDsY/WmX7ZcWENgBw8HbEw/T2Qx2KKs?= =?us-ascii?Q?59ZW8Rn71HyDvX9KrFJf7OpTnieg8b2+AAwce/Ul3NyOeelEMMQw6xNH1DOT?= =?us-ascii?Q?6X3QkV0PdtCS7YCUATmKEuOvGX4diej0cRVBzDp0SnWghpNPxC3SgbzY9AHb?= =?us-ascii?Q?VeuteUz1iISK1DqBPkveTEE2xTtz7Op3uaX9pW/Fzj8+HOqhBucZKI0yk1jb?= =?us-ascii?Q?gQUF3/xPz9GDOsk5V4x4wXA9yEMKd2z2SxLcqBcko6XI+GiQfwB762yGiDzL?= =?us-ascii?Q?NLsUVqTshSIF6SOHnm//+WJz4jz60rmP0ot9BMvNjwyfS2y3oVAGRBoU6dOQ?= =?us-ascii?Q?/LGaCu6LSJcHZbcXyL5pauig7/APbApQAFw2AulKt8zjsuDhFmUTJP4zPMAl?= =?us-ascii?Q?lXbA7bjTzrO+a2joxwICc4IWocQgLEE0XxBcrMbCa69uq+yBU5d5I43Uhss/?= =?us-ascii?Q?x+8CEi1UR6DCsZpS39pg9KLFGZ/IQxH4PMDJR9NZ7c3H+hzygFFfh6D9SaJn?= =?us-ascii?Q?iFKqcTiiHGlSoNSYcvjvEyzVspBdUMCp0pQV1npSeQYQx8dBTi5r2r2oHRmd?= =?us-ascii?Q?/CmqjqlRaLyORFamS7eNHr1k9UMxuClrR3XqZfMXB+avHAAd5svM6+LbddmA?= =?us-ascii?Q?WZiNaCuRKXmNSvegYHWQZj0U5bLaBrbl3VpyR76yILqtpenpteAaMgKGCKjv?= =?us-ascii?Q?Vaw3dosi6c/14jDYLrZLNqCFFiSO8+uc7UwgHDLv1tJLJC5rFkHRpNyPGcin?= =?us-ascii?Q?pGViBIQDTl02KB6TKzFxz9dt55YBjwxmFSHPy3/t2DON6Cdj/SBOsdZwVvFV?= =?us-ascii?Q?H+hnuI2P5wiGVsq6DtZfaYwpo5iazdSWmcHrm94YAziA0d8nSoo9MAd6No7J?= =?us-ascii?Q?4vEvfYhuOFBoRzdF/lldHIKQ7GYxgj+DiB5+NzrXbtYK/vY3YoD05TVu78mV?= =?us-ascii?Q?/+CG3C/A4gPEClf8iC8a9f/HRLe4B9hUOKy9euQ/N0aT5sj6NyrtBcuCuv+e?= =?us-ascii?Q?krRDFj24XtOofv0v5EVmnZhaPc9XuV8MpT1DTIskZKAp5Ecnu5daRBg4UJ4F?= =?us-ascii?Q?V54Ny7JCk03FdR7CMoNzg1+c44L8anxqRxLwsdauYMSCICHEOHNOcXlQ8oy5?= =?us-ascii?Q?pPmEm2Qa5h3RzsIOb9wC7/AnPMMq/WJB+uyoUInDQ8UF89k87zD3plxMcWHL?= =?us-ascii?Q?MNVKTy0h0uCmAJ5m4GwEaa0or+fmSIV0ZmqKO8MkAxMsAGojeqYeNbHm0xuT?= =?us-ascii?Q?Pcpi9Qaqx61qQrJlLqfiq0WUtf0Ut7EY5mlIETmX5UjNefuAvVRBOhc5AQox?= =?us-ascii?Q?NLDK2aYMROEarv3Sf9p/rHs/2yLF8SLpzA64g2O7JCFp8cX6lDXnL8yw3PP0?= =?us-ascii?Q?mrVnsnls5FTSVXL0Eix7Lz7kztCpxgt4QcplhZskITMoqbS5dHYYeHA+bxaa?= =?us-ascii?Q?Xj5y1UDWh85IzWWyNFi6DRSGICP27jKZxJXT2czq?= X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 05373536-fd54-4737-47c6-08da63659f05 X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1370.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2022 17:48:53.8366 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cxpNBuaQPdanFdY4dhEdtQDoTVqb7Xs8QcaEfD3bpbM/NrobBYuy8iPB3VytC/GdIjCyc5nvVVeD913cqHhNPg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR21MB1321 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation topic for clocks and timers when running as a guest on Hyper-V. Signed-off-by: Michael Kelley --- Documentation/virt/hyperv/clocks.rst | 73 ++++++++++++++++++++++++++++++++= ++++ Documentation/virt/hyperv/index.rst | 1 + 2 files changed, 74 insertions(+) create mode 100644 Documentation/virt/hyperv/clocks.rst diff --git a/Documentation/virt/hyperv/clocks.rst b/Documentation/virt/hype= rv/clocks.rst new file mode 100644 index 0000000..2da2879 --- /dev/null +++ b/Documentation/virt/hyperv/clocks.rst @@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Clocks and Timers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +arm64 +----- +On arm64, Hyper-V virtualizes the ARMv8 architectural system counter +and timer. Guest VMs use this virtualized hardware as the Linux +clocksource and clockevents via the standard arm_arch_timer.c +driver, just as they would on bare metal. Linux vDSO support for the +architectural system counter is functional in guest VMs on Hyper-V. +While Hyper-V also provides a synthetic system clock and four synthetic +per-CPU timers as described in the TLFS, they are not used by the +Linux kernel in a Hyper-V guest on arm64. However, older versions +of Hyper-V for arm64 only partially virtualize the ARMv8 +architectural timer, such that the timer does not generate +interrupts in the VM. Because of this limitation, running current +Linux kernel versions on these older Hyper-V versions requires an +out-of-tree patch to use the Hyper-V synthetic clocks/timers instead. + +x86/x64 +------- +On x86/x64, Hyper-V provides guest VMs with a synthetic system clock +and four synthetic per-CPU timers as described in the TLFS. Hyper-V +also provides access to the virtualized TSC via the RDTSC and +related instructions. These TSC instructions do not trap to +the hypervisor and so provide excellent performance in a VM. +Hyper-V performs TSC calibration, and provides the TSC frequency +to the guest VM via a synthetic MSR. Hyper-V initialization code +in Linux reads this MSR to get the frequency, so it skips TSC +calibration and sets tsc_reliable. Hyper-V provides virtualized +versions of the PIT (in Hyper-V Generation 1 VMs only), local +APIC timer, and RTC. Hyper-V does not provide a virtualized HPET in +guest VMs. + +The Hyper-V synthetic system clock can be read via a synthetic MSR, +but this access traps to the hypervisor. As a faster alternative, +the guest can configure a memory page to be shared between the guest +and the hypervisor. Hyper-V populates this memory page with a +64-bit scale value and offset value. To read the synthetic clock +value, the guest reads the TSC and then applies the scale and offset +as described in the Hyper-V TLFS. The resulting value advances +at a constant 10 MHz frequency. In the case of a live migration +to a host with a different TSC frequency, Hyper-V adjusts the +scale and offset values in the shared page so that the 10 MHz +frequency is maintained. + +Starting with Windows Server 2022 Hyper-V, Hyper-V uses hardware +support for TSC frequency scaling to enable live migration of VMs +across Hyper-V hosts where the TSC frequency may be different. +When a Linux guest detects that this Hyper-V functionality is +available, it prefers to use Linux's standard TSC-based clocksource. +Otherwise, it uses the clocksource for the Hyper-V synthetic system +clock implemented via the shared page (identified as +"hyperv_clocksource_tsc_page"). + +The Hyper-V synthetic system clock is available to user space via +vDSO, and gettimeofday() and related system calls can execute +entirely in user space. The vDSO is implemented by mapping the +shared page with scale and offset values into user space. User +space code performs the same algorithm of reading the TSC and +appying the scale and offset to get the constant 10 MHz clock. + +Linux clockevents are based on Hyper-V synthetic timer 0. While +Hyper-V offers 4 synthetic timers for each CPU, Linux only uses +timer 0. Interrupts from stimer0 are recorded on the "HVS" line in +/proc/interrupts. Clockevents based on the virtualized PIT and +local APIC timer also work, but the Hyper-V synthetic timer is +preferred. + +The driver for the Hyper-V synthetic system clock and timers is +drivers/clocksource/hyperv_timer.c. diff --git a/Documentation/virt/hyperv/index.rst b/Documentation/virt/hyper= v/index.rst index caa43ab..4a7a1b7 100644 --- a/Documentation/virt/hyperv/index.rst +++ b/Documentation/virt/hyperv/index.rst @@ -9,3 +9,4 @@ Hyper-V Enlightenments =20 overview vmbus + clocks --=20 1.8.3.1