From nobody Mon Dec 1 22:35:42 2025 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3D9827E04C for ; Wed, 26 Nov 2025 13:24:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764163445; cv=none; b=rHwdb2X8fOAmkaGAIHi7FBUXv31TwrMzMB5dcR3824Gg4TMiNPwB9Qo5JTtPrFBGndzWXlBh4FLX9T5tiHX5w26lUkCg9Km5fmEvKoTLFpYZx5CfBKHM7yGvRmhpVVlYn/xE0T5h977+w0yeFzpJDm1CI6A6uwk7+6pkEqmx2Fs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764163445; c=relaxed/simple; bh=oYg1p44Z8GaWMw9nPcAzfdJ7xMN6b9aEk/jGyrSAPOA=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=N7SyzdVYKDzKtb6n+3F0WOlzHnj6uwLgvLo62ShprtQ29/VdUgGvZO9AyK90lef741y44YMCy0WlH2nZQlDdRNlVhP3xMn9H2R+kGtxDH+SMfvvS90aVIED8kmpbzb3eeDBaFaE8Qgb5ZpRkJdj2yJZD4Z9zCX49H2ggSKissJY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Iwu/XTf7; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Iwu/XTf7" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-b76b5afdf04so453393066b.1 for ; Wed, 26 Nov 2025 05:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764163442; x=1764768242; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=mZlK0a0hRJYeQzUasmwscAQxOLWf6JuoYqaDVqUa+0E=; b=Iwu/XTf71NbE5aKYtup9HUWPVGUcKPlP5vjbYXeIvpF7VgQYWlN8b84ikxDxHNw4vu GqQRJxcIHz+tZITrzpA+xr/WWAui/aieVmaQC0crcl6VOTFUbYYUPzPCtV87wNLkFxgO 9ZMA0oOa1YQtRAJ5j92VluVyyF8p69/otE9ptafJwztnLSNOWhEYZ2sTJIaHTlHF5so8 G3xr3yC1WeYR0mnfIiKCY/TyGgrL4+HNenU1hMpGXnZfxBT20l49gxhq5U4mYWCagUvs bSfm35kpGPWMCyHZVyUYB/Aae47pQuPh9OYLQZG5GszLZbaqj/TNpfqLImscykCcu+Iq 7XCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764163442; x=1764768242; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=mZlK0a0hRJYeQzUasmwscAQxOLWf6JuoYqaDVqUa+0E=; b=YE+LAwE+uyNDSULMtwzywZqQRP8oZ149c8jD8EnPD4LROHs+I5S7Ij7ltbzuwnEn+E byVZghY7E6Esb2XHEB7vKKb6WU1cZISGs4nOJPjX+sn15v6dMsv7BciVcqbIwOYsg3hY Y57XzjnInA268PFHVXqnkyQNQG34pWl6mAgLFhliVZhvhSgo7EJ0MJx38slxNi1PpLEr SsV60yfLnU7anjhQ4Jj10iAEd4HKNkSyS7n2Gy1OkG24qqmiXhtu94Qa6NM079amWwr7 ctaHmnV2kF4R81/kk4t04nzaSNqolhVjQc8AJqDazFjVUlJzmUWuUmX7Z4gQ3ziSAfPU 7VYw== X-Forwarded-Encrypted: i=1; AJvYcCVOh+XP3rLP+OBozINFVtiv7QBxD48ArH6pCmwjfcPUk77FanQ9iExuGWoWtXWNW66WNixyBSS6FLiKi6I=@vger.kernel.org X-Gm-Message-State: AOJu0YwyFV4xiXIX6zPRWBfPwwn0+PxmzVg2Pp8NjDyxJC40Hze7ubkX UnrL2nskQ8M08ye0UIW1iN5mOVwehHSeAqJNUpkytH2ALgJHhX4zBnho6Up2caln X-Gm-Gg: ASbGncvUvDMx0bqi9Z8+cIHdnhzmoVCn0j912VPGuG4HwSe+euneshnsApYad/HW38o sCXfQ+u28JhdZ+kBtnmwkohN5VmYIqs36eQA1Li6E9jaE9O3nXELKhIwVevthS6fzaTGrpgY4B2 k5VTMmsF/ykyJJ8FFqGBjmxG35uShVRS29HWNY4JXkz2/Bw06u4BozjWVmxHXoxL/1kVls1EjTj AN6WLFoUFeBeIGlppqOHdErgJs8bKktYduQSb5Fw6zIT4koQ6ewzN+H7vQXQ0TQfy4fjsLZpXma j0WAkvkowmr2jSwWVQuJGQiRtn64AobdUphqQlFDJfGhMyes3ll95oDd6BOgcMrVMyBKJMeq1oS Ly+lh6EHOXgLSV2hX+txLhksWFAIgumb0HXF2EH0LFZQCyAYMyso8c1/Ne0NjZGf6kduVo0Dwtq DfWwbSpHqaWIc= X-Google-Smtp-Source: AGHT+IHlaAZ5h/R06BjW5jshtlOy9Icgtm6+FUvzqgRPhRCjC8mx+9GhUgt6CrfbbVr5XF7IshzsAA== X-Received: by 2002:a17:907:7203:b0:b76:3548:cdee with SMTP id a640c23a62f3a-b76715e630bmr2098605366b.25.1764163441813; Wed, 26 Nov 2025 05:24:01 -0800 (PST) Received: from fedora ([193.77.86.199]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b7654d73e6csm1892743466b.25.2025.11.26.05.24.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Nov 2025 05:24:01 -0800 (PST) From: Uros Bizjak To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH] x86: Optimize __get_user_asm() assembly Date: Wed, 26 Nov 2025 14:23:32 +0100 Message-ID: <20251126132352.448052-1-ubizjak@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On x86, byte and word moves (MOVB, MOVW) write only to the low portion of a 32-bit register, leaving the upper bits unchanged. Modern compilers therefore prefer using MOVZBL and MOVZWL to load 8-bit and 16-bit values with zero-extension to the full register width. Update __get_user_asm() to follow this convention by explicitly zero-extending 8-bit and 16-bit loads from memory. An additional benefit of this change is that it enables the full integer register set to be used for 8-bit loads. Also, it eliminates the need for manual zero-extension of 8-bit values. There is only a minimal increase in code size: text data bss dec hex filename 28258526 4810554 740932 33810012 03e65c vmlinux-old.o 28258550 4810554 740932 33810036 03e674 vmlinux-new.o Signed-off-by: Uros Bizjak Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" - --- arch/x86/include/asm/uaccess.h | 40 +++++++++++++++------------------- 1 file changed, 17 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 367297b188c3..cb463c1301f6 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -260,30 +260,27 @@ do { \ unsigned int __gu_low, __gu_high; \ const unsigned int __user *__gu_ptr; \ __gu_ptr =3D (const void __user *)(ptr); \ - __get_user_asm(__gu_low, __gu_ptr, "l", "=3Dr", label); \ - __get_user_asm(__gu_high, __gu_ptr+1, "l", "=3Dr", label); \ + __get_user_asm(__gu_low, __gu_ptr, "movl", "", label); \ + __get_user_asm(__gu_high, __gu_ptr+1, "movl", "", label); \ (x) =3D ((unsigned long long)__gu_high << 32) | __gu_low; \ } while (0) #else #define __get_user_asm_u64(x, ptr, label) \ - __get_user_asm(x, ptr, "q", "=3Dr", label) + __get_user_asm(x, ptr, "movq", "", label) #endif =20 #define __get_user_size(x, ptr, size, label) \ do { \ __chk_user_ptr(ptr); \ switch (size) { \ - case 1: { \ - unsigned char x_u8__; \ - __get_user_asm(x_u8__, ptr, "b", "=3Dq", label); \ - (x) =3D x_u8__; \ + case 1: \ + __get_user_asm(x, ptr, "movzbl", "k", label); \ break; \ - } \ case 2: \ - __get_user_asm(x, ptr, "w", "=3Dr", label); \ + __get_user_asm(x, ptr, "movzwl", "k", label); \ break; \ case 4: \ - __get_user_asm(x, ptr, "l", "=3Dr", label); \ + __get_user_asm(x, ptr, "movl", "", label); \ break; \ case 8: \ __get_user_asm_u64(x, ptr, label); \ @@ -294,11 +291,11 @@ do { \ instrument_get_user(x); \ } while (0) =20 -#define __get_user_asm(x, addr, itype, ltype, label) \ +#define __get_user_asm(x, addr, insn, opmod, label) \ asm_goto_output("\n" \ - "1: mov"itype" %[umem],%[output]\n" \ + "1: " insn " %[umem],%" opmod "[output]\n" \ _ASM_EXTABLE_UA(1b, %l2) \ - : [output] ltype(x) \ + : [output] "=3Dr" (x) \ : [umem] "m" (__m(addr)) \ : : label) =20 @@ -326,26 +323,23 @@ do { \ }) =20 #else -#define __get_user_asm_u64(x, ptr, retval) \ - __get_user_asm(x, ptr, retval, "q") +#define __get_user_asm_u64(x, ptr, retval) \ + __get_user_asm(x, ptr, "movq", "", retval) #endif =20 #define __get_user_size(x, ptr, size, retval) \ do { \ - unsigned char x_u8__; \ - \ retval =3D 0; \ __chk_user_ptr(ptr); \ switch (size) { \ case 1: \ - __get_user_asm(x_u8__, ptr, retval, "b"); \ - (x) =3D x_u8__; \ + __get_user_asm(x, ptr, "movzbl", "k", retval); \ break; \ case 2: \ - __get_user_asm(x, ptr, retval, "w"); \ + __get_user_asm(x, ptr, "movzwl", "k", retval); \ break; \ case 4: \ - __get_user_asm(x, ptr, retval, "l"); \ + __get_user_asm(x, ptr, "movl", "", retval); \ break; \ case 8: \ __get_user_asm_u64(x, ptr, retval); \ @@ -355,9 +349,9 @@ do { \ } \ } while (0) =20 -#define __get_user_asm(x, addr, err, itype) \ +#define __get_user_asm(x, addr, insn, opmod, err) \ asm volatile("\n" \ - "1: mov"itype" %[umem],%[output]\n" \ + "1: " insn " %[umem],%" opmod "[output]\n" \ "2:\n" \ _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG | \ EX_FLAG_CLEAR_AX, \ --=20 2.52.0