From nobody Mon Feb 9 13:49:04 2026 Received: from mail-oa1-f42.google.com (mail-oa1-f42.google.com [209.85.160.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A686BA50 for ; Fri, 21 Feb 2025 00:09:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096571; cv=none; b=t+lSBzcBk+azvgd8N+t3XMB4JnA98Kaq7jzDsVCi4cdRLh6GHUsexnzebAo1SkmSocg8WHv26ixvw9XU8v+9jTsCIlvCVgFyue6/L+YJIgBUPvY3dGzD8jSu7CESovEvIrnd/6E/eu45Jv6SQXc+cyxatJt/gm5+CL+MiFWkAQI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096571; c=relaxed/simple; bh=VggzceCPBii46RbE0l34G01cdRhcCActrNbfcOljT7w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bYTlEbFYBcaZdDatxyj/cBzbnsYyijCUHwC00mvKj8zZ8ymWyltKIcyDI3tymgr/wNRDsym02d2NZl1vgEWxUQ+fzqRjHEHvIWLCVT2l+gddUD9+St+6zjQfJGTLxmtjEGKTIzHy97NWG/C9JPVvkz9W6AhC5qB84SLITDLcN0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=WycCcGS4; arc=none smtp.client-ip=209.85.160.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="WycCcGS4" Received: by mail-oa1-f42.google.com with SMTP id 586e51a60fabf-2bc607b33d5so857348fac.3 for ; Thu, 20 Feb 2025 16:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096567; x=1740701367; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ynQVfR/dFk01f+alSHWcouno+EVl3KD0yb8OXJAhnGc=; b=WycCcGS4Me9/ilDac+iLEw/DzPVkJFEdjjhePFX1/D8Bi/SxMB99FdRmHVcjWlhGdF BpTH6XTjnBmv6dPFZ5e7SQFti72EUJ1Af4IBT5m7mujdE9LRP/bYHrNqyqLSGsnb7gdz zaGrmE3hOlk4358koKqwI8bCARnwbzQp87v+XnARCKxDApJISX9TBzFGDhkTchZiL7rv 1107YODm53+awPetybOr0QT+W/qXSFmEdRjTvZjzwXkG1s2ObpupT7AEnh8n4Jb9P04L JRJVBc0Vk1ruRx/vkuBA3E8KY7+jKoXlaHYHbZJpWKLol2Kd2jtUVfvTs/vLfACJ3G1L rSKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096567; x=1740701367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ynQVfR/dFk01f+alSHWcouno+EVl3KD0yb8OXJAhnGc=; b=E4mn9a9Ez7hNoHDjJlDu0T2/ltIEoe8BVqz4fVDbREPYljWnFNEeO5pstzCmEfVN/d DUnt8fJicBOKuui4iSg7vXR60cpTQv44hl4Rsc+c3Wxf5hsZ0WCrabDivJQOoZTIDHx0 heE+tApDvHrPZVKWX9fxpURYDs9w2appQ3EWeGl3cs2tjlgt1REsxgucWfwyNyTA+jYw xIA6LR8endAiGPn8/UF0KIteCWz+MM9diEcyV4FCxL1t2BYM0ezHkoLw9g+MD/QKPEql Q7qT8fJiKuzuHycogvZO/1DZca/PLbNKJbC6oY5IQuseAkx4CISFM9Li1cO0Ak62vOJk uh9w== X-Forwarded-Encrypted: i=1; AJvYcCXlGVYGsExD2nodZbIlASsY0jApUFnRDn7AKzNc6ouMD/zPj6T+w3ixsHHco2ttQhRC07HsluzNTDWfpJQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxCq7lNkbpkLHvZ/zhsdlVVEtk4jCyzCaGikZ26Tfgw5nl548sp U8N/gFs/9Qb7h5NUAb6ZuDMyjgga705Qj6J7Tu3VXkHNCrhhcmg6QjMVJY7kCw== X-Gm-Gg: ASbGncusVcAdVzIWP1tSXAcoue04zQ1JGxW4LxetseM1nR0j/haIWQXj/iU8k0e8SnF F7HlhQ5wYuAYV+Cj3xueX+IgsdkEt23Gpr2SwH80XG1UjQOWjhI2z9xJ+rymRRbZGvPPNtKQAnH Rrr2S0kULN41TWg/RDBx/z5HFfmNBnMOPR0zcrAipRMPFqHXDGHg3mbf1RoGEMSRmHSaj9P0txa Y+j5nXWwY4rNWn6xShz2CMxt9fW4m/FRw79+c7lfzSEdVIt9rtAg3ofLa+THLpxzn7x0LVWP7CA gz0JUcvS3NLe1hivZkF/M8cbb/ZkLs5CEg== X-Google-Smtp-Source: AGHT+IFhyE2QRwXal+GO2Cm8WYvshkYyOzXF4EFDI4EcmkmeGlVmtYav3txhL7PD6lMOsijhkF4Ggg== X-Received: by 2002:a05:6808:384b:b0:3f4:b0c:ab6 with SMTP id 5614622812f47-3f42469dfc3mr1403840b6e.1.1740096567630; Thu, 20 Feb 2025 16:09:27 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:26 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 1/4] riscv: implement user_access_begin() and families Date: Fri, 21 Feb 2025 00:09:21 +0000 Message-Id: <20250221000924.734006-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 63 ++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index fee56b0c8058..43db1d9c2f99 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struc= t mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") =20 +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x,char, \ + __typefits(x,short, \ + __typefits(x,int, \ + __typefits(x,long,0ULL))))) + +#define __typefits(x,type,not) \ + __builtin_choose_expr(sizeof(x)<=3Dsizeof(type),(unsigned type)0,not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,56 @@ do { \ goto err_label; \ } while (0) =20 +static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr,len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin(a,b) user_access_begin(a,b) +#define user_access_end() __disable_user_access() + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err =3D 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err =3D 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, label) \ + while (len >=3D sizeof(type)) { \ + unsafe_put_user(*(type *)(src),(type __user *)(dst),label); \ + dst +=3D sizeof(type); \ + src +=3D sizeof(type); \ + len -=3D sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst,_src,_len,label) \ +do { \ + char __user *__ucu_dst =3D (_dst); \ + const char *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, label); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ --=20 2.34.1 From nobody Mon Feb 9 13:49:04 2026 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 184AB2111 for ; Fri, 21 Feb 2025 00:09:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096571; cv=none; b=nAXkPPxUBob8yY/qM4RKQZio2YdnGzpFwqc/hmldVOhu+s5j35ux+oqWOOcHfD1hNV7hVEJAHfEEn6y1yPT6wfb8ix+BqagkOKnRUxeDGPFp0mooBvw6MzLeorg8TznH80Q+RkqkPeF3bCl7gDIwyFlx1xanjoVV6YkiVS0AyTY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096571; c=relaxed/simple; bh=8tbJj5z5g99BWv7aelIaSWo1aUJ9tKIbu+IV0jhMGM0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rTz8VG9LDZutRSXuV3oAvhaMMKHmf+0nZKyia/QdxIulGZSUbXYPo/JrT+uxqetiN2s2hCCAK6wGdqKxsqcXDx9FUxezWMJdZwOcrGclTlO5ir4+K5hH3phQRTpwl733Tt4dZKKh6oEHs8fTmVBDkBdivT38fwiIu0DfLd4WUvw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=JKmvoI7z; arc=none smtp.client-ip=209.85.167.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="JKmvoI7z" Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3f41091f7c5so829570b6e.3 for ; Thu, 20 Feb 2025 16:09:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096569; x=1740701369; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AVS2gJ1tGF7KfC44fJzPB6s+5bKduKUUF483rWDGJdM=; b=JKmvoI7z7UpELzL8/u6IlZ96brShkyefFqDK8nEgKAS3c0drmciJau0PjYjIxWO3Lg hziuPMdgslbKlUn6SFxD1RkhJfusuxuK/LwhAB0MdTuoEDyb8QqHOSDSR9HrdSF4YCRn 2m2hrUHzmgqisrg6WGG42GVJP20bOHSbHRlsfB6n/tuH/J3QmiAdLLc6SFKW5JnGO+Jm 5qMe5aHhlbVYbVoKwLEQ3Xsj++SCHgJunFth1ysS8zeVBU7fXE7ClAvKiZfWB/2pIt5o 6HWFXyZLDQmwTPyikI5/BKSicz1aETF0+qbH8re7NlEasDx1m2hdWvW7ayJqYN5IhK+i JO9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096569; x=1740701369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AVS2gJ1tGF7KfC44fJzPB6s+5bKduKUUF483rWDGJdM=; b=daw+sEC/ENc+iTnA2ZToSkU11QeV8GDw0xBC8dzEG73yHOjnpn953SsCGQeixE2gHG EBNmWW1uyMq36PabOAgzm0+ajo2ETe6vWoBK0vIU0oYsXjEPoNz7G5fAMgf/iarYLXDC 8QjfN7ZOvMGPm1dwIFCC7pqDAckwyem+9EO8RWTX2lHeWjYbPsJB7JnsSfi8S8gFapR6 dlTyaR6y8RmeUYDk7h4wV+eUmu971jeog/2AITky4tFYBsJ+fxPQr7GjwuNy4koQlO8Y xOz6roWfE0NZLx95BM8bjA+Wwld0xX0+ZNDQyvvsnOM2XoLIuX+wMObXUBIFyZInK/S2 YgOw== X-Forwarded-Encrypted: i=1; AJvYcCXlAOTVHtm/Hv8TiBovQ7YQ2B6tzTmOsPT35QrfwmqugD9zTrs/jL9DrAP9C3kWsiztq5UPdxDXlwPs/30=@vger.kernel.org X-Gm-Message-State: AOJu0YzQMxY6FZImoHQh0agDhXyGPVyVTPhsd4gzOVChNX6qxNcOYu6Y XewcOr3jJaTIofD4QDX8imGLBDSdF9wgYhrT3chpW4jKieknN5bkg/9UUaWAvg== X-Gm-Gg: ASbGnct4zHoW7rfeRO0f/qwFmuFI0vDERj4WI7pH1cJWO3ukWfcXiHdVGFEeEh2omxm sV0FIgkAziIie0yk/HgMOTI6PS8MN1ZxeTvy2LdV+IgLkYqs1/pmCDy15we456BxmkZ8Ph+j95F KBuaCZ4QA8jAsOm/2q4tTtH/foYeJXYXg0UD8rZ0UdemtR4h1Yg+pD/Xv2FamlSCAMmKcUx8Kud tQSxWU7WLpvIg3GYPL2XeRKRa0yKkL6Ou/Qp/s5jAgoAhMHFGeEkh999950Txbk5Q4PWHY9S6+o MftxFcTphPsbqS3mFsFsDJlAJNXDNJfkAA== X-Google-Smtp-Source: AGHT+IFKRfnHqdiP+z9IB6oatR7TvVXNmSxjSJMg1TAS9ThRlHl4CaTM35ogp20h33uxLtf3x9nOIw== X-Received: by 2002:a05:6808:1490:b0:3f3:db88:841c with SMTP id 5614622812f47-3f425a75607mr599965b6e.9.1740096568994; Thu, 20 Feb 2025 16:09:28 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:28 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 2/4] riscv: uaccess: use input constraints for ptr of __put_user() Date: Fri, 21 Feb 2025 00:09:22 +0000 Message-Id: <20250221000924.734006-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 43db1d9c2f99..fb3961ecf38b 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x =3D x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=3Dm" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) =20 #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x =3D (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=3Dm" (__ptr[__LSW]), \ - "=3Dm" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ =20 --=20 2.34.1 From nobody Mon Feb 9 13:49:04 2026 Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B47C433CA for ; Fri, 21 Feb 2025 00:09:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096572; cv=none; b=GfibyyB93tNo9c9tUr3g9Xx09TQ6AWyuTFxUQ5yDEjwvT+t0TZ0iB9wQgVOmMwQ0LH74Fw40XDDhMH074YC9nziom1Yl06A5lgufFNh5HEZgOevtGGTJvB8AGfdbsBNCZC7zSoDsL5oiNxmYsJTlSlx3Aap+LbDtiazvaqutjMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096572; c=relaxed/simple; bh=SQw27djHYUdQ5oODFTiumWVTyisqpnqd4IqqlvAk8oI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mgnDhI0TcVbYZzHynZ3qJgjZiiu9W7V6kKuHekHxtpK7LFv5BlbkOcVVpam2e0IE3sVpkZSETkWp9NB1ehWOqZJVDh6xfacgptIs6jc1cpgwBrjlJv3qdpvicT4X6cIr1tUheiO7+LzH6kiqgXKOgRoawSsCpaxJrZZDYolxB4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=YPZYIaUM; arc=none smtp.client-ip=209.85.161.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="YPZYIaUM" Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5fa9778fa2cso744106eaf.0 for ; Thu, 20 Feb 2025 16:09:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096570; x=1740701370; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l5hM23iRroiayTpV3ONpzy5pooTVMbj1TF8BFXhb4r4=; b=YPZYIaUMJb6lBeQ6mpvVMI533A7QwHinb9+//kUJZ0+3hIUMZFX1ciMAkr8W/rQU3o AO3l6xROMBxKUIkNDo82CTA6TxVg47b/ciZBuzCCjVL0/W6oLukQPJ2O2RJ18FO3JqNM B5og5pvyxzXz82OJJZU3TINcL4ysYuz5sGzBHtA5pbpjlL/dTSJ5bHj5kDXH1OVajDx3 8slbUDOhmltKdub1Glup9gErYl0fiVK6L4osy3rQmjvpR7zS+v+Tgxk2Tgw+PygGHgWH f3J55qKYtogLb7sD8m5SQzE6EdHG7MFoIX5uIMCDVyNbNsGQ3XyVCiQXcSIAeLRhS+rl 13qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096570; x=1740701370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l5hM23iRroiayTpV3ONpzy5pooTVMbj1TF8BFXhb4r4=; b=hMuBmpP21TjqANOYmKV82BCclFsJjoA/UZDkCm0ZqcGLNZy06xHtMHS6SOhEkRA59A EfCLQ5gvpZs8Tgz6NWHWT6xtQJwaMNUauRekSulSS9ku1QliLEEyPntqW4StYQgw2wbm 7U7jbCFSPxhVF7mPFy8322PtLJ/cwauMsfji9tXvbvc4DcEviqBj0UUXoZKoZMG2dZ6M IoZ+wfCT7vq3fRuIRjeXf0RbQTcs4VIHDhnNMIMbvJuXzMWbZ8du02wa5AhP7SUOkiES oEH5goqyHqXSdy7UAuzviQERm9OjVsfprPhKI3H8CmMykKEomGRsp8pvBmFOZ2Rpbc9F 7DiQ== X-Forwarded-Encrypted: i=1; AJvYcCXMuaXb1hTwtfiSIhl+QwDDMwDap/pPPBLxNqiHoWx1p6Fhb/Z1H8K01zQbw8WZUsbb/mV5OprCM6YUVHM=@vger.kernel.org X-Gm-Message-State: AOJu0YyuyaX++Whh+bFkIDBof3NjS3q93KqE+hm3nKNAbSZw673ZCvjk o6pWNoSt9VHqUS0yCOcIiIjya+FLNw1peLuP2hDUhSXHnWDGLUQu+nhP8mSRPA== X-Gm-Gg: ASbGncv0yJwzXgEltse+5nLsgGonnUHTXu0q/8/WwADNwyyKfdHmFgWc5PWvSadeIdf Gu6xavjQSjRZs5UdEC9TlDj0rUafuca2kWdJbClp0W+k5J88bAdhI+YQ0YPzuBFUoId9no32YwF 6m8OFv79m20OD6IN5URB8Q4Edr6eXV3AQia3F0hrfRJ2QPs7T+9pDhrSbMOMxai7oNwJuA0yr71 0XWitUCb3MZ2u3jKX27k1d61/+AQT+qDBGCyQ4k/z72xFR0740/C8qp/FaZr7N7SI5Q8xJnHJ4I LVxBCQ8hNCE9a991tcMO+4HFa6eG1a8d1g== X-Google-Smtp-Source: AGHT+IF2JM2LKC17wpZxVEWOnlCgEtBUNVIC6uqX4gRfMQHIPsGUXAqsCDrHVc3eY40FDoWxYpcitw== X-Received: by 2002:a05:6808:3191:b0:3f4:133d:f18 with SMTP id 5614622812f47-3f4247ecff3mr1177279b6e.34.1740096569876; Thu, 20 Feb 2025 16:09:29 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:29 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 3/4] riscv: uaccess: use 'asm goto' for put_user() Date: Fri, 21 Feb 2025 00:09:23 +0000 Message-Id: <20250221000924.734006-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 70 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 37 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index fb3961ecf38b..1b926b61af66 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) =3D (__force __typeof__(x))0, -EFAULT); \ }) =20 -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x =3D x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) =20 #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u64 __x =3D (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ =20 -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) =3D -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) =20 static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) { @@ -398,11 +397,8 @@ static inline void user_access_restore(unsigned long e= nabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err =3D 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ long __err =3D 0; \ --=20 2.34.1 From nobody Mon Feb 9 13:49:04 2026 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51F278F6E for ; Fri, 21 Feb 2025 00:09:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096574; cv=none; b=YQ9ZgH60c7dXfR+tPtYCcsPZx14iab5vzgtKis/7WDcRP/2+NgySWOTUzvuG/MGnNXf8vLqdyQel1qbdlqCqMyV4oN6NJhhEFb/3ilIMpMP8UW7UxYvcne8UGwIvGQ4zgSJHc9Vh9R0/9g5vVrjiHgizEjT/19hkEONNXQmFIKw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740096574; c=relaxed/simple; bh=CQzSTQWE4thoM09MiQ9gz19svPbcWrFRlVo+Hb+XTOI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d31L03293W4w0v2pc79tPzZdwMOwOm8WXlT9rqK1V8gqyP0LEjmb9KDqQtb/nRijoM3EzC+SqZMsUoDpCzZ2MZhJ5mYwSAV2FVxoyyKQ4GrN1hcjSTU+lMkEs3QQXSAeesE8CFJbAYtckhzl/eME181BA1vbTsStabPtqobLlGs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=FAFM9OXF; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="FAFM9OXF" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3f405e2c761so852640b6e.1 for ; Thu, 20 Feb 2025 16:09:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096571; x=1740701371; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+knCqqnB60be5uWSQ10xJ1S6sb+4Be8XtAoVkEDcmKA=; b=FAFM9OXFf1EXC7sylAEZ2je/NSpuaNBCpt3OTasjyb9ScN0nymRGQbMhZe4k2UB420 knhsz9CpHtVb3GwoCHuDM5ZlM7hAg7JCHoAoQPr2x0Z4FLQKoreyf6FSfHQakliPCNVi 4aYFspia/nvPxSo6wYi+Vp7U4bfFo8HgsYp9wZLk3437lasUvaP0f5WiafpsSbV45h2s xzvKFa6uHGcnr+jsmmHZKBVp552g7exc61Vl2XyOjeZlxkP1i3WYOIS0Gw9gTpdb/wYp dMxokcz56ti6GRPHr6rH+dZ5IW1yORJO53SmzlsB9grIuOxK7l3xVqRWbSRtf516mhNy NiNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096571; x=1740701371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+knCqqnB60be5uWSQ10xJ1S6sb+4Be8XtAoVkEDcmKA=; b=bxkySZxw4gUzzxb4AefEi1WVK3R87Yv+KQwyUMqFYj/ZVH/U5vqKQCFmfayWwd1x4g DvaG1vKG3OyLbN1ujaWdc0YQYEkjoTGnr0KKfQ5z4FrCe63wYToksY2dW9JxCEgH9HvW vVe3brW04Am5XR1USoBHBUMIkq+WdQPLdNptpWSaZr2S4xnHKOrICM4FV0VQCnSBrJkn ULN8fOrpN9i+rRdcPnbKwwN2d4u3y4xGKsh+2+DR7PTHT8C0E6+KJbrbJ2f0+423JMPd Qp36iXs+ilsVFXcvOpMwuAfJEXwlbu8MiYGrlW4BP/DoTbQ3wbkT5uZOfx+GmalwQbGA Xk3Q== X-Forwarded-Encrypted: i=1; AJvYcCXYNPP/IXIvhEE8EhlW5b0x1tjNk0DMKM2Q/nFmdrdHizDjCvFxxWljgxy0BCdaVrfTdenDepjoz6VGa7A=@vger.kernel.org X-Gm-Message-State: AOJu0YxArkjmW/diwFHY1Qn425Vkel9+AdWLa9YZQdvOt5icr40h7k9w swTNh8WLQYa15vUdT4yQIvK3tikMMu/uXfNwbGNwHpVN8AoZC3HFbRrYYKmV5A== X-Gm-Gg: ASbGncugnVpdvuBFUVDPNnmiVdQvper1xS2YnSRsAnxE+8BF5hvNo4mIiGNifAVcn3i 9yWFXe+Sl+YCSkqW4ATzKQvP+Ij6qhBC9sCtpcz7WjgUgu6KYlMYFWwiyyUHW2qVxnmZfRL7sXJ s6a5myabdt5wiYlZwwQglNU/Mim8k250rMjhZdmWz1ln4oBshbCfZPaUzKfKL9MHWDs57nV0juZ +f3zIRU38rh2BYwyj0+J5aC8i65UANJzpovXJYMwVF6FtLvkbaD400sAy2fVPITMS1thfG83csY XfGfeK3+nAC9gwND+lMvZ85yU2Mx55x1+g== X-Google-Smtp-Source: AGHT+IEMGnDVwYsV62cBocsB6VRb+qRff3a2UiPYmr/fXmxdBlq3NZLsTutH+x4Vzk0cSqwcF1F8Ow== X-Received: by 2002:a05:6808:384e:b0:3f4:d61:36cf with SMTP id 5614622812f47-3f4247a204fmr1275951b6e.30.1740096571285; Thu, 20 Feb 2025 16:09:31 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:30 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 4/4] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Fri, 21 Feb 2025 00:09:24 +0000 Message-Id: <20250221000924.734006-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 90 +++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 1b926b61af66..ab48ef57565e 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,56 @@ static inline unsigned long __untagged_addr_remote(stru= ct mm_struct *mm, unsigne * call. */ =20 -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=3D&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=3D&r" (__x) \ + : "+r" (__gua_err), "=3D&r" (x) \ : "m" (*(ptr))); \ - (x) =3D __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ =20 #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr =3D (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=3D&r" (__lo), "=3Dr" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +154,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=3D&r" (__lo), "=3Dr" (__hi) \ + : "+r" (__gu8_err), "=3D&r" (__lo), "=3Dr" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi =3D 0; \ + goto label; \ + } \ (x) =3D (__typeof__(x))((__typeof__((x)-(x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ =20 -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err =3D 0; \ + break; \ +__gu_failed: \ + x =3D 0; \ + err =3D -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +223,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr =3D untagged_addr(ptr); \ long __gu_err =3D 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) =3D __gu_val; \ + \ __gu_err; \ }) =20 @@ -369,13 +417,7 @@ unsigned long __must_check clear_user(void __user *to,= unsigned long n) } =20 #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,11 +443,9 @@ static inline void user_access_restore(unsigned long e= nabled) { } __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ - long __err =3D 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) goto label; \ } while (0) =20 #define unsafe_copy_loop(dst, src, len, type, label) \ --=20 2.34.1