From nobody Mon Oct 6 10:18:51 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B227A285412 for ; Tue, 22 Jul 2025 20:54:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217646; cv=none; b=cxlJSQyZqrEde8nwaqZH/Q/g3QlXyCrfXbupVCm9QTfFKGX80961tiV9IS3LBQI5J9uKj7iNZU2gs5uisepsUuxrSNK9g6OwCZItM9ILruj1i9oYBN9cAJBoz6zrnVAbXVfT4SLNKZYR2QO/k0iXwOCS+9UoXY1xyC3qxZvOvRA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217646; c=relaxed/simple; bh=zd5DznBVI4sVvso/l/68ScJA/RSIQeiYIJRJ5CEjGgY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KLLy/QdYDw/j6+ekJfUrUdbOvcjCF0agDSR+t/eE1sfSOzzB97FkrjBDI1wE4x4ylykHGv9C90xWcVu4eezjl4YKxhJkOR89xqahyYbiPkoiX1jQTADnVrdi4xt0oEFlh64Unrrft/QUT92iRt/ieKdRkIcC9Ewxod34pcD6/a8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=t1m50MBX; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t1m50MBX" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b3beafa8d60so7409512a12.3 for ; Tue, 22 Jul 2025 13:54:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1753217644; x=1753822444; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ATtT7J3eGVlX0DAsIWIni7wioYwQuHnqhrVZz3llpO4=; b=t1m50MBXZZaal5IVGJPr1t0vSbd5dNShxxt06B3FoyuXs6zdkCdof3r9sAl3WN/Oxx yZkHUfCAWmDg+Uz2QO93PDRT47gB/DAiXqZQoWGbhiIKqdTu/W2IyzBet+UlpsJL8h2t 3r6qtnMaCbpdgluyfy27Gr6Upm4Q979JeJO+MkhtFwk2vLk6yYRwuewZ0MrKWAEGZAoW C9iPPDRLQGbpoEOMaRhrJuRYIpUpBNwd1CEZo1VboLkZcrpZkDMhrZTtGXlcKiTb2SqO 1w7VpahmcbGkQYR2SXc4SzWPMah6uNCesyJck9jO7UbI4UEAgEKuZW7q4IKO3dyfmJIA zEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753217644; x=1753822444; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ATtT7J3eGVlX0DAsIWIni7wioYwQuHnqhrVZz3llpO4=; b=iix5M7Y9lTLKg1R69SBUrxSNEF+mCwWY7M8pCvrJkt6qv7eyiqpAb5MKPjy9fzlwNX NMXdYzAcZRtZChUVNgidmP5s/Dwj+keF8nkFjHOc1dMTn7UcNmbKfL87NGSXEnTlnzOF jj5sCEJW45nZnTF10Nlfzy4kz+lXv0R6jfchM7LcMvwHEVzzVVkgVuLJfYVXfpGlOx2P TtgGb9zPHXitlprxgYRLSXU3AJmPd1nFdPfxPHUerWtLqbZlNm4jfne1LHhD/qJS3Mf5 rAM8p1aqFY1Q+TatDN0fyWN4CwdJeSGhI8TDSRU3EMqGCiw71jukpnFLZJXBR0JsWMxZ Af3Q== X-Forwarded-Encrypted: i=1; AJvYcCWmJnbjdse2TRASDyVARCt2oAk1FIb/JN0PcWMRnXnFk38GCzFgPzmYnfPXUmYIRowzRd8VC4Y1Is9qM6U=@vger.kernel.org X-Gm-Message-State: AOJu0Yzk52cQeARVKNnTnPm2B7bYLvkigzPOUQr4WiRJBpxxGag5Vspw VK+KPHKniOkqjyHETJjnBuvA0vQE4m7QHTvucqwQ4pfuGR+JnU2Ha/P7USDk02qd1rUCKrUyXsQ 8MvtLD3Ai1HEmlTu96vyH5NdJH9W54w== X-Google-Smtp-Source: AGHT+IGcJ1yJH0fRzYxlDDUqZBhJPeXzC6DrCDd4ckApy9RMyQ4dsnkHLjAoOoLgwzZvQEvUMtYym8FpF7QRiypohqI= X-Received: from pfbbe3.prod.google.com ([2002:a05:6a00:1f03:b0:748:e071:298a]) (user=samitolvanen job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:b95:b0:220:94b1:f1b8 with SMTP id adf61e73a8af0-23d48e6859amr495572637.0.1753217643988; Tue, 22 Jul 2025 13:54:03 -0700 (PDT) Date: Tue, 22 Jul 2025 20:53:59 +0000 In-Reply-To: <20250722205357.3347626-5-samitolvanen@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250722205357.3347626-5-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=5301; i=samitolvanen@google.com; h=from:subject; bh=Dmv7MBdL2t4YlIN+ZApIs9QGn/40RNGjIGDoX876EIY=; b=owGbwMvMwCUWxa662nLh8irG02pJDBn1v9LqNh54s6quo15jb5tZrkxbpfL6Jv22VLWgfzd6F GTfOOzoKGVhEONikBVTZGn5unrr7u9Oqa8+F0nAzGFlAhnCwMUpABOZcoLhn80avWDtm/9ExOK3 JJcwrNkVbJq/Snrm8S09a9+8MG0wZWNk+CabYXP+em13pVXuzycBPey/Km8+XfDejO2F1Ak+b8+ F3AA= X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250722205357.3347626-6-samitolvanen@google.com> Subject: [PATCH bpf-next v13 1/3] cfi: add C CFI type macro From: Sami Tolvanen To: bpf@vger.kernel.org, Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann Cc: Catalin Marinas , Will Deacon , Andrii Nakryiko , Mark Rutland , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Maxwell Bland , Sami Tolvanen , Dao Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Mark Rutland Currently x86 and riscv open-code 4 instances of the same logic to define a u32 variable with the KCFI typeid of a given function. Replace the duplicate logic with a common macro. Signed-off-by: Mark Rutland Co-developed-by: Maxwell Bland Signed-off-by: Maxwell Bland Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Tested-by: Dao Huang Acked-by: Will Deacon --- arch/riscv/kernel/cfi.c | 35 +++-------------------------------- arch/x86/kernel/alternative.c | 31 +++---------------------------- include/linux/cfi_types.h | 23 +++++++++++++++++++++++ 3 files changed, 29 insertions(+), 60 deletions(-) diff --git a/arch/riscv/kernel/cfi.c b/arch/riscv/kernel/cfi.c index 64bdd3e1ab8c..e7aec5f36dd5 100644 --- a/arch/riscv/kernel/cfi.c +++ b/arch/riscv/kernel/cfi.c @@ -4,6 +4,7 @@ * * Copyright (C) 2023 Google LLC */ +#include #include #include =20 @@ -82,41 +83,11 @@ struct bpf_insn; /* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX(const void *ctx, const struct bpf_insn *insn); - -/* - * Force a reference to the external symbol so the compiler generates - * __kcfi_typid. - */ -__ADDRESSABLE(__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash =3D __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .word __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX); =20 /* Must match bpf_callback_t */ extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); - -__ADDRESSABLE(__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash =3D __kcfi_typeid___bpf_callba= ck_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .word __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn); =20 u32 cfi_get_func_hash(void *func) { diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index ea1d984166cd..a555665b4d9c 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -2,6 +2,7 @@ #define pr_fmt(fmt) "SMP alternatives: " fmt =20 #include +#include #include #include #include @@ -1189,37 +1190,11 @@ struct bpf_insn; /* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX(const void *ctx, const struct bpf_insn *insn); - -KCFI_REFERENCE(__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash =3D __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .long __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX); =20 /* Must match bpf_callback_t */ extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); - -KCFI_REFERENCE(__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash =3D __kcfi_typeid___bpf_callba= ck_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .long __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn); =20 u32 cfi_get_func_hash(void *func) { diff --git a/include/linux/cfi_types.h b/include/linux/cfi_types.h index 6b8713675765..685f7181780f 100644 --- a/include/linux/cfi_types.h +++ b/include/linux/cfi_types.h @@ -41,5 +41,28 @@ SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) #endif =20 +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_CFI_CLANG +#define DEFINE_CFI_TYPE(name, func) \ + /* \ + * Force a reference to the function so the compiler generates \ + * __kcfi_typeid_. \ + */ \ + __ADDRESSABLE(func); \ + /* u32 name __ro_after_init =3D __kcfi_typeid_ */ \ + extern u32 name; \ + asm ( \ + " .pushsection .data..ro_after_init,\"aw\",\%progbits \n" \ + " .type " #name ",\%object \n" \ + " .globl " #name " \n" \ + " .p2align 2, 0x0 \n" \ + #name ": \n" \ + " .4byte __kcfi_typeid_" #func " \n" \ + " .size " #name ", 4 \n" \ + " .popsection \n" \ + ); +#endif + #endif /* __ASSEMBLY__ */ #endif /* _LINUX_CFI_TYPES_H */ --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Mon Oct 6 10:18:51 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 548EF285CA8 for ; Tue, 22 Jul 2025 20:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217647; cv=none; b=ItqcyPDcHOI+4MMJ62voKetUYJ6Qzk4gMXPucWVMBJpuUevw+ZlnRBTGdvQdbBvga136j7FE6F9UcMUGc3lf32vDq1ABKuwLUHL+o8nYvZ+GQ5DwqNtUR2Lcyu+KUcfN3i8LSiST8DNMSf6zygh8jNZRiyUSktYQKZcVv0Z3kuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217647; c=relaxed/simple; bh=8sqyVESEJUMNiJl/uCjzhPfl9OVRxgVUJjpXrAl5Bgs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VBI6ipqCsI7Ozh0BVZ1cAKVWwoAoOKslgtmgBRTpYOPfR6virx4lbirwhkjbWTgDJJLKD097VejgskHVBHVbOde2j1ZhHKNahTUuHqHjGZdFvlAbMOsYiB5OV9wk3LLwIz2crZetZuM0+jSP2BhduIMiMdHd3K0vXvitwu8cP2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1hcQwKln; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1hcQwKln" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-748f30d56d1so2580358b3a.3 for ; Tue, 22 Jul 2025 13:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1753217645; x=1753822445; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FfCLGywMG5rCOsx3g823MlhxRQ/vXUvg0nXF69NJcN8=; b=1hcQwKlnOAUYNGCANDyqf4BBt3PsC4yFE7ALYNNNFnRLPluGAvfqtnxtSBIyVelGle I35tVuvJaxchn6fq1/6hZ7E6yz1nlPqE//A+/Q3sH8jnPycR7wwAGmLuv+QthT+Z6DLb 0vm2Aj/zNe5I7/ndvASpmLcv8NBfYhtH1/axp/QI/DBi8fq1ytT74pV8nAZM53qCXJ1U Qb+OFO310GPbQaqXdQtadeCf625aQGY3eTBZTrJTyQ7m44OzqYPFIK9W9XW9tSrDXjQ7 olOmY1fy5eYAStN6oUEEzSiGCacp9Dk1vSbYhhcSYFdDynFiUy14KIAWnE+8wJ8xIxaz 130Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753217645; x=1753822445; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FfCLGywMG5rCOsx3g823MlhxRQ/vXUvg0nXF69NJcN8=; b=DqSexMqSOenp5alaMblTxSgtiQ7vjrcTl2gVlZ6VYZFVcUZt3RgGQu+peCpefwVQzg yXHCMTVm0THblGRyquwXbbbMBHSWm8/JTmgZMKwEZHofbjUVrSMWwm1i4dnjLdXgNG5T Z/AX834QEWoAgTfBpUy95XZykILyD4iOFRFIyUnZqAVN5NaYieZ84Zlf9uf8EXjfQdce 32JlIytq3bZFfhUDzACXTIKVQKCBj4wX00REB0EpxwYYpLU92wYrx2X9AURWzkSvVvwd efJu6Vta8MUMwaeZpNFML5LlPSAxxHdXZNpaa4pnNfvx3sXzUjI8pF63x1STm4Pd8Xe9 mrvQ== X-Forwarded-Encrypted: i=1; AJvYcCXWl5YQy78gOHltRcwfeG60ZeyA5rmoBMhjJtkRZNBSS2vRi3urrtaxUlbaoPyhPRrOogS9V0G6pu7rU5U=@vger.kernel.org X-Gm-Message-State: AOJu0YwXaV+J/gJ+BQYHKIA6Co4mT4GRLFM1+aIFEMvLvKN9k6oMpr/x SYeQaybE+EZQajM+uCIEy6hhlVPIeWdKUbGFOkue7DoHgiHW2d2YN9PsBfF97cbckXKT1k9yLL5 XQluylJ3eXUZsFzfxJ2QjrfavabIXSA== X-Google-Smtp-Source: AGHT+IGj2mYJDItRw3m2soeh8qmNL3HkKmMm41UrOhWIIYH3X5YCpMlu0vttpjV7e4JBGv0PmXpCRcx9gMgupcUb7yo= X-Received: from pfbce15.prod.google.com ([2002:a05:6a00:2a0f:b0:749:937:54c8]) (user=samitolvanen job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:112:b0:234:4b39:182c with SMTP id adf61e73a8af0-23d49143ad9mr423743637.38.1753217645576; Tue, 22 Jul 2025 13:54:05 -0700 (PDT) Date: Tue, 22 Jul 2025 20:54:00 +0000 In-Reply-To: <20250722205357.3347626-5-samitolvanen@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250722205357.3347626-5-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=7400; i=samitolvanen@google.com; h=from:subject; bh=8sqyVESEJUMNiJl/uCjzhPfl9OVRxgVUJjpXrAl5Bgs=; b=owGbwMvMwCUWxa662nLh8irG02pJDBn1v9LubhOs9bG86TVvvUGxxO87f2/x30p4zuy/IZlJf ENdZCxLRykLgxgXg6yYIkvL19Vbd393Sn31uUgCZg4rE8gQBi5OAZjIOkdGhvMK0/mtvrC8dNX8 KlhQzRtbFsHivCigU+/XVQOXY2deKjMyvGANl2vkbW697vV5pemOmT0/rxlq21S3B9gIsd0rK/X hAwA= X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250722205357.3347626-7-samitolvanen@google.com> Subject: [PATCH bpf-next v13 2/3] cfi: Move BPF CFI types and helpers to generic code From: Sami Tolvanen To: bpf@vger.kernel.org, Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann Cc: Catalin Marinas , Will Deacon , Andrii Nakryiko , Mark Rutland , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Maxwell Bland , Sami Tolvanen Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of duplicating the same code for each architecture, move the CFI type hash variables for BPF function types and related helper functions to generic CFI code, and allow architectures to override the function definitions if needed. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/cfi.h | 16 ------------ arch/riscv/kernel/cfi.c | 24 ------------------ arch/x86/include/asm/cfi.h | 10 ++------ arch/x86/kernel/alternative.c | 12 --------- include/linux/cfi.h | 47 +++++++++++++++++++++++++++++------ kernel/cfi.c | 15 +++++++++++ 6 files changed, 56 insertions(+), 68 deletions(-) diff --git a/arch/riscv/include/asm/cfi.h b/arch/riscv/include/asm/cfi.h index fb9696d7a3f2..4508aaa7a2fd 100644 --- a/arch/riscv/include/asm/cfi.h +++ b/arch/riscv/include/asm/cfi.h @@ -14,27 +14,11 @@ struct pt_regs; #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); #define __bpfcall -static inline int cfi_get_offset(void) -{ - return 4; -} - -#define cfi_get_offset cfi_get_offset -extern u32 cfi_bpf_hash; -extern u32 cfi_bpf_subprog_hash; -extern u32 cfi_get_func_hash(void *func); #else static inline enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) { return BUG_TRAP_TYPE_NONE; } - -#define cfi_bpf_hash 0U -#define cfi_bpf_subprog_hash 0U -static inline u32 cfi_get_func_hash(void *func) -{ - return 0; -} #endif /* CONFIG_CFI_CLANG */ =20 #endif /* _ASM_RISCV_CFI_H */ diff --git a/arch/riscv/kernel/cfi.c b/arch/riscv/kernel/cfi.c index e7aec5f36dd5..6ec9dbd7292e 100644 --- a/arch/riscv/kernel/cfi.c +++ b/arch/riscv/kernel/cfi.c @@ -4,7 +4,6 @@ * * Copyright (C) 2023 Google LLC */ -#include #include #include =20 @@ -76,26 +75,3 @@ enum bug_trap_type handle_cfi_failure(struct pt_regs *re= gs) =20 return report_cfi_failure(regs, regs->epc, &target, type); } - -#ifdef CONFIG_CFI_CLANG -struct bpf_insn; - -/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ -extern unsigned int __bpf_prog_runX(const void *ctx, - const struct bpf_insn *insn); -DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX); - -/* Must match bpf_callback_t */ -extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); -DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn); - -u32 cfi_get_func_hash(void *func) -{ - u32 hash; - - if (get_kernel_nofault(hash, func - cfi_get_offset())) - return 0; - - return hash; -} -#endif diff --git a/arch/x86/include/asm/cfi.h b/arch/x86/include/asm/cfi.h index 3e51ba459154..1751f1eb95ef 100644 --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -116,8 +116,6 @@ struct pt_regs; #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); #define __bpfcall -extern u32 cfi_bpf_hash; -extern u32 cfi_bpf_subprog_hash; =20 static inline int cfi_get_offset(void) { @@ -135,6 +133,8 @@ static inline int cfi_get_offset(void) #define cfi_get_offset cfi_get_offset =20 extern u32 cfi_get_func_hash(void *func); +#define cfi_get_func_hash cfi_get_func_hash + extern int cfi_get_func_arity(void *func); =20 #ifdef CONFIG_FINEIBT @@ -153,12 +153,6 @@ static inline enum bug_trap_type handle_cfi_failure(st= ruct pt_regs *regs) { return BUG_TRAP_TYPE_NONE; } -#define cfi_bpf_hash 0U -#define cfi_bpf_subprog_hash 0U -static inline u32 cfi_get_func_hash(void *func) -{ - return 0; -} static inline int cfi_get_func_arity(void *func) { return 0; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index a555665b4d9c..9f6b7dab2d9a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -2,7 +2,6 @@ #define pr_fmt(fmt) "SMP alternatives: " fmt =20 #include -#include #include #include #include @@ -1185,17 +1184,6 @@ bool cfi_bhi __ro_after_init =3D false; #endif =20 #ifdef CONFIG_CFI_CLANG -struct bpf_insn; - -/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ -extern unsigned int __bpf_prog_runX(const void *ctx, - const struct bpf_insn *insn); -DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX); - -/* Must match bpf_callback_t */ -extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); -DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn); - u32 cfi_get_func_hash(void *func) { u32 hash; diff --git a/include/linux/cfi.h b/include/linux/cfi.h index 1db17ecbb86c..52a98886a455 100644 --- a/include/linux/cfi.h +++ b/include/linux/cfi.h @@ -11,16 +11,9 @@ #include #include =20 +#ifdef CONFIG_CFI_CLANG extern bool cfi_warn; =20 -#ifndef cfi_get_offset -static inline int cfi_get_offset(void) -{ - return 0; -} -#endif - -#ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long = addr, unsigned long *target, u32 type); =20 @@ -29,6 +22,44 @@ static inline enum bug_trap_type report_cfi_failure_noad= dr(struct pt_regs *regs, { return report_cfi_failure(regs, addr, NULL, 0); } + +#ifndef cfi_get_offset +/* + * Returns the CFI prefix offset. By default, the compiler emits only + * a 4-byte CFI type hash before the function. If an architecture + * uses -fpatchable-function-entry=3DN,M where M>0 to change the prefix + * offset, they must override this function. + */ +static inline int cfi_get_offset(void) +{ + return 4; +} +#endif + +#ifndef cfi_get_func_hash +static inline u32 cfi_get_func_hash(void *func) +{ + u32 hash; + + if (get_kernel_nofault(hash, func - cfi_get_offset())) + return 0; + + return hash; +} +#endif + +/* CFI type hashes for BPF function types */ +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; + +#else /* CONFIG_CFI_CLANG */ + +static inline int cfi_get_offset(void) { return 0; } +static inline u32 cfi_get_func_hash(void *func) { return 0; } + +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U + #endif /* CONFIG_CFI_CLANG */ =20 #ifdef CONFIG_ARCH_USES_CFI_TRAPS diff --git a/kernel/cfi.c b/kernel/cfi.c index 422fa4f958ae..4dad04ead06c 100644 --- a/kernel/cfi.c +++ b/kernel/cfi.c @@ -5,6 +5,8 @@ * Copyright (C) 2022 Google LLC */ =20 +#include +#include #include =20 bool cfi_warn __ro_after_init =3D IS_ENABLED(CONFIG_CFI_PERMISSIVE); @@ -27,6 +29,19 @@ enum bug_trap_type report_cfi_failure(struct pt_regs *re= gs, unsigned long addr, return BUG_TRAP_TYPE_BUG; } =20 +/* + * Declare two non-existent functions with types that match bpf_func_t and + * bpf_callback_t pointers, and use DEFINE_CFI_TYPE to define type hash + * variables for each function type. The cfi_bpf_* variables are used by + * arch-specific BPF JIT implementations to ensure indirectly callable JIT + * code has matching CFI type hashes. + */ +extern typeof(*(bpf_func_t)0) __bpf_prog_runX; +DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX); + +extern typeof(*(bpf_callback_t)0) __bpf_callback_fn; +DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn); + #ifdef CONFIG_ARCH_USES_CFI_TRAPS static inline unsigned long trap_address(s32 *p) { --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Mon Oct 6 10:18:51 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3F6327FD56 for ; Tue, 22 Jul 2025 20:54:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217649; cv=none; b=HimP0GEkkwo6rfazSt+stErif8d/1n4zcCMQ8cVtLhW4eYzxVMXTwXUgdstzdruLrHj7dgvkSuKlYCFJLLnWbPnkJas4ImkQsGW/ci6ir3UmyQUn0yMR4AKn1kxqUdbBOHJq5fqvIP+ZsfccQ/YMVesWkW+gzKaKgS8mrPw+JkU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753217649; c=relaxed/simple; bh=66hctsnDvcPeo4bcqizkbbYExehSgXT0Gg3yUrHXPfc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OsGB20kCq+tGjeyzFyeH5ELTOnnpKYOTq2cfEWsfdtWdrQnGgc8a6mo4+yi2ilSCzcrFDSK+HiStEWWQb4kAHrMDGYHCCfu8fc2bgNiWPeqVCBJD3OzeEHlyYUHC4vDLzM9KsArZDjWQgShCSjvWRn/voDTnj8zR1s8h4jSPIC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xN5WF5sX; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--samitolvanen.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xN5WF5sX" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-74ea5d9982cso4474162b3a.2 for ; Tue, 22 Jul 2025 13:54:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1753217647; x=1753822447; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=v0xky3Tz++dR8wOWXy0XI+fGsD8WnNLUtoliWtDdioM=; b=xN5WF5sXn6P7AA42S3lTILffuA9N4fySl2C2qry0n7o/Qo/m11eLOOpTKgvG+mL6kL H9WeQhfNjcpnpcHsIx5/ewVB9GQ6HEs8oaU3kqAs/sV/h7NVnphJxuMeKJVnK0znamuA rCUmwq/nvO6BCsVde6aDeIFuOYxXoRGq26TaEsRBXhCLoRuVI1jYZw8N/BTO8ELBpoCn CI/104+7orPlKfoHJomIaQPF6PnzTnYLsFWYsY2Y9YaC5xe8qhCxFzlWx9sEJKwct+fX BSnlJwZCKDMPZ3y+qh78ZuhpGYX4w2kGnwLQ9ysy5vg7to0wqd8hXfPN6V7gBBaXh7kZ 51ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753217647; x=1753822447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v0xky3Tz++dR8wOWXy0XI+fGsD8WnNLUtoliWtDdioM=; b=SnWtkonYkaWmhBpKeQIq8+qZ4h5LzistVxP06pBwWUHsnhypjDjsEwGAsbTFAYg6aw rjab5xXUBtEo5OGBkgfbVvPPT/R80Cwnpf+K+bRfomhF3TZYgSceuHOx6oA4Am6dc6+T rzdk6DzE0RsQo4kEtQB/0QGBgUnSzhh84sYq+AecPLo/A6r26uSYxKtodnXNKTjoPMg9 MpHFhoSPYVCrhCCFKxDReHA1C7UvSzehYDNC+ykxXqhTp0OK6OamiHH7mGQhed2DNImP sLSSUUM4SFRd58Srsw++luvoTBjDIA3CnK4WSOV+/ksY6Lg+O80jgtexOW3Qa8BFot0g fq3Q== X-Forwarded-Encrypted: i=1; AJvYcCWpSU6SqiLzHSa+xWVAb89NgdGzWD3jvawZb870l1PQmSG6XggcVMKTGpDr4eA06ovlkaVKjt5/X2jo2U4=@vger.kernel.org X-Gm-Message-State: AOJu0YzCibJeYnW4YdsjEI5vrwSZHUbFuvMNhID20mFDhiF/GMiGFhxf S+TZC/CxhY4S7X8BfBVx42b27H1zKp3VdGuSoV2hKGmfFcd+JVrTc4dauJlHPNwu1Bh+F3rsZrt uBLNkpeaMCMpc0QRvMtFAXA4+Z6D87A== X-Google-Smtp-Source: AGHT+IH6x4q5kLczvA9H+m286fHAokh7HU92JRe53jNpVYETLy7h6qxVO9bNE0XMECCrfd5TA/Z0QoAyJ5vJf8ucB2k= X-Received: from pfbha4.prod.google.com ([2002:a05:6a00:8504:b0:746:2897:67e3]) (user=samitolvanen job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:1fc3:b0:23d:35f2:4e69 with SMTP id adf61e73a8af0-23d490f6100mr432417637.23.1753217647145; Tue, 22 Jul 2025 13:54:07 -0700 (PDT) Date: Tue, 22 Jul 2025 20:54:01 +0000 In-Reply-To: <20250722205357.3347626-5-samitolvanen@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250722205357.3347626-5-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=4527; i=samitolvanen@google.com; h=from:subject; bh=V2hbK6L5Ngvs6CaLj1XnAp6J1iiarTYTez80LhYwsI4=; b=owGbwMvMwCUWxa662nLh8irG02pJDBn1v9I5n4RtPnZpu4LCIsOEhytmr2wSm7Bl7r4PE5YcN AzVFpxh3VHKwiDGxSArpsjS8nX11t3fnVJffS6SgJnDygQyhIGLUwAmwhzCyHA8ZdFRGXGj7H3m hcF8nw+xrJNYb3aRyVxu/b1fSybV9aYw/Pe4pPziyteDP3gibqY7T+XMu1ZoJl5VMkO3/+uVq9t uHmIHAA== X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250722205357.3347626-8-samitolvanen@google.com> Subject: [PATCH bpf-next v13 3/3] arm64/cfi,bpf: Support kCFI + BPF on arm64 From: Sami Tolvanen To: bpf@vger.kernel.org, Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann Cc: Catalin Marinas , Will Deacon , Andrii Nakryiko , Mark Rutland , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Maxwell Bland , Puranjay Mohan , Sami Tolvanen , Dao Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Puranjay Mohan Currently, bpf_dispatcher_*_func() is marked with `__nocfi` therefore calling BPF programs from this interface doesn't cause CFI warnings. When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated. Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared. Signed-off-by: Puranjay Mohan Co-developed-by: Maxwell Bland Signed-off-by: Maxwell Bland Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Tested-by: Dao Huang Acked-by: Will Deacon --- arch/arm64/include/asm/cfi.h | 7 +++++++ arch/arm64/net/bpf_jit_comp.c | 30 +++++++++++++++++++++++++++--- 2 files changed, 34 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h diff --git a/arch/arm64/include/asm/cfi.h b/arch/arm64/include/asm/cfi.h new file mode 100644 index 000000000000..ab90f0351b7a --- /dev/null +++ b/arch/arm64/include/asm/cfi.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#define __bpfcall + +#endif /* _ASM_ARM64_CFI_H */ diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 89b1b8c248c6..993b5d6e1525 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -106,6 +107,14 @@ static inline void emit(const u32 insn, struct jit_ctx= *ctx) ctx->idx++; } =20 +static inline void emit_u32_data(const u32 data, struct jit_ctx *ctx) +{ + if (ctx->image !=3D NULL && ctx->write) + ctx->image[ctx->idx] =3D data; + + ctx->idx++; +} + static inline void emit_a64_mov_i(const int is64, const int reg, const s32 val, struct jit_ctx *ctx) { @@ -166,6 +175,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *= ctx) emit(insn, ctx); } =20 +static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx) +{ + if (IS_ENABLED(CONFIG_CFI_CLANG)) + emit_u32_data(hash, ctx); +} + /* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address @@ -476,7 +491,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebp= f_from_cbpf) const bool is_main_prog =3D !bpf_is_subprog(prog); const u8 fp =3D bpf2a64[BPF_REG_FP]; const u8 arena_vm_base =3D bpf2a64[ARENA_VM_START]; - const int idx0 =3D ctx->idx; int cur_offset; =20 /* @@ -502,6 +516,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebp= f_from_cbpf) * */ =20 + emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); + const int idx0 =3D ctx->idx; + /* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump @@ -2055,9 +2072,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) jit_data->ro_header =3D ro_header; } =20 - prog->bpf_func =3D (void *)ctx.ro_image; + prog->bpf_func =3D (void *)ctx.ro_image + cfi_get_offset(); prog->jited =3D 1; - prog->jited_len =3D prog_size; + prog->jited_len =3D prog_size - cfi_get_offset(); =20 if (!prog->is_func || extra_pass) { int i; @@ -2426,6 +2443,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, s= truct bpf_tramp_image *im, /* return address locates above FP */ retaddr_off =3D stack_size + 8; =20 + if (flags & BPF_TRAMP_F_INDIRECT) { + /* + * Indirect call for bpf_struct_ops + */ + emit_kcfi(cfi_get_func_hash(func_addr), ctx); + } /* bpf trampoline may be invoked by 3 instruction types: * 1. bl, attached to bpf prog or kernel function via short jump * 2. br, attached to bpf prog or kernel function via long jump @@ -2942,6 +2965,7 @@ void bpf_jit_free(struct bpf_prog *prog) sizeof(jit_data->header->size)); kfree(jit_data); } + prog->bpf_func -=3D cfi_get_offset(); hdr =3D bpf_jit_binary_pack_hdr(prog); bpf_jit_binary_pack_free(hdr, NULL); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); --=20 2.50.0.727.gbf7dc18ff4-goog