From nobody Wed Nov 27 10:32:56 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 506661DFE13 for ; Thu, 10 Oct 2024 19:24:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728588255; cv=none; b=LVS+boU2XnHOnRQUIC2C1COPJwd0NHQYQUTONx8BDDNRFHuwX/MCi29sa/ez4bRNP4VDbpX5UhkARJ94ECBBpeJWLEofK6uzYkU2aceygEDoiTB7qgSELhS6T1Ff25xTbJb4KIgGjzBOraKehz5hwbdHXjtj+97egap85+ORe/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728588255; c=relaxed/simple; bh=zVodnoYgNm1DYBK5hKpuTtqHMKwNmjLuCknrbH33Aow=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RvJgoeXyvbbDghoiqHorefcCHeEf93PdW/xQHwr+cOi2qxQ2T+TpSgfCsmaGM6TNcXNKvyDQARMUb9RGznGsbj9T2hXMOmtwx63gobHZBKuLn21Gd2VKUkc0W03d+eC7E2rCjjrwkTNZkVo07a1AeI8T4skvqYhbaT1Pyi+7NPc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--xur.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=g1ULejyr; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--xur.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="g1ULejyr" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e2d1860a62so24784737b3.0 for ; Thu, 10 Oct 2024 12:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728588252; x=1729193052; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bXvfnp7Z+BLB7t8cyHpKNDqx6J3lf8zmPrAsIDA7Ieo=; b=g1ULejyr20RZg5oj0I5IOugHg7+y4on1aNrHB9qRb4tEIWQ6YvsKZDBzLHcvFXs+n1 ngz5fjpEl/tIDKehqBFevDLb0xJjjx9XVV6cmPuNbvDb41HAfUZ9pvBLtqfKvfwGhhoe rbsL1pHozUmFAEB2I6TfSPb++tOhJ53H5eZM127k1mf/dOW7ku+mH5gfT03x6SBm338d +4qDQVGgwa+Pgh0kEfft+K4CUwzrDcJAGkvxhoYEBvOwAOrbsRxatWDLuFafnAicy4W6 oNW/LVdIqG2gbV/Ix3WZz3e3nMGdybdBAU0lp/l24EjDbQI4WxnHzTUJ2z1JXIW5lnr+ eVAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728588252; x=1729193052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bXvfnp7Z+BLB7t8cyHpKNDqx6J3lf8zmPrAsIDA7Ieo=; b=mK3YQhbdfUBnW6WMNfYCvTiOsnMDdjNfwMjz1EY3b42+Z9k8AgnDHKMrM3s38W+hv4 RpAO18vCIsX36UNk1L30SqdrQO/CisyJBFM38QXqEqYmiwkCXwvHDd2sVIMyIndtcBy0 DWt4FG1ozDlXdJxWurhtWL+kSW1T4yoFs67n/8UUeJT9okKJIX9ZLXkn9rD9EyFDMUIf BLw1aQedgDW3dw4Rm2EDqQviU23ivhcrLytHx1W/OSVi0uhG/ISRSobX7tqAVxrQs3NG 3pkYrF0Fo4yna3XTtobpkDcMYJVuvgSUJvTCvG2k+AIzkp+mD+Q7XAoDnBNahiUjvdJH cTbA== X-Forwarded-Encrypted: i=1; AJvYcCWLisbtk5AfafMBrJqwFkew3Zgd3+uW90SArsMxMBrxP8l5vhuITdBEteCbhhd9+JEulbk+h6lpk7dBHZ0=@vger.kernel.org X-Gm-Message-State: AOJu0Ywmx17nCLpRCq+rB60FEWcZ8r8LWb9sfL4/6oxUmYX9z0QOtCLa oEdnbrRSDKMOWQoFBPH3cFouFlpkZxkQiYm3idvuX1rueJdZ6bUG2itssxqbFnV/Gg== X-Google-Smtp-Source: AGHT+IGMu32Wgj2a9dn+nhr1MFY2d/lT0LJJ8GMl/nvZiUcXkvZk1kPhfz7t2WYIwd1XNqtF7GeWJaE= X-Received: from xur.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:2330]) (user=xur job=sendgmr) by 2002:a25:d084:0:b0:e25:cea9:b0e with SMTP id 3f1490d57ef6-e2919fe8406mr8276.9.1728588252247; Thu, 10 Oct 2024 12:24:12 -0700 (PDT) Date: Thu, 10 Oct 2024 12:23:56 -0700 In-Reply-To: <20241010192400.451187-1-xur@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241010192400.451187-1-xur@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241010192400.451187-5-xur@google.com> Subject: [PATCH v3 4/6] AutoFDO: Enable -ffunction-sections for the AutoFDO build From: Rong Xu To: Alice Ryhl , Andrew Morton , Arnd Bergmann , Bill Wendling , Borislav Petkov , Breno Leitao , Brian Gerst , Dave Hansen , David Li , Han Shen , Heiko Carstens , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Jonathan Corbet , Josh Poimboeuf , Juergen Gross , Justin Stitt , Kees Cook , Masahiro Yamada , "Mike Rapoport (IBM)" , Nathan Chancellor , Nick Desaulniers , Nicolas Schier , "Paul E. McKenney" , Peter Zijlstra , Rong Xu , Sami Tolvanen , Thomas Gleixner , Wei Yang , workflows@vger.kernel.org, Miguel Ojeda , Maksim Panchenko Cc: x86@kernel.org, linux-arch@vger.kernel.org, linux-doc@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, llvm@lists.linux.dev, Sriraman Tallam Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable -ffunction-sections by default for the AutoFDO build. With -ffunction-sections, the compiler places each function in its own section named .text.function_name instead of placing all functions in the .text section. In the AutoFDO build, this allows the linker to utilize profile information to reorganize functions for improved utilization of iCache and iTLB. Co-developed-by: Han Shen Signed-off-by: Han Shen Signed-off-by: Rong Xu Suggested-by: Sriraman Tallam --- include/asm-generic/vmlinux.lds.h | 37 ++++++++++++++++++++++++------- scripts/Makefile.autofdo | 2 +- 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 5df589c60401..ace617d1af9b 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -95,18 +95,25 @@ * With LTO_CLANG, the linker also splits sections by default, so we need * these macros to combine the sections during the final link. * + * With LTO_CLANG, the linker also splits sections by default, so we need + * these macros to combine the sections during the final link. + * * RODATA_MAIN is not used because existing code already defines .rodata.x * sections to be brought in with rodata. */ -#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CL= ANG) +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CL= ANG) || \ +defined(CONFIG_AUTOFDO_CLANG) #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* +#else +#define TEXT_MAIN .text +#endif +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CL= ANG) #define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliter= al* .data.$__unnamed_* .data.$L* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L* #define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..L* .bss..compoundliteral* #define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]* #else -#define TEXT_MAIN .text #define DATA_MAIN .data #define SDATA_MAIN .sdata #define RODATA_MAIN .rodata @@ -549,6 +556,20 @@ __cpuidle_text_end =3D .; \ __noinstr_text_end =3D .; =20 +#ifdef CONFIG_AUTOFDO_CLANG +#define TEXT_HOT \ + __hot_text_start =3D .; \ + *(.text.hot .text.hot.*) \ + __hot_text_end =3D .; +#define TEXT_UNLIKELY \ + __unlikely_text_start =3D .; \ + *(.text.unlikely .text.unlikely.*) \ + __unlikely_text_end =3D .; +#else +#define TEXT_HOT *(.text.hot .text.hot.*) +#define TEXT_UNLIKELY *(.text.unlikely .text.unlikely.*) +#endif + /* * .text section. Map to function alignment to avoid address changes * during second ld run in second ld pass when generating System.map @@ -557,30 +578,30 @@ * code elimination or function-section is enabled. Match these symbols * first when in these builds. */ -#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CL= ANG) +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CL= ANG) || \ +defined(CONFIG_AUTOFDO_CLANG) #define TEXT_TEXT \ ALIGN_FUNCTION(); \ *(.text.asan.* .text.tsan.*) \ *(.text.unknown .text.unknown.*) \ - *(.text.unlikely .text.unlikely.*) \ + TEXT_UNLIKELY \ . =3D ALIGN(PAGE_SIZE); \ - *(.text.hot .text.hot.*) \ + TEXT_HOT \ *(TEXT_MAIN .text.fixup) \ NOINSTR_TEXT \ *(.ref.text) #else #define TEXT_TEXT \ ALIGN_FUNCTION(); \ - *(.text.hot .text.hot.*) \ + TEXT_HOT \ *(TEXT_MAIN .text.fixup) \ - *(.text.unlikely .text.unlikely.*) \ + TEXT_UNLIKELY \ *(.text.unknown .text.unknown.*) \ NOINSTR_TEXT \ *(.ref.text) \ *(.text.asan.* .text.tsan.*) #endif =20 - /* sched.text is aling to function alignment to secure we have same * address even at second ld pass when generating System.map */ #define SCHED_TEXT \ diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo index 1c9f224bc221..9c9a530ef090 100644 --- a/scripts/Makefile.autofdo +++ b/scripts/Makefile.autofdo @@ -10,7 +10,7 @@ ifndef CONFIG_DEBUG_INFO endif =20 ifdef CLANG_AUTOFDO_PROFILE - CFLAGS_AUTOFDO_CLANG +=3D -fprofile-sample-use=3D$(CLANG_AUTOFDO_PROFILE) + CFLAGS_AUTOFDO_CLANG +=3D -fprofile-sample-use=3D$(CLANG_AUTOFDO_PROFILE= ) -ffunction-sections endif =20 ifdef CONFIG_LTO_CLANG_THIN --=20 2.47.0.rc1.288.g06298d1525-goog