From nobody Mon Apr 6 06:08:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51343ECAAA1 for ; Fri, 9 Sep 2022 16:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231821AbiIIQsF (ORCPT ); Fri, 9 Sep 2022 12:48:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231709AbiIIQr0 (ORCPT ); Fri, 9 Sep 2022 12:47:26 -0400 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38D11146D02; Fri, 9 Sep 2022 09:47:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pm6Hrbz511lxE9lupJKyu8U0WZdYMEmW+tSf5Tm1RKYvM8LlOC9XRU1x0tCyEQDFq50xmmV6Vg89lPqQgqTC9xUVPnR1RpcqF4LFdvOWA7sCOCtl6MF6P22Nf3QPsjyvUsYxaa0FLIuWxnM1r40lbsR6kk1y5yM5lZG8oWUezITd/aSAbWZ1bWRiFAtrVhOhtG2D6y0PVCL1sL4K/n6XC3jLZQmkBfXAnBRnoWPdJyUbq+GuHK7XJLHQBox1hcsr7foDyp8KVaSwaXz5y6yaz+eVETtygZvQ4hc2SJL8ofZJ+QwJ2DDmPc7rZ0djYc5aFMZVvgewNvOAgiKGX2AR1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Mwba5O6u2AqiwszJ7wr2LfLQjiVhEK0hCMWXy3TYOmM=; b=Gho8SYBO0ymD01advwDMbwqbdjmQ87g6tQ8MHtB+mEpbcW4rnv9NQYnf27jaVorcAUBoNwacpFWEvsPtHstYFXVvh7lLKUbH40xRb7jcNnhA7rprm3AobAYPWUmZgKElHrDTT13N0XGs60Jx2IBurdwXP/6RerfQVFoXdTxuCMPSx922dF0SKnU2fJP2UyoO3cNGu/TgJAZGfEU7YeRE3PKUed4t+j6JJMKBwilfhslzHKe3GXrpufSO5Bu4UEpDIHY6jQK7ceEIMcWuJGEX1ltvz0uTe29pRU6uDjgRthsDKBOa7SqS8VSjfhsgGS5gW1ei5d0d/herFZDopaDxow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=intel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mwba5O6u2AqiwszJ7wr2LfLQjiVhEK0hCMWXy3TYOmM=; b=Vyc7EvjWdazVE0AYMV61mDaKWaez1R0xaqBFdAt8JrIupSlCyzZY5fAzMtkgsQ9PozXaTIiL2MBhf88cLhUN53XUfA2MwTZU3yG2pxsOoOibPeKkHmPJbp4ezC/+TLoVf5Zn99qn3iAWnYmk7bXuxgOf/QMsp4HvqIvujOcm774= Received: from BN0PR02CA0020.namprd02.prod.outlook.com (2603:10b6:408:e4::25) by SJ1PR12MB6291.namprd12.prod.outlook.com (2603:10b6:a03:456::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5588.12; Fri, 9 Sep 2022 16:47:17 +0000 Received: from BN8NAM11FT091.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e4:cafe::83) by BN0PR02CA0020.outlook.office365.com (2603:10b6:408:e4::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.12 via Frontend Transport; Fri, 9 Sep 2022 16:47:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT091.mail.protection.outlook.com (10.13.176.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5612.13 via Frontend Transport; Fri, 9 Sep 2022 16:47:16 +0000 Received: from pyuan-Cloudripper.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 9 Sep 2022 11:47:10 -0500 From: Perry Yuan To: , , CC: , , , , , , , , , , Perry Yuan Subject: [PATCH 5/7] cpufreq: amd_pstate: add AMD Pstate EPP support for the MSR based processors Date: Sat, 10 Sep 2022 00:45:32 +0800 Message-ID: <20220909164534.71864-6-Perry.Yuan@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220909164534.71864-1-Perry.Yuan@amd.com> References: <20220909164534.71864-1-Perry.Yuan@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT091:EE_|SJ1PR12MB6291:EE_ X-MS-Office365-Filtering-Correlation-Id: ea909c96-7ed9-4180-043d-08da9282f44a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: od7KROi2Fv6Qzy5f1bQtjCHA37cRCkOgNy+v/2DlWi5du+6t5vInIeDK4NLSK8YnMLkPLmwnUHr4yL1oxhcTEUC6bnidxNFC6+8YVFzPn3sRLcAIh3hT2eNEy4jdIc6PUTyLVLwfCpguO3HoZh7bWFPDTbapXqNpSYHjgybIs7PXGg5Mk1KXva9G0H+kjQqRQvZE3GyCFEHIids5OO3lur42gNNacsp/Gvo8X1GrWcovB0HpIX+juvgYETkWoHvOhuSTnyW+cPvPfq0Aq78dV5Yi3UpmVjzwTIhNuprlMolBYZ+YsHIy/2a4gFPsWr5HdXrA02oFEuRGKat6SxSZjxZycUJcH+lB1EWaLi/cpjG6UwUUfKkc4zFoA2JZvbIJWFEFaN8INpMRIMLfecDMrxJq+z/JGkJVrZmYx6ioUU8rKKfPDy6GSMaP9YGWoCDxS32hzTB49/8MLBmSgAIlkzVZeuQ1a6z8tdu2lCwUTRiLw9mLna9chuwEnciE8E4T6mN1ifyXEWG2Q55L7DUBYe0iK59oEC8gcmeSO2APNAGZY2KRTlucy6Hf8ORwum73A9P0/7I+/H6HDDLzLICmC76fDLKn/eLEuXOpe1qUDonVrKSiyzvcQBTuh6mnB5EzY08zNQvhQxkcnJFWc34qiIZqu358PUsU5jKDcHyOLsbyVrE5beChZtgE8GukKphy4cJSLwB+fiU8AsXFCDzFO715ioH/BTZSqiEIn1xuXTb4055HrRQzpr8BWyYJwEC+fW1egB9PvRqJ7CKIVnPBvoYdUn/9tjsVhrmnICiq+HO/EM45/JPDjxsPIa3X2tHBYH45/hBt//6TP6f/FbRw967BWrMkbWf0pHnx2+lCTbU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230016)(4636009)(346002)(376002)(136003)(396003)(39860400002)(40470700004)(46966006)(36840700001)(83380400001)(1076003)(356005)(81166007)(36756003)(16526019)(86362001)(6666004)(47076005)(26005)(336012)(186003)(2616005)(7696005)(41300700001)(426003)(40480700001)(82310400005)(316002)(54906003)(4326008)(70206006)(110136005)(8676002)(478600001)(8936002)(30864003)(70586007)(2906002)(5660300002)(36860700001)(82740400003)(40460700003)(36900700001)(2101003)(309714004);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2022 16:47:16.7304 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ea909c96-7ed9-4180-043d-08da9282f44a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT091.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6291 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add EPP driver support for those AMD CPUs which has full MSR feature enabled, The EPP is used in the DPM controller to drive the frequency that a core is going to operate during short periods of activity. EPP values will be utilized for different OS profiles (balanced, performanc= e, power savings). cppc performance can be controlled by the user space interf= ace sys attributes for min and max frequency limits, when pstate driver is working under power save policy. EPP scale is 0 - 255, 0 is the max performance and 255 is min level. balance_performance (0x80) can provide best balance performance and watt for most of system, meanwhile user can choose performance policy on needs. $ cat /sys/devices/system/cpu/cpufreq/policy0/energy_performance_available_= preferences default performance balance_performance balance_power power $ cat /sys/devices/system/cpu/cpufreq/policy0/energy_performance_preference balance_performance Signed-off-by: Perry Yuan --- arch/x86/include/asm/msr-index.h | 4 + drivers/cpufreq/amd-pstate.c | 818 ++++++++++++++++++++++++++++++- 2 files changed, 804 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-in= dex.h index 43a3d8e4eb9a..4c540badab4e 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -577,6 +577,10 @@ #define MSR_AMD64_PERF_CNTR_GLOBAL_CTL 0xc0000301 #define MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR 0xc0000302 =20 +#define AMD_CPPC_EPP_PERFORMANCE 0x00 +#define AMD_CPPC_EPP_BALANCE_PERFORMANCE 0x80 +#define AMD_CPPC_EPP_BALANCE_POWERSAVE 0xBF +#define AMD_CPPC_EPP_POWERSAVE 0xFF /* Fam 17h MSRs */ #define MSR_F17H_IRPERF 0xc00000e9 =20 diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index fff298744a8e..e71b06e20050 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -63,8 +63,8 @@ module_param(epp_enabled, bool, 0444); MODULE_PARM_DESC(epp_enabled, "load amd_pstate or amd_pstate_epp (true =3D amd_pstate_ep= p driver instance (default), false =3D amd_pstate driver instance)"); =20 - -static struct cpufreq_driver amd_pstate_driver; +static struct cpufreq_driver *default_pstate_driver; +static struct amd_cpudata **all_cpu_data; =20 /** * struct amd_aperf_mperf @@ -76,6 +76,7 @@ struct amd_aperf_mperf { u64 aperf; u64 mperf; u64 tsc; + u64 time; }; =20 /** @@ -98,7 +99,19 @@ struct amd_aperf_mperf { * @prev: Last Aperf/Mperf/tsc count value read from register * @freq: current cpu frequency value * @boost_supported: check whether the Processor or SBIOS supports boost m= ode - * + * @epp_powersave: Last saved CPPC energy performance preference + when policy switched to performance + * @epp_policy: Last saved policy used to set energy-performance preference + * @epp_cached: Cached CPPC energy-performance preference value + * @policy: Cpufreq policy value + * @sched_flags: Store scheduler flags for possible cross CPU update + * @update_util_set: CPUFreq utility callback is set + * @last_update: Time stamp of the last performance state update + * @cppc_boost_min: Last CPPC boosted min performance state + * @cppc_cap1_cached: Cached value of the last CPPC Capabilities MSR + * @update_util: Cpufreq utility callback information + * @sample: the stored performance sample + * The amd_cpudata is key private data for each CPU thread in AMD P-State,= and * represents all the attributes and goals that AMD P-State requests at ru= ntime. */ @@ -124,8 +137,195 @@ struct amd_cpudata { u64 freq; bool boost_supported; u64 cppc_hw_conf_cached; + + /* EPP feature related attributes*/ + s16 epp_powersave; + s16 epp_policy; + s16 epp_cached; + u32 policy; + u32 sched_flags; + bool update_util_set; + u64 last_update; + u64 last_io_update; + u32 cppc_boost_min; + u64 cppc_cap1_cached; + struct update_util_data update_util; + struct amd_aperf_mperf sample; +}; + +/** + * struct amd_pstate_params - global parameters for the performance control + * @ cppc_boost_disabled Wheter or not the core performance boost disabled + */ +struct amd_pstate_params { + bool cppc_boost_disabled; +}; + +/* + * AMD Energy Preference Performance (EPP) + * The EPP is used in the CCLK DPM controller to drive + * the frequency that a core is going to operate during + * short periods of activity. EPP values will be utilized for + * different OS profiles (balanced, performance, power savings) + * display strings corresponding to EPP index in the + * energy_perf_strings[] + * index String + *------------------------------------- + * 0 default + * 1 performance + * 2 balance_performance + * 3 balance_power + * 4 power + */ + enum energy_perf_value_index { + EPP_INDEX_DEFAULT =3D 0, + EPP_INDEX_PERFORMANCE, + EPP_INDEX_BALANCE_PERFORMANCE, + EPP_INDEX_BALANCE_POWERSAVE, + EPP_INDEX_POWERSAVE, +}; + +static const char * const energy_perf_strings[] =3D { + [EPP_INDEX_DEFAULT] =3D "default", + [EPP_INDEX_PERFORMANCE] =3D "performance", + [EPP_INDEX_BALANCE_PERFORMANCE] =3D "balance_performance", + [EPP_INDEX_BALANCE_POWERSAVE] =3D "balance_power", + [EPP_INDEX_POWERSAVE] =3D "power", + NULL }; =20 +static unsigned int epp_values[] =3D { + [EPP_INDEX_DEFAULT] =3D 0, + [EPP_INDEX_PERFORMANCE] =3D AMD_CPPC_EPP_PERFORMANCE, + [EPP_INDEX_BALANCE_PERFORMANCE] =3D AMD_CPPC_EPP_BALANCE_PERFORMANCE, + [EPP_INDEX_BALANCE_POWERSAVE] =3D AMD_CPPC_EPP_BALANCE_POWERSAVE, + [EPP_INDEX_POWERSAVE] =3D AMD_CPPC_EPP_POWERSAVE, +}; + +static struct amd_pstate_params global; + +static DEFINE_MUTEX(amd_pstate_limits_lock); +static DEFINE_MUTEX(amd_pstate_driver_lock); +static DEFINE_SPINLOCK(amd_pstate_cpu_lock); + +static bool cppc_boost __read_mostly; +struct kobject *amd_pstate_kobj; + +#ifdef CONFIG_ACPI_CPPC_LIB +static s16 amd_pstate_get_epp(struct amd_cpudata *cpudata, u64 cppc_req_ca= ched) +{ + s16 epp; + struct cppc_perf_caps perf_caps; + int ret; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + if (!cppc_req_cached) { + epp =3D rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, + &cppc_req_cached); + if (epp) + return epp; + } + epp =3D (cppc_req_cached >> 24) & 0xFF; + } else { + ret =3D cppc_get_epp_caps(cpudata->cpu, &perf_caps); + if (ret < 0) { + pr_debug("Could not retrieve energy perf value (%d)\n", ret); + return -EIO; + } + epp =3D (s16) perf_caps.energy_perf; + } + + return epp; +} +#endif + +static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata, i= nt *raw_epp) +{ + s16 epp; + int index =3D -EINVAL; + + *raw_epp =3D 0; + epp =3D amd_pstate_get_epp(cpudata, 0); + if (epp < 0) + return epp; + + switch (epp) { + case AMD_CPPC_EPP_PERFORMANCE: + index =3D EPP_INDEX_PERFORMANCE; + break; + case AMD_CPPC_EPP_BALANCE_PERFORMANCE: + index =3D EPP_INDEX_BALANCE_PERFORMANCE; + break; + case AMD_CPPC_EPP_BALANCE_POWERSAVE: + index =3D EPP_INDEX_BALANCE_POWERSAVE; + break; + case AMD_CPPC_EPP_POWERSAVE: + index =3D EPP_INDEX_POWERSAVE; + break; + default: + *raw_epp =3D epp; + index =3D 0; + } + + return index; +} + +#ifdef CONFIG_ACPI_CPPC_LIB +static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp) +{ + int ret; + struct cppc_perf_ctrls perf_ctrls; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + u64 value =3D READ_ONCE(cpudata->cppc_req_cached); + + value &=3D ~GENMASK_ULL(31, 24); + value |=3D (u64)epp << 24; + WRITE_ONCE(cpudata->cppc_req_cached, value); + ret =3D wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value); + if (!ret) { + cpudata->epp_cached =3D epp; + } + } else { + perf_ctrls.energy_perf =3D epp; + ret =3D cppc_set_epp_perf(cpudata->cpu, &perf_ctrls); + if (ret){ + pr_debug("failed to set energy perf value (%d)\n", ret); + return ret; + } + cpudata->epp_cached =3D epp; + } + return ret; +} + +static int amd_pstate_set_energy_pref_index(struct amd_cpudata *cpudata, + int pref_index, bool use_raw, + u32 raw_epp) +{ + int epp =3D -EINVAL; + int ret; + + if (!pref_index) { + pr_debug("EPP pref_index is invaid\n"); + return -EINVAL; + } + + if (use_raw) + epp =3D raw_epp; + else if (epp =3D=3D -EINVAL) + epp =3D epp_values[pref_index]; + + if (epp > 0 && cpudata->policy =3D=3D CPUFREQ_POLICY_PERFORMANCE) { + pr_debug("EPP cannot be set under performance policy\n"); + return -EBUSY; + } + + ret =3D amd_pstate_set_epp(cpudata, epp); + + return ret; +} +#endif + static inline int pstate_enable(bool enable) { return wrmsrl_safe(MSR_AMD_CPPC_ENABLE, enable); @@ -141,16 +341,18 @@ static int cppc_enable(bool enable) if (ret) return ret; =20 - /* Enable active mode for EPP */ - ret =3D cppc_set_auto_epp(cpu, enable); - if (ret) - return ret; - - /* Set zero to desired perf to enable EPP control*/ - perf_ctrls.desired_perf =3D 0; - ret =3D cppc_set_perf(cpu, &perf_ctrls); - if (ret) - return ret; + if (epp_enabled) { + /* Enable autonomous mode for EPP */ + ret =3D cppc_set_auto_epp(cpu, enable); + if (ret) + return ret; + + /* Set zero to desired perf to allow EPP firmware control*/ + perf_ctrls.desired_perf =3D 0; + ret =3D cppc_set_perf(cpu, &perf_ctrls); + if (ret) + return ret; + } } =20 return ret; @@ -496,7 +698,7 @@ static void amd_pstate_boost_init(struct amd_cpudata *c= pudata) return; =20 cpudata->boost_supported =3D true; - amd_pstate_driver.boost_enabled =3D true; + default_pstate_driver->boost_enabled =3D true; } =20 static int amd_pstate_cpu_init(struct cpufreq_policy *policy) @@ -671,10 +873,108 @@ static ssize_t show_amd_pstate_highest_perf(struct c= pufreq_policy *policy, return sprintf(&buf[0], "%u\n", perf); } =20 +static ssize_t show_energy_performance_available_preferences( + struct cpufreq_policy *policy, char *buf) +{ + int i =3D 0; + int ret =3D 0; + + while (energy_perf_strings[i] !=3D NULL) + ret +=3D sprintf(&buf[ret], "%s ", energy_perf_strings[i++]); + + ret +=3D sprintf(&buf[ret], "\n"); + + return ret; +} + +static ssize_t store_energy_performance_preference( + struct cpufreq_policy *policy, const char *buf, size_t count) +{ + struct amd_cpudata *cpudata =3D policy->driver_data; + char str_preference[21]; + bool raw =3D false; + ssize_t ret; + u32 epp =3D 0; + + ret =3D sscanf(buf, "%20s", str_preference); + if (ret !=3D 1) + return -EINVAL; + + ret =3D match_string(energy_perf_strings, -1, str_preference); + if (ret < 0) { + ret =3D kstrtouint(buf, 10, &epp); + if (ret) + return ret; + + if ((epp > 255) || (epp < 0)) + return -EINVAL; + + raw =3D true; + } + + mutex_lock(&amd_pstate_limits_lock); + ret =3D amd_pstate_set_energy_pref_index(cpudata, ret, raw, epp); + mutex_unlock(&amd_pstate_limits_lock); + + return ret ?: count; +} + +static ssize_t show_energy_performance_preference( + struct cpufreq_policy *policy, char *buf) +{ + struct amd_cpudata *cpudata =3D policy->driver_data; + int preference, raw_epp; + + preference =3D amd_pstate_get_energy_pref_index(cpudata, &raw_epp); + if (preference < 0) + return preference; + + if (raw_epp) + return sprintf(buf, "%d\n", raw_epp); + else + return sprintf(buf, "%s\n", energy_perf_strings[preference]); +} + +static void amd_pstate_update_policies(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + cpufreq_update_policy(cpu); +} + +static ssize_t show_pstate_dynamic_boost(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sprintf(buf, "%u\n", cppc_boost); +} + +static ssize_t store_pstate_dynamic_boost(struct kobject *a, + struct kobj_attribute *b, + const char *buf, size_t count) +{ + unsigned int input; + int ret; + + ret =3D kstrtouint(buf, 10, &input); + if (ret) + return ret; + + mutex_lock(&amd_pstate_driver_lock); + cppc_boost =3D !!input; + amd_pstate_update_policies(); + mutex_unlock(&amd_pstate_driver_lock); + + return count; +} + cpufreq_freq_attr_ro(amd_pstate_max_freq); cpufreq_freq_attr_ro(amd_pstate_lowest_nonlinear_freq); =20 cpufreq_freq_attr_ro(amd_pstate_highest_perf); +cpufreq_freq_attr_rw(energy_performance_preference); +cpufreq_freq_attr_ro(energy_performance_available_preferences); +define_one_global_rw(pstate_dynamic_boost); =20 static struct freq_attr *amd_pstate_attr[] =3D { &amd_pstate_max_freq, @@ -683,6 +983,424 @@ static struct freq_attr *amd_pstate_attr[] =3D { NULL, }; =20 +static struct freq_attr *amd_pstate_epp_attr[] =3D { + &amd_pstate_max_freq, + &amd_pstate_lowest_nonlinear_freq, + &amd_pstate_highest_perf, + &energy_performance_preference, + &energy_performance_available_preferences, + NULL, +}; + +static struct attribute *pstate_global_attributes[] =3D { + &pstate_dynamic_boost.attr, + NULL +}; + +static const struct attribute_group amd_pstate_global_attr_group =3D { + .attrs =3D pstate_global_attributes, +}; + +static inline void update_boost_state(void) +{ + u64 misc_en; + struct amd_cpudata *cpudata; + + cpudata =3D all_cpu_data[0]; + rdmsrl(MSR_AMD_CPPC_HW_CTL, misc_en); + global.cppc_boost_disabled =3D misc_en & AMD_CPPC_PRECISION_BOOST_ENABLED; +} + +static int amd_pstate_init_cpu(unsigned int cpunum) +{ + struct amd_cpudata *cpudata; + + cpudata =3D all_cpu_data[cpunum]; + if (!cpudata) { + cpudata =3D kzalloc(sizeof(*cpudata), GFP_KERNEL); + if (!cpudata) + return -ENOMEM; + WRITE_ONCE(all_cpu_data[cpunum], cpudata); + + cpudata->cpu =3D cpunum; + } + cpudata->epp_powersave =3D -EINVAL; + cpudata->epp_policy =3D 0; + pr_debug("controlling: cpu %d\n", cpunum); + return 0; +} + +static int __amd_pstate_cpu_init(struct cpufreq_policy *policy) +{ + int min_freq, max_freq, nominal_freq, lowest_nonlinear_freq, ret; + struct amd_cpudata *cpudata; + struct device *dev; + int rc; + u64 value; + + rc =3D amd_pstate_init_cpu(policy->cpu); + if (rc) + return rc; + + cpudata =3D all_cpu_data[policy->cpu]; + + dev =3D get_cpu_device(policy->cpu); + if (!dev) + goto free_cpudata1; + + rc =3D amd_pstate_init_perf(cpudata); + if (rc) + goto free_cpudata1; + + min_freq =3D amd_get_min_freq(cpudata); + max_freq =3D amd_get_max_freq(cpudata); + nominal_freq =3D amd_get_nominal_freq(cpudata); + lowest_nonlinear_freq =3D amd_get_lowest_nonlinear_freq(cpudata); + if (min_freq < 0 || max_freq < 0 || min_freq > max_freq) { + dev_err(dev, "min_freq(%d) or max_freq(%d) value is incorrect\n", + min_freq, max_freq); + ret =3D -EINVAL; + goto free_cpudata1; + } + + policy->min =3D min_freq; + policy->max =3D max_freq; + + policy->cpuinfo.min_freq =3D min_freq; + policy->cpuinfo.max_freq =3D max_freq; + /* It will be updated by governor */ + policy->cur =3D policy->cpuinfo.min_freq; + + /* Initial processor data capability frequencies */ + cpudata->max_freq =3D max_freq; + cpudata->min_freq =3D min_freq; + cpudata->nominal_freq =3D nominal_freq; + cpudata->lowest_nonlinear_freq =3D lowest_nonlinear_freq; + + policy->driver_data =3D cpudata; + + update_boost_state(); + cpudata->epp_cached =3D amd_pstate_get_epp(cpudata, value); + + policy->min =3D policy->cpuinfo.min_freq; + policy->max =3D policy->cpuinfo.max_freq; + + if (boot_cpu_has(X86_FEATURE_CPPC)) + policy->fast_switch_possible =3D true; + + if (!shared_mem && boot_cpu_has(X86_FEATURE_CPPC)) { + ret =3D rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, &value); + if (ret) + return ret; + WRITE_ONCE(cpudata->cppc_req_cached, value); + + ret =3D rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_CAP1, &value); + if (ret) + return ret; + WRITE_ONCE(cpudata->cppc_cap1_cached, value); + } + amd_pstate_boost_init(cpudata); + + return 0; + +free_cpudata1: + kfree(cpudata); + return ret; +} + +static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) +{ + int ret; + + ret =3D __amd_pstate_cpu_init(policy); + if (ret) + return ret; + /* + * Set the policy to powersave to provide a valid fallback value in case + * the default cpufreq governor is neither powersave nor performance. + */ + policy->policy =3D CPUFREQ_POLICY_POWERSAVE; + + return 0; +} + +static int amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) +{ + pr_debug("amd-pstate: CPU %d exiting\n", policy->cpu); + policy->fast_switch_possible =3D false; + return 0; +} + +static void amd_pstate_update_max_freq(unsigned int cpu) +{ + struct cpufreq_policy *policy =3D cpufreq_cpu_acquire(cpu); + + if (!policy) + return; + + refresh_frequency_limits(policy); + cpufreq_cpu_release(policy); +} + +static void amd_pstate_epp_update_limits(unsigned int cpu) +{ + mutex_lock(&amd_pstate_driver_lock); + update_boost_state(); + if (global.cppc_boost_disabled) { + for_each_possible_cpu(cpu) + amd_pstate_update_max_freq(cpu); + } else { + cpufreq_update_policy(cpu); + } + mutex_unlock(&amd_pstate_driver_lock); +} + +static int cppc_boost_hold_time_ns =3D 3 * NSEC_PER_MSEC; + +static inline void amd_pstate_boost_up(struct amd_cpudata *cpudata) +{ + u64 hwp_req =3D READ_ONCE(cpudata->cppc_req_cached); + u64 hwp_cap =3D READ_ONCE(cpudata->cppc_cap1_cached); + u32 max_limit =3D (hwp_req & 0xff); + u32 min_limit =3D (hwp_req & 0xff00) >> 8; + u32 boost_level1; + + /* If max and min are equal or already at max, nothing to boost */ + if (max_limit =3D=3D min_limit) + return; + + /* Set boost max and min to initial value */ + if (!cpudata->cppc_boost_min) + cpudata->cppc_boost_min =3D min_limit; + + boost_level1 =3D ((AMD_CPPC_NOMINAL_PERF(hwp_cap) + min_limit) >> 1); + + if (cpudata->cppc_boost_min < boost_level1) + cpudata->cppc_boost_min =3D boost_level1; + else if (cpudata->cppc_boost_min < AMD_CPPC_NOMINAL_PERF(hwp_cap)) + cpudata->cppc_boost_min =3D AMD_CPPC_NOMINAL_PERF(hwp_cap); + else if (cpudata->cppc_boost_min =3D=3D AMD_CPPC_NOMINAL_PERF(hwp_cap)) + cpudata->cppc_boost_min =3D max_limit; + else + return; + + hwp_req &=3D ~AMD_CPPC_MIN_PERF(~0L); + hwp_req |=3D AMD_CPPC_MIN_PERF(cpudata->cppc_boost_min); + wrmsrl_safe_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, hwp_req); + cpudata->last_update =3D cpudata->sample.time; +} + +static inline void amd_pstate_boost_down(struct amd_cpudata *cpudata) +{ + bool expired; + + if (cpudata->cppc_boost_min) { + expired =3D time_after64(cpudata->sample.time, cpudata->last_update + + cppc_boost_hold_time_ns); + + if (expired) { + wrmsrl_safe_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, cpudata->cppc_req_ca= ched); + cpudata->cppc_boost_min =3D 0; + } + } + + cpudata->last_update =3D cpudata->sample.time; +} + +static inline void amd_pstate_boost_update_util(struct amd_cpudata *cpudat= a, + u64 time) +{ + cpudata->sample.time =3D time; + if (smp_processor_id() !=3D cpudata->cpu) + return; + + if (cpudata->sched_flags & SCHED_CPUFREQ_IOWAIT) { + bool do_io =3D false; + + cpudata->sched_flags =3D 0; + /* + * Set iowait_boost flag and update time. Since IO WAIT flag + * is set all the time, we can't just conclude that there is + * some IO bound activity is scheduled on this CPU with just + * one occurrence. If we receive at least two in two + * consecutive ticks, then we treat as boost candidate. + * This is leveraged from Intel Pstate driver. + */ + if (time_before64(time, cpudata->last_io_update + 2 * TICK_NSEC)) + do_io =3D true; + + cpudata->last_io_update =3D time; + + if (do_io) + amd_pstate_boost_up(cpudata); + + } else { + amd_pstate_boost_down(cpudata); + } +} + +static inline void amd_pstate_cppc_update_hook(struct update_util_data *da= ta, + u64 time, unsigned int flags) +{ + struct amd_cpudata *cpudata =3D container_of(data, + struct amd_cpudata, update_util); + + cpudata->sched_flags |=3D flags; + + if (smp_processor_id() =3D=3D cpudata->cpu) + amd_pstate_boost_update_util(cpudata, time); +} + +static void amd_pstate_clear_update_util_hook(unsigned int cpu) +{ + struct amd_cpudata *cpudata =3D all_cpu_data[cpu]; + + if (!cpudata->update_util_set) + return; + + cpufreq_remove_update_util_hook(cpu); + cpudata->update_util_set =3D false; + synchronize_rcu(); +} + +static void amd_pstate_set_update_util_hook(unsigned int cpu_num) +{ + struct amd_cpudata *cpudata =3D all_cpu_data[cpu_num]; + + if (!cppc_boost) { + if (cpudata->update_util_set) + amd_pstate_clear_update_util_hook(cpudata->cpu); + return; + } + + if (cpudata->update_util_set) + return; + + cpudata->sample.time =3D 0; + cpufreq_add_update_util_hook(cpu_num, &cpudata->update_util, + amd_pstate_cppc_update_hook); + cpudata->update_util_set =3D true; +} + +static void amd_pstate_epp_init(unsigned int cpu) +{ + struct amd_cpudata *cpudata =3D all_cpu_data[cpu]; + u32 max_perf, min_perf; + u64 value; + s16 epp; + int ret; + + max_perf =3D READ_ONCE(cpudata->highest_perf); + min_perf =3D READ_ONCE(cpudata->lowest_perf); + + value =3D READ_ONCE(cpudata->cppc_req_cached); + + if (cpudata->policy =3D=3D CPUFREQ_POLICY_PERFORMANCE) + min_perf =3D max_perf; + + /* Initial min/max values for CPPC Performance Controls Register */ + value &=3D ~AMD_CPPC_MIN_PERF(~0L); + value |=3D AMD_CPPC_MIN_PERF(min_perf); + + value &=3D ~AMD_CPPC_MAX_PERF(~0L); + value |=3D AMD_CPPC_MAX_PERF(max_perf); + + /* CPPC EPP feature require to set zero to the desire perf bit */ + value &=3D ~AMD_CPPC_DES_PERF(~0L); + value |=3D AMD_CPPC_DES_PERF(0); + + if (cpudata->epp_policy =3D=3D cpudata->policy) + goto skip_epp; + + cpudata->epp_policy =3D cpudata->policy; + + if (cpudata->policy =3D=3D CPUFREQ_POLICY_PERFORMANCE) { + epp =3D amd_pstate_get_epp(cpudata, value); + cpudata->epp_powersave =3D epp; + if (epp < 0) + goto skip_epp; + + epp =3D 0; + } else { + if (cpudata->epp_powersave < 0) + goto skip_epp; + /* Get BIOS pre-defined epp value */ + epp =3D amd_pstate_get_epp(cpudata, value); + if (epp) + goto skip_epp; + epp =3D cpudata->epp_powersave; + } + /* Set initial EPP value */ + if (boot_cpu_has(X86_FEATURE_CPPC)) { + value &=3D ~GENMASK_ULL(31, 24); + value |=3D (u64)epp << 24; + } + +skip_epp: + WRITE_ONCE(cpudata->cppc_req_cached, value); + ret =3D wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value); + if (!ret) { + cpudata->epp_cached =3D epp; + } +} + +static void amd_pstate_set_max_limits(struct amd_cpudata *cpudata) +{ + u64 hwp_cap =3D READ_ONCE(cpudata->cppc_cap1_cached); + u64 hwp_req =3D READ_ONCE(cpudata->cppc_req_cached); + u32 max_limit =3D (hwp_cap >> 24) & 0xff; + + hwp_req &=3D ~AMD_CPPC_MIN_PERF(~0L); + hwp_req |=3D AMD_CPPC_MIN_PERF(max_limit); + wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, hwp_req); +} + +static int amd_pstate_epp_set_policy(struct cpufreq_policy *policy) +{ + struct amd_cpudata *cpudata; + + if (!policy->cpuinfo.max_freq) + return -ENODEV; + + pr_debug("set_policy: cpuinfo.max %u policy->max %u\n", + policy->cpuinfo.max_freq, policy->max); + + cpudata =3D all_cpu_data[policy->cpu]; + cpudata->policy =3D policy->policy; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + mutex_lock(&amd_pstate_limits_lock); + + if (cpudata->policy =3D=3D CPUFREQ_POLICY_PERFORMANCE) { + amd_pstate_clear_update_util_hook(policy->cpu); + amd_pstate_set_max_limits(cpudata); + } else { + amd_pstate_set_update_util_hook(policy->cpu); + } + + if (boot_cpu_has(X86_FEATURE_CPPC)) + amd_pstate_epp_init(policy->cpu); + + mutex_unlock(&amd_pstate_limits_lock); + } + + return 0; +} + +static void amd_pstate_verify_cpu_policy(struct amd_cpudata *cpudata, + struct cpufreq_policy_data *policy) +{ + update_boost_state(); + cpufreq_verify_within_cpu_limits(policy); +} + +static int amd_pstate_epp_verify_policy(struct cpufreq_policy_data *policy) +{ + amd_pstate_verify_cpu_policy(all_cpu_data[policy->cpu], policy); + pr_debug("policy_max =3D%d, policy_min=3D%d\n", policy->max, policy->min = ); + return 0; +} + static struct cpufreq_driver amd_pstate_driver =3D { .flags =3D CPUFREQ_CONST_LOOPS | CPUFREQ_NEED_UPDATE_LIMITS, .verify =3D amd_pstate_verify, @@ -696,8 +1414,20 @@ static struct cpufreq_driver amd_pstate_driver =3D { .attr =3D amd_pstate_attr, }; =20 +static struct cpufreq_driver amd_pstate_epp_driver =3D { + .flags =3D CPUFREQ_CONST_LOOPS, + .verify =3D amd_pstate_epp_verify_policy, + .setpolicy =3D amd_pstate_epp_set_policy, + .init =3D amd_pstate_epp_cpu_init, + .exit =3D amd_pstate_epp_cpu_exit, + .update_limits =3D amd_pstate_epp_update_limits, + .name =3D "amd_pstate_epp", + .attr =3D amd_pstate_epp_attr, +}; + static int __init amd_pstate_init(void) { + static struct amd_cpudata **cpudata; int ret; =20 if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD) @@ -712,10 +1442,25 @@ static int __init amd_pstate_init(void) if (cpufreq_get_current_driver()) return -EEXIST; =20 + cpudata =3D vzalloc(array_size(sizeof(void *), num_possible_cpus())); + if (!cpudata) + return -ENOMEM; + WRITE_ONCE(all_cpu_data, cpudata); + + if (epp_enabled) { + pr_info("AMD CPPC loading with amd_pstate_epp driver instance.\n"); + default_pstate_driver =3D &amd_pstate_epp_driver; + } else { + pr_info("AMD CPPC loading with amd_pstate driver instance.\n"); + default_pstate_driver =3D &amd_pstate_driver; + } + /* capability check */ if (boot_cpu_has(X86_FEATURE_CPPC)) { + if (!epp_enabled) { + default_pstate_driver->adjust_perf =3D amd_pstate_adjust_perf; + } pr_debug("AMD CPPC MSR based functionality is supported\n"); - amd_pstate_driver.adjust_perf =3D amd_pstate_adjust_perf; } else if (shared_mem) { static_call_update(amd_pstate_enable, cppc_enable); static_call_update(amd_pstate_init_perf, cppc_init_perf); @@ -732,19 +1477,56 @@ static int __init amd_pstate_init(void) return ret; } =20 - ret =3D cpufreq_register_driver(&amd_pstate_driver); + ret =3D cpufreq_register_driver(default_pstate_driver); if (ret) - pr_err("failed to register amd_pstate_driver with return %d\n", + pr_err("failed to register amd pstate driver with return %d\n", ret); =20 + amd_pstate_kobj =3D kobject_create_and_add("amd-pstate", &cpu_subsys.dev_= root->kobj); + if (!amd_pstate_kobj) { + pr_err("amd-pstate: Global sysfs registration failed.\n"); + } + + ret =3D sysfs_create_group(amd_pstate_kobj, &amd_pstate_global_attr_group= ); + if (ret) { + pr_err("amd-pstate: Sysfs attribute export failed with error %d.\n", + ret); + } + return ret; } =20 +static inline void amd_pstate_kobj_cleanup(struct kobject *kobj) +{ + kobject_del(kobj); + kobject_put(kobj); +} + static void __exit amd_pstate_exit(void) { - cpufreq_unregister_driver(&amd_pstate_driver); + unsigned int cpu; + + cpufreq_unregister_driver(default_pstate_driver); =20 amd_pstate_enable(false); + + sysfs_remove_group(amd_pstate_kobj, &amd_pstate_global_attr_group); + amd_pstate_kobj_cleanup(amd_pstate_kobj); + + cpus_read_lock(); + for_each_online_cpu(cpu) { + if (all_cpu_data[cpu]) { + if (default_pstate_driver =3D=3D &amd_pstate_epp_driver) + amd_pstate_clear_update_util_hook(cpu); + + spin_lock(&amd_pstate_cpu_lock); + kfree(all_cpu_data[cpu]); + WRITE_ONCE(all_cpu_data[cpu], NULL); + spin_unlock(&amd_pstate_cpu_lock); + } + } + cpus_read_unlock(); + } =20 module_init(amd_pstate_init); --=20 2.34.1