From nobody Sun Feb 8 03:58:21 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5034014264F for ; Tue, 12 Mar 2024 19:39:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710272366; cv=none; b=Tb/V5yWMePs+UIpjCjizh6IHC6S+b18IDsekvmmeGbLN2FtXgxGC5gzPrbXuKJyq4T0s284cSU+sCRmumq/qqycB0P4UO15yGet3gvgVU2Lpcg28Ndfm/+YNo/Pudglsi2zZM4QEODkg/+q+Kozy8kjUdEX7rKGDQWvssSzfDCA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710272366; c=relaxed/simple; bh=bXN6LpucX8g1IPBaUvL4KJPwl6tCjlllO1SSWPVol+A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cvD92CGw2XhgUQHNKyoatHWeu9B0+jTIdtvX+btkGTfn4+GHvB+s9loWXvquNP7UXKnbk8LYKtxMUskxdkaz76CGtYLATOXQ0j5JsLCw5vhcmtPLDwhXtaAO1JtvnAQVwDii9phNlD14L3zP5IkWbxE7SoETn/A3yHBTy1kJnaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=t9v2jRF6; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t9v2jRF6" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-5e4df21f22dso128807a12.0 for ; Tue, 12 Mar 2024 12:39:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710272364; x=1710877164; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KtuhoJjlc6aLRLXbBM6dO4klGCHwBSsnzlu9NVynTBU=; b=t9v2jRF6DE9PLyL2F8yxOJkcD9emTER3LDRWrC6VLuvMooQNVLoYkT8WV8zPlmMBG0 tyWHs8xY+1Q9WDmxKVCXxPwSpOQUygYMjq3XATZEghHi81zEJLHLlXsHXpmIHb0kQcoE WlDjVzNMerH7j4YIcyn6ifV+zp1wSq2rYVA8uXpp3t/0RDe/D1D1GfS0mI0sPN9daWFK wTeeIE11PcFMSpgiaBlw1quaHiUQ4aGxA6lGRc+4nV/wo2g+9DJbK/1Q0257+Fe8wfeF b3dvlvVeJGoJs74BS2HxlcY11hIpivWTuU+ErARRLW7ROS8cweyhzShOpQe+5Khj7rvt sOsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710272364; x=1710877164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KtuhoJjlc6aLRLXbBM6dO4klGCHwBSsnzlu9NVynTBU=; b=tH5w2oos1NQhO00zlhisMik+7l6WcwnCQDnlKiMHO4F/Szz+Nz8f7DmpUH+JNoKMnW eEPAlPWr8mbSUuUj1Tmh8cKN2xIv/SweZ5MjRWxb9iaeO7PiZDpzG/q2NQpYp11Zy+Im /tHBwx9x1t4B8DpMOAJ6EpwbtwV1oYF0A7+rR3PTVhcLeawlsa/C/bFaC7ok0lG1UvoK veye2eIuWWPZ7j402xVzvm7pKVyncS0FDizHGsnGejRVbEzqtocrTXz0JPYFiUeSolD9 hVr0SbYsQyVCT3+XZNI19g6BqZP+NtRphHnUc9pxUl3TGhgZiufiXct9QnZcUBkncsjX G0gg== X-Forwarded-Encrypted: i=1; AJvYcCVADqzYy9zYQMXthfpGnghhbGPwuBTKSPPFB5F+eUY1Nz0odqof8uFIRN0MLJuw+S3U0lexkh3pgqlxQTOODY49SwQGyekkTWmWYBIS X-Gm-Message-State: AOJu0YygjqICC19u9Zyk952yqzFjPP89rAh04tDQRuzLXgD+lndG1Vgw 6tLzr91Wg1c/14oHE09yCfyZpaWud7XnaftFySVQ9m8r+RN0zAu3yCuqFJxeJDG1vhd/mNlaKYd WKQ== X-Google-Smtp-Source: AGHT+IHsa9gDpy8/PdSR11Pa+j9cL56mibi/uiU75NrIWPWyexGCLR0qj9UL3IZemnedxqaGh3LUTydlAtE= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:e647:0:b0:5dc:2a75:9a19 with SMTP id p7-20020a63e647000000b005dc2a759a19mr1193pgj.2.1710272364469; Tue, 12 Mar 2024 12:39:24 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 12 Mar 2024 12:39:10 -0700 In-Reply-To: <20240312193911.1796717-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240312193911.1796717-1-seanjc@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312193911.1796717-2-seanjc@google.com> Subject: [PATCH v2 1/2] sched/core: Move preempt_model_*() helpers from sched.h to preempt.h From: Sean Christopherson To: Jonathan Corbet , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Will Deacon Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Valentin Schneider , Marco Elver , Frederic Weisbecker , David Matlack , Friedrich Weber , Ankur Arora , Thomas Gleixner Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the declarations and inlined implementations of the preempt_model_*() helpers to preempt.h so that they can be referenced in spinlock.h without creating a potential circular dependency between spinlock.h and sched.h. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Ankur Arora --- include/linux/preempt.h | 41 +++++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 41 ----------------------------------------- 2 files changed, 41 insertions(+), 41 deletions(-) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 7233e9cf1bab..ce76f1a45722 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -481,4 +481,45 @@ DEFINE_LOCK_GUARD_0(preempt, preempt_disable(), preemp= t_enable()) DEFINE_LOCK_GUARD_0(preempt_notrace, preempt_disable_notrace(), preempt_en= able_notrace()) DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) =20 +#ifdef CONFIG_PREEMPT_DYNAMIC + +extern bool preempt_model_none(void); +extern bool preempt_model_voluntary(void); +extern bool preempt_model_full(void); + +#else + +static inline bool preempt_model_none(void) +{ + return IS_ENABLED(CONFIG_PREEMPT_NONE); +} +static inline bool preempt_model_voluntary(void) +{ + return IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY); +} +static inline bool preempt_model_full(void) +{ + return IS_ENABLED(CONFIG_PREEMPT); +} + +#endif + +static inline bool preempt_model_rt(void) +{ + return IS_ENABLED(CONFIG_PREEMPT_RT); +} + +/* + * Does the preemption model allow non-cooperative preemption? + * + * For !CONFIG_PREEMPT_DYNAMIC kernels this is an exact match with + * CONFIG_PREEMPTION; for CONFIG_PREEMPT_DYNAMIC this doesn't work as the + * kernel is *built* with CONFIG_PREEMPTION=3Dy but may run with e.g. the + * PREEMPT_NONE model. + */ +static inline bool preempt_model_preemptible(void) +{ + return preempt_model_full() || preempt_model_rt(); +} + #endif /* __LINUX_PREEMPT_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 17cb0761ff65..e9dc10f7a463 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2058,47 +2058,6 @@ extern int __cond_resched_rwlock_write(rwlock_t *loc= k); __cond_resched_rwlock_write(lock); \ }) =20 -#ifdef CONFIG_PREEMPT_DYNAMIC - -extern bool preempt_model_none(void); -extern bool preempt_model_voluntary(void); -extern bool preempt_model_full(void); - -#else - -static inline bool preempt_model_none(void) -{ - return IS_ENABLED(CONFIG_PREEMPT_NONE); -} -static inline bool preempt_model_voluntary(void) -{ - return IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY); -} -static inline bool preempt_model_full(void) -{ - return IS_ENABLED(CONFIG_PREEMPT); -} - -#endif - -static inline bool preempt_model_rt(void) -{ - return IS_ENABLED(CONFIG_PREEMPT_RT); -} - -/* - * Does the preemption model allow non-cooperative preemption? - * - * For !CONFIG_PREEMPT_DYNAMIC kernels this is an exact match with - * CONFIG_PREEMPTION; for CONFIG_PREEMPT_DYNAMIC this doesn't work as the - * kernel is *built* with CONFIG_PREEMPTION=3Dy but may run with e.g. the - * PREEMPT_NONE model. - */ -static inline bool preempt_model_preemptible(void) -{ - return preempt_model_full() || preempt_model_rt(); -} - static __always_inline bool need_resched(void) { return unlikely(tif_need_resched()); --=20 2.44.0.278.ge034bb2e1d-goog From nobody Sun Feb 8 03:58:21 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F88A1428F7 for ; Tue, 12 Mar 2024 19:39:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710272368; cv=none; b=mrk5ak+ZEsvO3HB3PVu0WzVJnX27v3WQ2fFBeUf7hEbg2iFL1cmLODK8LMch7nnyR/enlE9ArK/lg83rVTXUDHOOW4nPB7KGN5V0Vn8GgAB0JbPUs//tUoanaOAVP14Ywnb9oKXT4aEzNaiAtA7A+mZ+dFm+PC02ngQNo47i1eU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710272368; c=relaxed/simple; bh=7F4aenK3C8j4SGFBioKeFeh7bXOjOVxZfo8KHxdPG24=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KUNu0xlku65AKMA9RwdxoQp8oT2kYsKawz2yasBegZAlH9LhxTUt4sx2C3PKiZeefGPePomuxHrWjX5+AjmXXQcT0PNpJUcU57ydvAnFbCmSuY0Kvnup3LqEvvELJ6SClqj1vFpqWjcAWNq4Mu07qA4YwZ+AIPbbPP6PbYwzsdE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Np2mpj5m; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Np2mpj5m" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-5e4f312a995so182109a12.2 for ; Tue, 12 Mar 2024 12:39:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710272366; x=1710877166; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=y37QTRRCaOyhTss07/2XbpFXeVVO9uDT6HFjFeJL8cA=; b=Np2mpj5mDgAsX3zwczEbvWJa1bO9R+/asoE3YdhmN7PWnVNjMe1p23BLatbw5YTc+V 4x314HooLgaNYgtEDo1Qb2ees9lxeW7q6V043OyNv/Nsaz0pAGNjtgqlPRdM+MqEfg7L zN55OXSszluXqVGQXNFDbkOzB0PR/BlzoXfIYPrWmPOFwWRmdiS7Lpv4O6kr38RSXXJy Y9LCye8ek3vA1GEgfjOfd7vfZy+6eRPmQ6wSrPFIcfueH2UAhol/QDUr631IJIMfqAa7 nxJvuwkIIWQ0Q75Es05fpd5I3vcOqF6lstMNl/RzGI95IyLcXZj/1SVG5zxRhQFGdpRt pFYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710272366; x=1710877166; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=y37QTRRCaOyhTss07/2XbpFXeVVO9uDT6HFjFeJL8cA=; b=SgJZI2MY6WUPFR+vAPgzA2lQtdm0MjIVx3dQRkEFuTh/jiwca9+zQPPkC9PoijE22E P37oXZhCt/7cQtU2x2k0RNV8Lfh11cFLo4HKXOAxyUHutOXmYHatZGUY3OrsUJJwxrmy caoP8RAU8kuMv/YHspwdTcBM5s1ATpAnZa5JoZvpC9EvzV40klJCS1UM+7GG16NRQn2/ 5aFoNUffSe6eVbnLBjsV5QLTr0PkJZ/n1MR19Umh72BgHuCFwg007D5vp2xrkxasVGHg OCqYftQ0XiKEcybIUHIny7fPr4+SeP4xTLTDOrKsp1ZvNIFbd2NZVKRPwRGx/7pDbLtb IH2g== X-Forwarded-Encrypted: i=1; AJvYcCV7NNU9a6ZteR3IfaSP90IqwngQjOjGQWEspgV5vK6fAQjiDliNmKVvDYCVt8zVRTR74NlsO8sp2GeH+bz8ofhb1MoJx9axnlaCQKHJ X-Gm-Message-State: AOJu0YzgdnRSC+MM1thUloIFLNY28U+bduAOyaJ77K/EI6m9YJY4h639 ouPXqB3aQ9RAV/pie/i8+XPzZRiyIZFey48F2vg0OhAxWecUytgH/hoO2rdjFPLGNsfu6S1+w7q 2bQ== X-Google-Smtp-Source: AGHT+IGI7dHHFkEz/HTwu1O+HPjj3k60ryUQ+qwg4B5aW7wTMmFk2zSK0bYqYkStNTwT9sTditr0+hkTqZ4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a02:714:b0:5dc:8f95:3d with SMTP id cb20-20020a056a02071400b005dc8f95003dmr2886pgb.2.1710272366615; Tue, 12 Mar 2024 12:39:26 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 12 Mar 2024 12:39:11 -0700 In-Reply-To: <20240312193911.1796717-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240312193911.1796717-1-seanjc@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312193911.1796717-3-seanjc@google.com> Subject: [PATCH v2 2/2] sched/core: Drop spinlocks on contention iff kernel is preemptible From: Sean Christopherson To: Jonathan Corbet , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Will Deacon Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Valentin Schneider , Marco Elver , Frederic Weisbecker , David Matlack , Friedrich Weber , Ankur Arora , Thomas Gleixner Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use preempt_model_preemptible() to detect a preemptible kernel when deciding whether or not to reschedule in order to drop a contended spinlock or rwlock. Because PREEMPT_DYNAMIC selects PREEMPTION, kernels built with PREEMPT_DYNAMIC=3Dy will yield contended locks even if the live preemption model is "none" or "voluntary". In short, make kernels with dynamically selected models behave the same as kernels with statically selected models. Somewhat counter-intuitively, NOT yielding a lock can provide better latency for the relevant tasks/processes. E.g. KVM x86's mmu_lock, a rwlock, is often contended between an invalidation event (takes mmu_lock for write) and a vCPU servicing a guest page fault (takes mmu_lock for read). For _some_ setups, letting the invalidation task complete even if there is mmu_lock contention provides lower latency for *all* tasks, i.e. the invalidation completes sooner *and* the vCPU services the guest page fault sooner. But even KVM's mmu_lock behavior isn't uniform, e.g. the "best" behavior can vary depending on the host VMM, the guest workload, the number of vCPUs, the number of pCPUs in the host, why there is lock contention, etc. In other words, simply deleting the CONFIG_PREEMPTION guard (or doing the opposite and removing contention yielding entirely) needs to come with a big pile of data proving that changing the status quo is a net positive. Opportunistically document this side effect of preempt=3Dfull, as yielding contended spinlocks can have significant, user-visible impact. Fixes: c597bfddc9e9 ("sched: Provide Kconfig support for default dynamic pr= eempt mode") Link: https://lore.kernel.org/kvm/ef81ff36-64bb-4cfe-ae9b-e3acf47bff24@prox= mox.com Cc: Valentin Schneider Cc: Peter Zijlstra (Intel) Cc: Marco Elver Cc: Frederic Weisbecker Cc: David Matlack Cc: Friedrich Weber Cc: Ankur Arora Cc: Thomas Gleixner Signed-off-by: Sean Christopherson Reviewed-by: Ankur Arora Reviewed-by: Chen Yu --- Documentation/admin-guide/kernel-parameters.txt | 4 +++- include/linux/spinlock.h | 14 ++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 825398d66c69..fdeddb066439 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4689,7 +4689,9 @@ none - Limited to cond_resched() calls voluntary - Limited to cond_resched() and might_sleep() calls full - Any section that isn't explicitly preempt disabled - can be preempted anytime. + can be preempted anytime. Tasks will also yield + contended spinlocks (if the critical section isn't + explicitly preempt disabled beyond the lock itself). =20 print-fatal-signals=3D [KNL] debug: print fatal signals diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 3fcd20de6ca8..63dd8cf3c3c2 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -462,11 +462,10 @@ static __always_inline int spin_is_contended(spinlock= _t *lock) */ static inline int spin_needbreak(spinlock_t *lock) { -#ifdef CONFIG_PREEMPTION + if (!preempt_model_preemptible()) + return 0; + return spin_is_contended(lock); -#else - return 0; -#endif } =20 /* @@ -479,11 +478,10 @@ static inline int spin_needbreak(spinlock_t *lock) */ static inline int rwlock_needbreak(rwlock_t *lock) { -#ifdef CONFIG_PREEMPTION + if (!preempt_model_preemptible()) + return 0; + return rwlock_is_contended(lock); -#else - return 0; -#endif } =20 /* --=20 2.44.0.278.ge034bb2e1d-goog