From nobody Mon Sep 28 13:18:02 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10C993DABE3; Fri, 21 Aug 2026 10:24:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307872; cv=none; b=Eh08uvEzdEz3gz5WDhOgfStokZiQMpMBwd4hJK/44DF0t1TxnJn6RdFO2ZvXf3gqHsHAq911LK64lOgZg/vdw73A7ODA1czEFix06pAS0u/8Bf7c+aIHgUjBx3/BFfVUrLgIMWRgnjE6haZcwxCxwX9HI5QUpr+cQhvDBgKjFQY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307872; c=relaxed/simple; bh=gc0aPv8jRNrLG2l2U/ppz2zQ2v8zFd3jRszGDWu+atQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=oCLPBkfrCN5EHcO8GspCpjFXxczylGw74rH+mAtPKnplT5mcfFKxxF4OLZWKbL01r9A0fxT6iQoDuj4KxZIPufAgdYHWqHhbxwUx/+zyKhxfgxv64/FR3caoMhtLFp2l6kgzM1rIMIkTpUteSZHiWddqdug+RUoHaES1hzIPvsw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=fBCypKmI; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="fBCypKmI" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=6Hho0TITdQeYLfNeIBNU2j2g48FSD7UnHlK0l1sFoug=; b=fBCypKmIjZd42fyeJhdVyIosxB pVwkDKR+i4VPptyCpuPS5gURXkfyiQLMgkSS4xoW+Sw6NK4zNUFNugFAcVwIa4Hw/qgR+uqu/HsWy Z+cj5f4epC7xjVsZh/i/f+tJrdMIT2dluOyOk/3KNpu2HTdWzHXLd28qTkMfuxfzFaKO66Sc2n698 HDa4D+UIp40CF4c++RcS7GYHOX8Z0sjy7xG50BJ8E1MsJasvs+B1c1HSBxR1eLEuVZc9glFe46Y2C w8I3NN4GjBVEZ4y9Lytpn0rkAGP7z4HyyY3Dma3gIZlGVAmSzWf717o+QJMy8eoHWq2Xloi07iKFD cWVvzgjg==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wxMQ7-00BH2x-1m; Fri, 21 Aug 2026 10:24:19 +0000 From: Breno Leitao Date: Fri, 21 Aug 2026 03:24:07 -0700 Subject: [PATCH 1/2] x86/mce: Rate-limit the CMCI storm transition notices Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260821-mce-panic-on-storm-v1-1-7a465c708d82@debian.org> References: <20260821-mce-panic-on-storm-v1-0-7a465c708d82@debian.org> In-Reply-To: <20260821-mce-panic-on-storm-v1-0-7a465c708d82@debian.org> To: Tony Luck , Borislav Petkov , Thomas Gleixner , Ingo Molnar , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Jonathan Corbet , Shuah Khan , Randy Dunlap Cc: linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.16-dev-f8e9d X-Developer-Signature: v=1; a=openpgp-sha256; l=2317; i=leitao@debian.org; h=from:subject:message-id; bh=gc0aPv8jRNrLG2l2U/ppz2zQ2v8zFd3jRszGDWu+atQ=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqiCdLv7yXolgo+J/qJ8JOAfPj75qqpoEYJRxSY G905HDSXoaJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaognSwAKCRA1o5Of/Hh3 bbjhD/9s3YSGAhCMTBeXhV6w6f2EN1EnkJKs+1/Q2gfqrGmbqLuZ9z2JZVNcG5WTKj3tb9Aq7j1 G1BkMApdiRXq/k9ZZD98NSt4df+/hVrsYmbu14xoxK3NirywXd77HzBx1lrT12I5EpTZYRYXQu2 g9aGHd6zv8fJjyCjUj4/zqNS1wZhYBRwytP4Byfh4n0SyDBymJvOV0bqX2BPk2JYrpTl883DdTV E0D7x0K3nnP+lnFkbQ+eWdowyDxB30mtrbHh5fRTB2TPvipewR/DU9/vpCR4A1k8lpfjwnaWCWa a+SdA3Efl+OT8hMLYKnY7eYtVnlU1Cnhksvum9wPbCX4Fm+zwBFEO0SvOObmu18oU8WxP2BMDKy V+qeODl2bxoi6immx1tm0bYuWxQ740EEGBV2zzbYFyHlNFNkVdVnU6z374Evdi5KK7ytybkGC3q VXFBXos5MpI3TXPrRyHP0q4UyRniBLKmQCteDOP8ks5sICbp/rkLs4D/SaYnKH8pkHbnbLDz5eG aktKMy995lUzQ5uQxV+zWTbOIIbWeI/QFONIWmUMJHIPRMy9Y8gMyZM5H4ah++EgbqgzSqvstlV /Ea7k0v+IfAGthz+2/OW/HPKgACqlHORbQxzpUrwbeA6I4z+az6hz9COxmBFLSTHNQrhN5fxWYj Cm9vI8y9NqwAZPg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao mce_track_storm() prints a line every time a bank enters storm mode and another every time it leaves. Neither is bounded, so the log volume follows whatever rate the bank flaps at. One host in our fleet logged 9970726 "CMCI storm detected" and 9970734 "CMCI storm subsided" lines for a single bank over four days, which is a bit excessive for all monitoring purposes. Put both transitions behind one ratelimit. Signed-off-by: Breno Leitao Acked-by: Tony Luck --- arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/= threshold.c index 6c370d5af5bd2..8591715ce2430 100644 --- a/arch/x86/kernel/cpu/mce/threshold.c +++ b/arch/x86/kernel/cpu/mce/threshold.c @@ -4,6 +4,7 @@ */ #include #include +#include =20 #include #include @@ -111,6 +112,13 @@ void cmci_storm_end(unsigned int bank) mce_timer_kick(false); } =20 +/* + * Shared by both transitions so that a bank flapping between them cannot + * outrun the console. + */ +static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + void mce_track_storm(struct mce *mce) { struct mca_storm_desc *storm =3D this_cpu_ptr(&storm_desc); @@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce) if (storm->banks[mce->bank].in_storm_mode) { if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0)) return; - printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_pr= ocessor_id(), mce->bank); + if (__ratelimit(&storm_rs)) + printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", + smp_processor_id(), mce->bank); mce_handle_storm(mce->bank, false); cmci_storm_end(mce->bank); } else { if (hweight64(history) < STORM_BEGIN_THRESHOLD) return; - printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_pr= ocessor_id(), mce->bank); + if (__ratelimit(&storm_rs)) + printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", + smp_processor_id(), mce->bank); mce_handle_storm(mce->bank, true); cmci_storm_begin(mce->bank); } --=20 2.53.0-Meta From nobody Mon Sep 28 13:18:02 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11F3A4756D2; Fri, 21 Aug 2026 10:24:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307874; cv=none; b=B1+7zuAFmc5Ep8+DoEh7zvO94h8UXm5x75LI4FqyaiaNMIyeXZnB6JCqw3+gnIftNRmqpoXJxMNhscCtf9QHUOUnMY/mU2DZWg1oQ0uETWoE6cPEHeeGX7COAne73ZLIZx1IvzNcqOF6lh2ZsQyn544FXVGT0Saw7cuazFQKYjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307874; c=relaxed/simple; bh=xyIHTH07+xKIY0xqKbzsHLS/FThvipvksxH9hFx8Gy0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=bq+l1kUNK2iW1YAlUR9sPKGD9Dsxb8/4WBPtM/d7ODfADCFYxX7YmsObe1AdOI2RjB9vrFLSwTYy9rzcM8iOrDATotqDrZHNu0f532f5D0YOWP0/fk/tpyf9VgoT9GCCTJISJW7xxdC0wbhfDeFyoI5KTsULrw+jIM4i+1HA0GI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=VB+4WkBG; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="VB+4WkBG" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=ma/oyiZEnUiqsZRo5AgRobLf9XEFeJTg6Dx6RZ5ZKGc=; b=VB+4WkBGyndesnxF+HcCY0/c8X RjLSAwDHjxUqgobzOj/witm5SAIzaAsLOgr8wyeFAiu/6pa/66L//zxMSoDnG/3mzt+JtLeum/0Mj dEUaEuuFZzziZ6oP5g6eruGQvYa4Vffg2Pu7VIs2ux0k0TlPXO39G7yccDpYP5t0228bcWVmvBuk8 JaYz5lNqstLWpHEhgC81DhV3VUwYitEOz/cuOm+PmSBGywt/t5oTuO+ptZ4nxhjlAaSAMwJvfRiqK 5I0v5/s4mfFCMSEBA4VH61Y2pM2MegbwupFhvTolu11iJjwg09AKRlUex+Z0yiUsz2nfQ4prEeLjB 9i9jYeRQ==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wxMQB-00BH34-2N; Fri, 21 Aug 2026 10:24:24 +0000 From: Breno Leitao Date: Fri, 21 Aug 2026 03:24:08 -0700 Subject: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260821-mce-panic-on-storm-v1-2-7a465c708d82@debian.org> References: <20260821-mce-panic-on-storm-v1-0-7a465c708d82@debian.org> In-Reply-To: <20260821-mce-panic-on-storm-v1-0-7a465c708d82@debian.org> To: Tony Luck , Borislav Petkov , Thomas Gleixner , Ingo Molnar , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Jonathan Corbet , Shuah Khan , Randy Dunlap Cc: linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.16-dev-f8e9d X-Developer-Signature: v=1; a=openpgp-sha256; l=8343; i=leitao@debian.org; h=from:subject:message-id; bh=xyIHTH07+xKIY0xqKbzsHLS/FThvipvksxH9hFx8Gy0=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqiCdL7QXhlwGG2CvKpvgLepna319lBoyhQ8GFF IG27sjGwIiJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaognSwAKCRA1o5Of/Hh3 bTyFD/wI39Ba7slTj66RduHa7D1ytc15mdQJUYKwREGR2BpTFXgOCIg0oXpvAuDJIIFJakxxsNb PsWVPtLmJQ0bsyAuA98DqkQzUs0ms0ErPV0ihkd+W6ZepeL+HnzVP30wxUYMC4W1MSImJBb91bH s7OBFURezizMWviAb3Ixzr3XoF+WLdmPoDNr2pi1C7GnUsvhRfGdeT+TRy/9rq/oKhRK7/Z44KP 3LaXiHM8cGgH0Y0jmKKKw2NnRb3u1eHOGT4rbhz1PpEnN4xWu2FA2J/a6oeyYXGkkj4Snnjpt0t vNPfekz0WiN/3vZUMrkbuPQ4t0VBN+cNVK4t915nxrmHlC6lyhalGolsnvtWkh5SFThBGDQIVvN xinpeZO2lbm5q4Pk5B4EA5v9HG+v9HCSbcXxHFKorXjrcdxkwT4IvikrGI73D9i+dY2CaowO820 0IYg5QxlVGEjJ3glx61FGgd43UWssh7yRh0Gp5xA+ZNXGLckrTzSpLuQOuOECp3ceQUwa8gcqyZ USAABOczZ6X2IIlD8AkgwMhKYLRneYU5ERYJDvdLXy24UdpdlzlWOXVPNmSeIFAOhvs3GyS+P/J 8KPGlsoGQrlkIajmLi4BJEsEQj0FM38eV5Kn4hHY+10BFWBZNR9IP+fDGc6XIyZamuzgCt/aSNc spY8VpHJwT5/ZCA== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao A machine check bank producing corrected errors faster than the kernel can drain them is not a machine anyone wants to keep in service, but nothing takes it out. mce_track_storm() throttles CMCI for the bank and the machine stays up. Count corrected errors per bank and add mce=3Dpanic_on_ce_count=3D and panic the host if we have more events than set. Off by default, for obvious reasons. Signed-off-by: Breno Leitao --- Documentation/ABI/testing/sysfs-mce | 12 ++++++++++++ Documentation/admin-guide/kernel-parameters.txt | 10 ++++++++++ arch/x86/kernel/cpu/mce/core.c | 14 ++++++++++++-- arch/x86/kernel/cpu/mce/internal.h | 6 ++++++ arch/x86/kernel/cpu/mce/threshold.c | 23 +++++++++++++++++++++= ++ 5 files changed, 63 insertions(+), 2 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-mce b/Documentation/ABI/testin= g/sysfs-mce index 83172f50e27c6..02009b5d581a4 100644 --- a/Documentation/ABI/testing/sysfs-mce +++ b/Documentation/ABI/testing/sysfs-mce @@ -95,3 +95,15 @@ Contact: Hidetoshi Seto Date: Jun 2009 Description: Disables the CMCI feature. + +What: /sys/devices/system/machinecheck/machinecheckX/panic_on_ce_count +Contact: Breno Leitao +Date: Aug 2026 +Description: + Panic once a machine check bank has logged this many corrected + errors. 0, the default, disables it. + + The setting is global rather than per-CPU, and it is compared + against a running total kept per CPU and per bank. Lowering it + below a total a bank has already reached takes the machine + down on that bank's next corrected error. diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index d55524e3b7246..d5005bfefb7be 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3924,6 +3924,16 @@ Kernel parameters print_all print all machine check logs to the console. =20 + panic_on_ce_count=3D + panic once a machine check bank has logged this many + corrected errors. 0, the default, disables it. + + This is a running total per CPU and per bank for the + life of the boot, not a rate, so a machine that logs + a slow trickle for long enough will reach any value + eventually. Pick one a failing part reaches in + minutes and a healthy one does not reach at all. + monarchtimeout (number) sets the time in us to wait for other CPUs on machine checks. 0 to disable. diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index ab469605fc893..43dcdef1a7bbe 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -261,7 +261,7 @@ static const char *mce_dump_aux_info(struct mce *m) return NULL; } =20 -static noinstr void mce_panic(const char *msg, struct mce_hw_err *final, c= har *exp) +noinstr void mce_panic(const char *msg, struct mce_hw_err *final, char *ex= p) { struct llist_node *pending; struct mce_evt_llist *l; @@ -813,6 +813,10 @@ void machine_check_poll(enum mcp_flags flags, mce_bank= s_t *b) barrier(); m->status =3D mce_rdmsrq(mca_msr_reg(i, MCA_STATUS)); =20 + /* The boot time poll replays errors from before this boot. */ + if (!(flags & MCP_QUEUE_LOG)) + mce_track_ce_count(m); + /* * Update storm tracking here, before checking for the * MCI_STATUS_VAL bit. Valid corrected errors count @@ -2326,6 +2330,7 @@ void mce_disable_bank(int bank) * mce=3Dnobootlog Don't log MCEs from before booting. * mce=3Dbios_cmci_threshold Don't program the CMCI threshold * mce=3Drecovery force enable copy_mc_fragile() + * mce=3Dpanic_on_ce_count=3D Panic after n corrected errors on one bank */ static int __init mcheck_enable(char *str) { @@ -2349,7 +2354,10 @@ static int __init mcheck_enable(char *str) cfg->print_all =3D true; else if (!strcmp(str, "ignore_ce")) cfg->ignore_ce =3D true; - else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog")) + else if (str_has_prefix(str, "panic_on_ce_count=3D")) { + str +=3D strlen("panic_on_ce_count=3D"); + get_option(&str, &cfg->panic_on_ce_count); + } else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog")) cfg->bootlog =3D (str[0] =3D=3D 'b'); else if (!strcmp(str, "bios_cmci_threshold")) cfg->bios_cmci_threshold =3D 1; @@ -2617,6 +2625,7 @@ static ssize_t store_int_with_restart(struct device *= s, } =20 static DEVICE_INT_ATTR(monarch_timeout, 0644, mca_cfg.monarch_timeout); +static DEVICE_INT_ATTR(panic_on_ce_count, 0644, mca_cfg.panic_on_ce_count); static DEVICE_BOOL_ATTR(dont_log_ce, 0644, mca_cfg.dont_log_ce); static DEVICE_BOOL_ATTR(print_all, 0644, mca_cfg.print_all); =20 @@ -2641,6 +2650,7 @@ static struct device_attribute *mce_device_attrs[] = =3D { &dev_attr_trigger, #endif &dev_attr_monarch_timeout.attr, + &dev_attr_panic_on_ce_count.attr, &dev_attr_dont_log_ce.attr, &dev_attr_print_all.attr, &dev_attr_ignore_ce.attr, diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/i= nternal.h index a31cf984619ca..7ebf87f6ca346 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -37,6 +37,7 @@ struct llist_node *mce_gen_pool_prepare_records(void); =20 int mce_severity(struct mce *a, struct pt_regs *regs, char **msg, bool is_= excp); struct dentry *mce_get_debugfs_dir(void); +noinstr void mce_panic(const char *msg, struct mce_hw_err *final, char *ex= p); =20 extern mce_banks_t mce_banks_ce_disabled; =20 @@ -64,6 +65,7 @@ void mce_timer_kick(bool storm); void cmci_storm_begin(unsigned int bank); void cmci_storm_end(unsigned int bank); void mce_track_storm(struct mce *mce); +void mce_track_ce_count(struct mce *mce); void mce_inherit_storm(unsigned int bank); bool mce_get_storm_mode(void); void mce_set_storm_mode(bool storm); @@ -72,6 +74,7 @@ u32 mce_get_apei_thr_limit(void); static inline void cmci_storm_begin(unsigned int bank) {} static inline void cmci_storm_end(unsigned int bank) {} static inline void mce_track_storm(struct mce *mce) {} +static inline void mce_track_ce_count(struct mce *mce) {} static inline void mce_inherit_storm(unsigned int bank) {} static inline bool mce_get_storm_mode(void) { return false; } static inline void mce_set_storm_mode(bool storm) {} @@ -83,12 +86,14 @@ static inline u32 mce_get_apei_thr_limit(void) { retur= n 0; } * represents an error seen. * * timestamp: Last time (in jiffies) that the bank was polled. + * ce_count: Corrected errors logged since boot. * in_storm_mode: Is this bank in storm mode? * poll_only: Bank does not support CMCI, skip storm tracking. */ struct storm_bank { u64 history; u64 timestamp; + u64 ce_count; bool in_storm_mode; bool poll_only; }; @@ -183,6 +188,7 @@ struct mca_config { bool ignore_ce; bool print_all; =20 + int panic_on_ce_count; int monarch_timeout; int panic_timeout; u32 rip_msr; diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/= threshold.c index 8591715ce2430..208dd6c8a5b18 100644 --- a/arch/x86/kernel/cpu/mce/threshold.c +++ b/arch/x86/kernel/cpu/mce/threshold.c @@ -119,6 +119,29 @@ void cmci_storm_end(unsigned int bank) static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST); =20 +/* + * A bank that keeps reporting corrected errors is repairing them faster t= han + * anything acts on it. Count them and let the admin put a ceiling on it. + */ +void mce_track_ce_count(struct mce *mce) +{ + struct storm_bank *bank =3D &this_cpu_ptr(&storm_desc)->banks[mce->bank]; + int limit =3D READ_ONCE(mca_cfg.panic_on_ce_count); + + if (limit <=3D 0) + return; + + if (!(mce->status & MCI_STATUS_VAL) || !mce_is_correctable(mce)) + return; + + if (++bank->ce_count < (u64)limit) + return; + + printk_deferred(KERN_EMERG "CPU%d BANK%d logged %llu corrected errors\n", + smp_processor_id(), mce->bank, bank->ce_count); + mce_panic("Too many corrected errors", NULL, NULL); +} + void mce_track_storm(struct mce *mce) { struct mca_storm_desc *storm =3D this_cpu_ptr(&storm_desc); --=20 2.53.0-Meta