From nobody Mon Feb 9 14:34:46 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DD2726056A for ; Tue, 23 Dec 2025 06:22:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766470932; cv=none; b=BRzgj6hAQCVODLzERwi99B76Jf3EL6nlaJCJSr3es1u5YUgJYvijt2CR1/chzL2H+1YqaltaNZWv0DuJHuro0m+GI+UJ+2ojZ9c+qOX7KC4aF3CbSdq3JI9kfOkgM6Aipi1+Fsq2T8pLYiZJ7qAGF2aaCpRtYS+xJTTNOZYPwx4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766470932; c=relaxed/simple; bh=ii3tKR+4BjiW44lmltitk7q4UioocLQVYvSPZpv3Mqw=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=DL7saA+BFcL+tNdtufJRezHPFXZCkIcxL18e0q/K/a6Vpf8RqnI33QYBkKCB9tlFAuio1oxO8COsf3nxCgJPY44V0DnmKWjX8EgROvq/0lN9JQb6PwFxttnN0TvHicA872haTdYm4x2cER8m6v2MCCU5Eve8IV0iTjRiIkIMWE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Rmxzna8g; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Rmxzna8g" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2a09d981507so33735155ad.1 for ; Mon, 22 Dec 2025 22:22:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1766470930; x=1767075730; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=yHjVs+ePCiPAolPHVFOqDlm6Kow86qZw302+wVnuJm4=; b=Rmxzna8gzpY8/aoNmVQ5oM2j4oE4FqWK3h0OX8rdJkUKrM+AAYbQsS243AcJDwvtmf 4lG8G/xP0s90CuYCtd/rFaVXwNUNS2yb889xlUgVVkXg5lmxcWFzHmjsXzcfj568jUZD N5U2938/PFE+py3G1OPxXw1fVlboo7G7yC/mZuwshel/pD/LYpup9ozaKTeMRLlM+zYE VGP9pTgKLIYI/d6/AmVPPkgoJ64Ws0ErCIuBTDcX2gF+JvguLa7Sscu1s1WiFI9x1ahz /IvR0F4RahDMPqhPydCCUzqH7oLxrx8vdHO2bjRfD4PNsC2UfQ5LxpxEQpoezxBWQOtB I8oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766470930; x=1767075730; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=yHjVs+ePCiPAolPHVFOqDlm6Kow86qZw302+wVnuJm4=; b=GU2/e9RdfHrQD2YlhR4YFltaoemLLes4MLYfLHUpmQeaA7ZwYujfprVA5UyU53jQq4 +DHlRJG0IUO3m88Tdm4+M6CkRuz8c7MiFFiofqibghPwfoon/ah6HVcxnKm37plemMzv /Q92vAOcbTEm9vv7+CVGzxgtPwLoqnS3ktq5g8gc7289LPGmxjizd6dp1L9WClNR9PmM HMzOjcgPF6qcRmv5SnV4iTSMRYjXK4E6srMM6WFPVM3RN/pY63K0c6EtqfvU2ETO49FD km1vkziRT4dmUzFFTyDRqqTzxQMly3HyQH/9WZvDOvXqM6zbQpjJpwzDHxJNkoRgQRb+ yv4A== X-Forwarded-Encrypted: i=1; AJvYcCUT4tOtrcQ+JnShzIjeQBIt0BlR1UxqA0CrwbabJQxxvjzrRAfSLHNGh5AoDcGMLLd4VjzF5HBJuE0a/Y4=@vger.kernel.org X-Gm-Message-State: AOJu0YzDC1S2rmlerygjneRiiKX8oCv1xQ9k/QfccSzClpeRzoJVCPTJ QLA8P5FsKaposMIieN8e14E5sdIWUpbU3DuzIevIwTLzALfHNwyF/8VtG0B48nD8+V0= X-Gm-Gg: AY/fxX4dWm+jDIUOt2FeJlac/x+kxFkyrl6A9kXTe3hMwh44lWpDX4geOLz0avn/YmI jzRrWdZs95/AGQmXcu8a+ijaIO8N4dcgL1xiG2+mBFPNtrqy7Tii9eGsc0kI0ahf32vdTePkgnG UuVtdx1UQ68S40Txv0u+ABEsfOc1tpqEZsLcxbTKI/OORB8A0aEzPXwS2IPloFh6c1iEuGW/HxJ BTWlhAdD9Hmv9ohPwA5/U0+woYJtoKAdQ35hvVgV/rce8I1fcCsjR2M9TpNx+ju7WwTLrVACKDP MZATbCXoxw0a4kM7K2qCou9RLYBWrqsI+lYFD/3FLPj+3f57i2X0idPWmRpmnYNLFfAViWitrJb 3c1EVZhL6hfYIzNKjAxAoA+AJN8vnxYgFyCs8Ol14QYgmbepdjujkW92G+8D4XKIw9+VT1RSTer 0LvvfvdXFyB/djf8gNCsxc1E+o29o+Zp6QlNYk4FAuuy2DTxUVsco= X-Google-Smtp-Source: AGHT+IHkR80oZPE/nGAYfrvytYdflPvZplf9xx+aFYkbbPKy0wtYSX190VZI/iJQ8MAj/w4fLfaBGA== X-Received: by 2002:a17:902:e845:b0:2a0:7f8b:c0cb with SMTP id d9443c01a7336-2a2f0caa42amr155169735ad.4.1766470929914; Mon, 22 Dec 2025 22:22:09 -0800 (PST) Received: from tianci-mac.bytedance.net ([61.213.176.14]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3c71853sm117632915ad.19.2025.12.22.22.22.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 22 Dec 2025 22:22:09 -0800 (PST) From: Zhang Tianci To: miklos@szeredi.hu Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, xieyongji@bytedance.com, zhujia.zj@bytedance.com, Zhang Tianci , Jiachen Zhang Subject: [PATCH] fuse: add hang check in request_wait_answer() Date: Tue, 23 Dec 2025 14:21:13 +0800 Message-ID: <20251223062113.52477-1-zhangtianci.1997@bytedance.com> X-Mailer: git-send-email 2.48.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the FUSEDaemon fails to respond to FUSE requests due to certain reasons (e.g., deadlock), the kernel can detect this situation and issue an alert via logging. Based on monitoring of such alerts in the kernel logs, we can configure hang event alerts for large-scale deployed FUSEDaemon clusters. Signed-off-by: Jiachen Zhang Signed-off-by: Zhang Tianci --- fs/fuse/dev.c | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 6d59cbc877c6a..7b3d4160647a2 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -30,6 +30,9 @@ MODULE_ALIAS_MISCDEV(FUSE_MINOR); MODULE_ALIAS("devname:fuse"); =20 +static unsigned long hang_complain_secs =3D 60; +module_param(hang_complain_secs, ulong, 0644); + static struct kmem_cache *fuse_req_cachep; =20 const unsigned long fuse_timeout_timer_freq =3D @@ -545,14 +548,24 @@ static void request_wait_answer(struct fuse_req *req) { struct fuse_conn *fc =3D req->fm->fc; struct fuse_iqueue *fiq =3D &fc->iq; + unsigned int hang_check_time =3D 0; int err; =20 if (!fc->no_interrupt) { - /* Any signal may interrupt this */ - err =3D wait_event_interruptible(req->waitq, - test_bit(FR_FINISHED, &req->flags)); - if (!err) - return; + while (true) { + /* Any signal may interrupt this */ + err =3D wait_event_interruptible_timeout( + req->waitq, test_bit(FR_FINISHED, &req->flags), + READ_ONCE(hang_complain_secs) * HZ); + if (err > 0) + goto out; + if (err =3D=3D -ERESTARTSYS) + break; + if (hang_check_time++ =3D=3D 0) { + pr_debug("fuse conn %u req %llu (opcode %u) may hang.\n", + fc->dev, req->in.h.unique, req->args->opcode); + } + } =20 set_bit(FR_INTERRUPTED, &req->flags); /* matches barrier in fuse_dev_do_read() */ @@ -568,21 +581,38 @@ static void request_wait_answer(struct fuse_req *req) err =3D wait_event_killable(req->waitq, test_bit(FR_FINISHED, &req->flags)); if (!err) - return; + goto out; =20 if (test_bit(FR_URING, &req->flags)) removed =3D fuse_uring_remove_pending_req(req); else removed =3D fuse_remove_pending_req(req, &fiq->lock); if (removed) - return; + goto out; } =20 /* * Either request is already in userspace, or it was forced. * Wait it out. */ - wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags)); + while (true) { + err =3D wait_event_timeout(req->waitq, test_bit(FR_FINISHED, &req->flags= ), + READ_ONCE(hang_complain_secs) * HZ); + if (err > 0) + goto out; + if (err =3D=3D -ERESTARTSYS) + break; + if (hang_check_time++ =3D=3D 0) { + pr_debug("fuse conn %u req %llu (opcode %u) may hang.\n", + fc->dev, req->in.h.unique, req->args->opcode); + } + } +out: + if (hang_check_time) { + pr_debug("fuse conn %u req %llu (opcode %u) recovery after %lu seconds\n= ", + fc->dev, req->in.h.unique, req->args->opcode, + hang_check_time * READ_ONCE(hang_complain_secs)); + } } =20 static void __fuse_request_send(struct fuse_req *req) --=20 2.39.5