From nobody Mon Oct 6 03:15:58 2025 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 403FF1DDA09 for ; Fri, 25 Jul 2025 14:16:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753452982; cv=none; b=ZTxACkK19jlZQBZLQcB1cjZavRt4HgENglWxVvoEZcv1IW6buTqKQeCM5RmPg2D9hvcnd5G8hWx+9xMeNYF8P83tGEYBUt8dTo4wXXCYUz1fPuCY9oZh1XJ80uwZwXO9iHxw1IS24ELdxkb7+rajUYIMHjoCcXenyZaaXGqDaZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753452982; c=relaxed/simple; bh=Xip8jE5pNwtMgx7ORp3HsgqbQwAw/yGfdOPBmn7/ojQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GgRoBM7+6vo7twQotKrmvQuKPtM94+mL005WOi3kQvdlIT6t06LUMWZpY7w+08pJcnTBfSCBoZ+eZWupR+L3oSG6zd1MlMxbK+t4nekB534tXCIOXxsG6BCQJXPB3DxJtEK/07sFzo3/rVWNqsLaeW4F2k72oowhV46sPW5KiD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=qa5fzeGC; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="qa5fzeGC" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-ae0b2ead33cso386786666b.0 for ; Fri, 25 Jul 2025 07:16:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1753452978; x=1754057778; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=/zam/LW3oj60VYDi/BrVgQJFij9S3YsCyteAYO8PX1E=; b=qa5fzeGCaPC7T/LANLBabBtX9iOfWWLDcXhdXjjKv4h96ps0BDv05BoMs6+cHbr7YA 9wjbMSLbv8YqFDk3BuXvRnNx9AaMwLKASd9oehza2BDJnfLX3iEMnOJ/YO5lfOU9HszQ /cylqojecTdDERHbC8v4pqhkUgKKP2oJ2NRg8ybd6p37z8GwzvahflWZ5jzzv0L+Nb13 AnEsgrjsSqb+oHrLgKdhzMYc6c14cdaaFcodT14LHa7hddAyrnLgjU0QmnD11pBdSutv rARAB67OwXYOiVSNlfB91NVpb/5Adnocnf3cX/dV4y8hGlXaeW1VJaBkEKZZoqqOMB4S KK1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753452978; x=1754057778; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/zam/LW3oj60VYDi/BrVgQJFij9S3YsCyteAYO8PX1E=; b=JmxTj0YBrwhtvA2PDRX4Dst0Kua6x7YFTPoA1lr8AFhhlS0CJ2kmA9HSmwpV82iYFv 7ojyq8t96d/vLexTjrAhvdtLl2Pt4vU+XZaUj2bwmJGbTy8zmnuY3rpTupgjXs9PUBP+ eC8WvJ8aMx7MwA/nPBkyaDJ6hKSaJsZuGyUbeYJQtTRKA7MKhz/MxVo0SfMLfAfU95yS DQEFvx/nvc6FsP+E+YhhR7INRXpjAop7sFagRl+cCForlDN6jvaJdn8DQpSErNcBpNJy 6drhBXdrdb8eWy1tkFeQWNLgj7raXcrnaLF4K0lVF54psrMik4AxfzT3VT+J/DCooEpJ DX5Q== X-Forwarded-Encrypted: i=1; AJvYcCUVQqY+J/2LQI8+6UO8ma8Ng6LAvYkgfyMMa6L41kkEQ0qsGFsM/Oi//b3C0FAhbugyruXYlhqf8c3t9xo=@vger.kernel.org X-Gm-Message-State: AOJu0YzPlsVncaKfSVS3kGRu9lqMPwwkCj6rDqcdaXQVtOb/peJd2GOp reDzkCM1wPPHt9wdtwqZZthJS+P1kJEk48UlKmWqzRJ6iBJVWY4Mg86LP2mTK5W96Ek= X-Gm-Gg: ASbGnct8qXDlQsmc+TiXXHM+Iw8RiLI5u4ZGsssETMshJAnAGl1BpVTWRkv+QSnf2cs qlQePDbRG+kd7P0nkCuG4BBOJ8D32DoqPrqDKrdYb9PqJE3a2PYeyZGOlxbidtNF26MpRM6dGnO VeycOXn5Jp/pjtgbo1jJC2pAEwu051BzWs0+1+O67+e3FwJ19Tu8M5rjMKAeIdRj5Ab6trGrQR2 fUC/lzdgpMr6jxV7l7iOvaFIsO+JElNhcfqzZJwD0H7xaySrLFw7Jj4Q9+6I3XX34SEA6cn95EF EKCW13LEbIfD7MPuKBXwvJW71SoWTbZqxV0JnecKmTn9EYlTWBTC1nzwAuhVM6m8pDFdjDXX8rP zWrcfoMsGFSxgFhFMbsKASCPfLnVI4s9BHZkGJ810HKGMJLnE4YBKT0CtFSinLnfmoX6bCAJBeG AcS4UbGQ== X-Google-Smtp-Source: AGHT+IFX5GnNzDHfFClTllxEfxM5knR3JPoKqdSRHH8rDBPQpxv4Ttm+JHD8dxakGBd8eBWGd9ILPw== X-Received: by 2002:a17:907:9623:b0:ae3:c777:6e5e with SMTP id a640c23a62f3a-af61df52d26mr252419366b.19.1753452978372; Fri, 25 Jul 2025 07:16:18 -0700 (PDT) Received: from puffmais.c.googlers.com (8.239.204.35.bc.googleusercontent.com. [35.204.239.8]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-af47f85e60bsm278398966b.96.2025.07.25.07.16.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Jul 2025 07:16:17 -0700 (PDT) From: =?utf-8?q?Andr=C3=A9_Draszik?= Date: Fri, 25 Jul 2025 15:16:15 +0100 Subject: [PATCH v2 1/2] scsi: ufs: core: complete polled requests also from interrupt context Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250725-ufshcd-hardirq-v2-1-884c11e0b0df@linaro.org> References: <20250725-ufshcd-hardirq-v2-0-884c11e0b0df@linaro.org> In-Reply-To: <20250725-ufshcd-hardirq-v2-0-884c11e0b0df@linaro.org> To: Alim Akhtar , Avri Altman , Bart Van Assche , "James E.J. Bottomley" , "Martin K. Petersen" , Neil Armstrong Cc: Peter Griffin , Tudor Ambarus , Will McVicker , Manivannan Sadhasivam , kernel-team@android.com, linux-arm-msm@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, =?utf-8?q?Andr=C3=A9_Draszik?= X-Mailer: b4 0.14.2 When commit ee8c88cab4af ("scsi: ufs: core: Fix the polling implementation") was added, the block layer did not support completing polled requests from interrupt context. This has changed since (presumably with commit b99182c501c3 ("bio: add pcpu caching for non-polling bio_put")) and it is possible to complete polled requests also from interrupt context. Therefore, this commit here changes the code to also complete polled requests from interrupt context, mostly reverting above referenced commit as it is not necessary anymore. We do keep the fix to make ufshcd_poll() return a positive number. The change has been verified using: fio --name=3Drandread --rw=3Drandread --ioengine=3Dpvsync2 --hipri=3D1 \ --direct=3D1 --bs=3D4k --numjobs=3D8 --size=3D32m --runtime=3D30 \ --time_based --end_fsync=3D1 --group_reporting --filename=3D/foo which appears to have completed with no errors with this commit. Suggested-by: Bart Van Assche Signed-off-by: Andr=C3=A9 Draszik --- drivers/ufs/core/ufshcd.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-) diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index 13f7e0469141619cfc5e180aa730171ff01b8fb1..d8e2eabacd3efbf07458e81cc4d= 15ba7f05d3913 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -5613,26 +5613,6 @@ static void __ufshcd_transfer_req_compl(struct ufs_h= ba *hba, ufshcd_compl_one_cqe(hba, tag, NULL); } =20 -/* Any value that is not an existing queue number is fine for this constan= t. */ -enum { - UFSHCD_POLL_FROM_INTERRUPT_CONTEXT =3D -1 -}; - -static void ufshcd_clear_polled(struct ufs_hba *hba, - unsigned long *completed_reqs) -{ - int tag; - - for_each_set_bit(tag, completed_reqs, hba->nutrs) { - struct scsi_cmnd *cmd =3D hba->lrb[tag].cmd; - - if (!cmd) - continue; - if (scsi_cmd_to_rq(cmd)->cmd_flags & REQ_POLLED) - __clear_bit(tag, completed_reqs); - } -} - /* * Return: > 0 if one or more commands have been completed or 0 if no * requests have been completed. @@ -5656,10 +5636,6 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsi= gned int queue_num) WARN_ONCE(completed_reqs & ~hba->outstanding_reqs, "completed: %#lx; outstanding: %#lx\n", completed_reqs, hba->outstanding_reqs); - if (queue_num =3D=3D UFSHCD_POLL_FROM_INTERRUPT_CONTEXT) { - /* Do not complete polled requests from interrupt context. */ - ufshcd_clear_polled(hba, &completed_reqs); - } hba->outstanding_reqs &=3D ~completed_reqs; spin_unlock_irqrestore(&hba->outstanding_lock, flags); =20 @@ -5747,7 +5723,7 @@ static irqreturn_t ufshcd_transfer_req_compl(struct u= fs_hba *hba) * Ignore the ufshcd_poll() return value and return IRQ_HANDLED since we * do not want polling to trigger spurious interrupt complaints. */ - ufshcd_poll(hba->host, UFSHCD_POLL_FROM_INTERRUPT_CONTEXT); + ufshcd_poll(hba->host, 0); =20 return IRQ_HANDLED; } --=20 2.50.1.487.gc89ff58d15-goog From nobody Mon Oct 6 03:15:58 2025 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30EFE1FBE8B for ; Fri, 25 Jul 2025 14:16:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753452983; cv=none; b=cT4a39BmJ7IOJYIiq6EGUl6lf2p7alv2DjRQNRdPkQcSURtuoDqt7li3OtPPw+C4oB+Tbpu4tYMX751lqdqLerx2yqlJL/G5JMVE3eUKvhV40aeWnShO2qnRWW/xihMfrdmbv0taOwEnaNg0LtIVmVIOkZ4X5uHwe2Gjz/LW/88= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753452983; c=relaxed/simple; bh=0iuv0CuwWYN9pVhM3tpd+X0oxmalOPxMD79ong3ip58=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=B4MeaqCAEsWVesV4++kMUEeeReio1fJQ4MxkgudLZsYF+9UzzRR7zA9XFR1gR8CqIC7MZxb1RBNh7JjyWqu8bgeiCaqjjgKTRicHx9tKzxt+qBwoT7HZCSvQ5A720IVuSLB1fjv5OaDVAgG8OKNaW2XDSMcIZahnmdPyHn15y18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=e5RPjWqu; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="e5RPjWqu" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-612a338aed8so3485239a12.1 for ; Fri, 25 Jul 2025 07:16:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1753452979; x=1754057779; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=19qwY/QEDYUm2dm/vqD5B4okN8i0fK7LHJFqpIvnT/E=; b=e5RPjWqukl8RIJUw0JjYbCJsN0mafJL2ill17BoyPC8hDP9vsVaDdYIK3vqssYHc21 Fn1nSLnG5VTYifSwMGmIe9NURqmmZzOJ/P//ZZ34ymr5x8HCb8fhu9PDTxK8apmOOajb lLu4D89IKYZTbWxS5kPGuzO7UJSQqdBkLOUERozlx0xpZJzS5fVB0h5seI/7V4LCDI4G e+NkWShXdlVAYSNQa1sj3cIes+tQ846Y9Qc/rvXRDQYq/j4gN7aWg1AYInwv7eSME9/y s2k2cCdCAzzZ7bxTzvHYm61ru+dGjNds4WpgC2bfveW7gVCLAugNvfO62giTDj+FdtS8 ugRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753452979; x=1754057779; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=19qwY/QEDYUm2dm/vqD5B4okN8i0fK7LHJFqpIvnT/E=; b=jRkVz9bGyTZaVH55GZxEpHxAIH7rRE8xw3RPr7dk97UGCbbfBfpLjVAMgy8Krilc/Z cuDiOJ+fyJ0qqxilJ7QdfPX72SDTNrbSI4ehdqeTbT/koriJ6n9zDwCkJEaxdEK8syNa cByW/WV3gTyiz44Q3qXwrhpg1tqnCbt9dyoBc6tRc08w/FI5nZYK43yA1Z3LsqlilL37 taRgv9HBx6/G7QLiQSZk0Bs9P2I9OnZs8QqsufvOvNKMEyhzXNNDhK8j4clhtPzfDxDy hI1XKrv5Oj1AqfLJZelOMQjKQMqMCWcHzjLeVy0MYwRaGBf+d/tLDk1jMaiUZ8SwSrAy w1oQ== X-Forwarded-Encrypted: i=1; AJvYcCXTKJ+P56TkdMJiuXLEcsmUuPMqDVSoUWQH+kzwoJFls8KoecF6MT8MKYBQjlMTcQg+OB8tT3tMRkUdvuo=@vger.kernel.org X-Gm-Message-State: AOJu0Yyix+SiWnCX1YlgMHRzbD4NRtOxTFTM37vylK0XUN1P5+EFse00 +pFU0suH/X2SDbtBR28SWAAv0A4eLrRijgT3TLGzDM312JWssrK92eBMCqYKmpMgh9U= X-Gm-Gg: ASbGncudBvrkaifLa96HVLAPMlQgu0p++ZvinPWlkVL0x3W9b9bIrxWCk7ZGJlGrl1r 1WrR9ljLpZ1bQZtI7ywyCvmlnOyTLJx3/kJ3gu1VLbVbopPvGF6vTTBwplEsejRnIxBKKoW8PY7 A54LhXbwTf3i/NmNj2R2WreVekGd7rM4Vi25kz0M3n6yqeW2UEDfyVeySKnHjvtw4urnnF6P62i uazgfpqcJ1mr0ZdoFYvlA60/NRAVAuXRwM6KEiNByBdtBgxDGA6WoB3Vl8yeEjxE5C1Ar2Uf3pD SfX6Wz3XrRueh3Cf7RA2nsHao32mfB2A42ZdJLD202QLD8uFK/AXS7kCy8LFqE6MJ32NbowMhOB GmqFiZVU1gLX6906gNUpgehnzoSjJ802qzEKpsWb02gL9H74Z667FY95Om50qfgk5lDpnPopYDg GEdnw/1w== X-Google-Smtp-Source: AGHT+IEuW1ZoCs+ZSWwE5Bo66CeJoUWJvZNLBXvzG+Kl5PuCvUGyeu+LSCpZQqj+uLNwC/vztfLMOw== X-Received: by 2002:a17:907:9708:b0:ade:3bec:ea30 with SMTP id a640c23a62f3a-af61740d12dmr266751566b.1.1753452979088; Fri, 25 Jul 2025 07:16:19 -0700 (PDT) Received: from puffmais.c.googlers.com (8.239.204.35.bc.googleusercontent.com. [35.204.239.8]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-af47f85e60bsm278398966b.96.2025.07.25.07.16.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Jul 2025 07:16:18 -0700 (PDT) From: =?utf-8?q?Andr=C3=A9_Draszik?= Date: Fri, 25 Jul 2025 15:16:16 +0100 Subject: [PATCH v2 2/2] scsi: ufs: core: move some irq handling back to hardirq (with time limit) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250725-ufshcd-hardirq-v2-2-884c11e0b0df@linaro.org> References: <20250725-ufshcd-hardirq-v2-0-884c11e0b0df@linaro.org> In-Reply-To: <20250725-ufshcd-hardirq-v2-0-884c11e0b0df@linaro.org> To: Alim Akhtar , Avri Altman , Bart Van Assche , "James E.J. Bottomley" , "Martin K. Petersen" , Neil Armstrong Cc: Peter Griffin , Tudor Ambarus , Will McVicker , Manivannan Sadhasivam , kernel-team@android.com, linux-arm-msm@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, =?utf-8?q?Andr=C3=A9_Draszik?= , stable@vger.kernel.org X-Mailer: b4 0.14.2 Commit 3c7ac40d7322 ("scsi: ufs: core: Delegate the interrupt service routine to a threaded IRQ handler") introduced a massive performance drop for various work loads on UFSHC versions < 4 due to the extra latency introduced by moving all of the IRQ handling into a threaded handler. See below for a summary. To resolve this performance drop, move IRQ handling back into hardirq context, but apply a time limit which, once expired, will cause the remainder of the work to be deferred to the threaded handler. Above commit is trying to avoid unduly delay of other subsystem interrupts while the UFS events are being handled. By limiting the amount of time spent in hardirq context, we can still ensure that. The time limit itself was chosen because I have generally seen interrupt handling to have been completed within 20 usecs, with the occasional spikes of a couple 100 usecs. This commits brings UFS performance roughly back to original performance, and should still avoid other subsystem's starvation thanks to dealing with these spikes. fio results for 4k block size on Pixel 6, all values being the average of 5 runs each: read / 1 job original after this commit min IOPS 4,653.60 2,704.40 3,902.80 max IOPS 6,151.80 4,847.60 6,103.40 avg IOPS 5,488.82 4,226.61 5,314.89 cpu % usr 1.85 1.72 1.97 cpu % sys 32.46 28.88 33.29 bw MB/s 21.46 16.50 20.76 read / 8 jobs original after this commit min IOPS 18,207.80 11,323.00 17,911.80 max IOPS 25,535.80 14,477.40 24,373.60 avg IOPS 22,529.93 13,325.59 21,868.85 cpu % usr 1.70 1.41 1.67 cpu % sys 27.89 21.85 27.23 bw MB/s 88.10 52.10 84.48 write / 1 job original after this commit min IOPS 6,524.20 3,136.00 5,988.40 max IOPS 7,303.60 5,144.40 7,232.40 avg IOPS 7,169.80 4,608.29 7,014.66 cpu % usr 2.29 2.34 2.23 cpu % sys 41.91 39.34 42.48 bw MB/s 28.02 18.00 27.42 write / 8 jobs original after this commit min IOPS 12,685.40 13,783.00 12,622.40 max IOPS 30,814.20 22,122.00 29,636.00 avg IOPS 21,539.04 18,552.63 21,134.65 cpu % usr 2.08 1.61 2.07 cpu % sys 30.86 23.88 30.64 bw MB/s 84.18 72.54 82.62 Closes: https://lore.kernel.org/all/1e06161bf49a3a88c4ea2e7a406815be56114c4= f.camel@linaro.org Fixes: 3c7ac40d7322 ("scsi: ufs: core: Delegate the interrupt service routi= ne to a threaded IRQ handler") Cc: stable@vger.kernel.org Signed-off-by: Andr=C3=A9 Draszik --- v2: * update some inline & kerneldoc comments * mention 4k block size and 5 runs were used in fio runs * add missing jiffies.h include --- drivers/ufs/core/ufshcd.c | 191 +++++++++++++++++++++++++++++++++++++-----= ---- 1 file changed, 154 insertions(+), 37 deletions(-) diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index d8e2eabacd3efbf07458e81cc4d15ba7f05d3913..404a4e075a21e73d22ae6bb89f7= 7f69aebb7cd6a 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -111,6 +112,9 @@ enum { /* bMaxNumOfRTT is equal to two after device manufacturing */ #define DEFAULT_MAX_NUM_RTT 2 =20 +/* Time limit in usecs for hardirq context */ +#define HARDIRQ_TIMELIMIT 20 + /* UFSHC 4.0 compliant HC support this mode. */ static bool use_mcq_mode =3D true; =20 @@ -5603,26 +5607,56 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int = task_tag, * __ufshcd_transfer_req_compl - handle SCSI and query command completion * @hba: per adapter instance * @completed_reqs: bitmask that indicates which requests to complete + * @time_limit: time limit in jiffies to not exceed executing command comp= letion + * + * This completes the individual requests as per @completed_reqs with an + * optional time limit. If a time limit is given and it expired before all + * requests were handled, the return value will indicate which requests ha= ve not + * been handled. + * + * Return: Bitmask that indicates which requests have not been completed d= ue to + * time limit expiry. */ -static void __ufshcd_transfer_req_compl(struct ufs_hba *hba, - unsigned long completed_reqs) +static unsigned long __ufshcd_transfer_req_compl(struct ufs_hba *hba, + unsigned long completed_reqs, + unsigned long time_limit) { int tag; =20 - for_each_set_bit(tag, &completed_reqs, hba->nutrs) + for_each_set_bit(tag, &completed_reqs, hba->nutrs) { ufshcd_compl_one_cqe(hba, tag, NULL); + __clear_bit(tag, &completed_reqs); + if (time_limit && time_after_eq(jiffies, time_limit)) + break; + } + + /* any bits still set represent unhandled requests due to timeout */ + return completed_reqs; } =20 -/* - * Return: > 0 if one or more commands have been completed or 0 if no - * requests have been completed. +/** + * ufshcd_poll_impl - handle SCSI and query command completion helper + * @shost: Scsi_Host instance + * @queue_num: The h/w queue number, or UFSHCD_POLL_FROM_INTERRUPT_CONTEXT= when + * invoked from the interrupt handler + * @time_limit: time limit in jiffies to not exceed executing command comp= letion + * @__pending: Pointer to store any still pending requests in case of time= limit + * expiry + * + * This handles completed commands with an optional time limit. If a time = limit + * is given and it expires, @__pending will be set to the requests that co= uld + * not be completed in time and are still pending. + * + * Return: true if one or more commands have been completed, false otherwi= se. */ -static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num) +static int ufshcd_poll_impl(struct Scsi_Host *shost, unsigned int queue_nu= m, + unsigned long time_limit, unsigned long *__pending) { struct ufs_hba *hba =3D shost_priv(shost); unsigned long completed_reqs, flags; u32 tr_doorbell; struct ufs_hw_queue *hwq; + unsigned long pending =3D 0; =20 if (hba->mcq_enabled) { hwq =3D &hba->uhq[queue_num]; @@ -5636,15 +5670,34 @@ static int ufshcd_poll(struct Scsi_Host *shost, uns= igned int queue_num) WARN_ONCE(completed_reqs & ~hba->outstanding_reqs, "completed: %#lx; outstanding: %#lx\n", completed_reqs, hba->outstanding_reqs); - hba->outstanding_reqs &=3D ~completed_reqs; + + if (completed_reqs) { + pending =3D __ufshcd_transfer_req_compl(hba, completed_reqs, + time_limit); + completed_reqs &=3D ~pending; + hba->outstanding_reqs &=3D ~completed_reqs; + } + spin_unlock_irqrestore(&hba->outstanding_lock, flags); =20 - if (completed_reqs) - __ufshcd_transfer_req_compl(hba, completed_reqs); + if (__pending) + *__pending =3D pending; =20 return completed_reqs !=3D 0; } =20 +/* + * ufshcd_poll - SCSI interface of blk_poll to poll for IO completions + * @shost: Scsi_Host instance + * @queue_num: The h/w queue number + * + * Return: true if one or more commands have been completed, false otherwi= se. + */ +static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num) +{ + return ufshcd_poll_impl(shost, queue_num, 0, NULL); +} + /** * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is * invoked from the error handler context or ufshcd_host_reset_and_restore= () @@ -5698,13 +5751,19 @@ static void ufshcd_mcq_compl_pending_transfer(struc= t ufs_hba *hba, /** * ufshcd_transfer_req_compl - handle SCSI and query command completion * @hba: per adapter instance + * @time_limit: time limit in jiffies to not exceed executing command comp= letion * * Return: - * IRQ_HANDLED - If interrupt is valid - * IRQ_NONE - If invalid interrupt + * IRQ_HANDLED - If interrupt is valid + * IRQ_WAKE_THREAD - If further interrupt processing should be delegated = to the + * thread + * IRQ_NONE - If invalid interrupt */ -static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba) +static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba, + unsigned long time_limit) { + unsigned long pending; + /* Resetting interrupt aggregation counters first and reading the * DOOR_BELL afterward allows us to handle all the completed requests. * In order to prevent other interrupts starvation the DB is read once @@ -5720,12 +5779,18 @@ static irqreturn_t ufshcd_transfer_req_compl(struct= ufs_hba *hba) return IRQ_HANDLED; =20 /* - * Ignore the ufshcd_poll() return value and return IRQ_HANDLED since we - * do not want polling to trigger spurious interrupt complaints. + * Ignore the ufshcd_poll() return value and return IRQ_HANDLED or + * IRQ_WAKE_THREAD since we do not want polling to trigger spurious + * interrupt complaints. */ - ufshcd_poll(hba->host, 0); + ufshcd_poll_impl(hba->host, 0, time_limit, &pending); =20 - return IRQ_HANDLED; + /* + * If a time limit was set, some request completions might not have been + * handled yet and will need to be dealt with in the threaded interrupt + * handler. + */ + return pending ? IRQ_WAKE_THREAD : IRQ_HANDLED; } =20 int __ufshcd_write_ee_control(struct ufs_hba *hba, u32 ee_ctrl_mask) @@ -6286,7 +6351,7 @@ static void ufshcd_complete_requests(struct ufs_hba *= hba, bool force_compl) if (hba->mcq_enabled) ufshcd_mcq_compl_pending_transfer(hba, force_compl); else - ufshcd_transfer_req_compl(hba); + ufshcd_transfer_req_compl(hba, 0); =20 ufshcd_tmc_handler(hba); } @@ -6988,12 +7053,16 @@ static irqreturn_t ufshcd_handle_mcq_cq_events(stru= ct ufs_hba *hba) * ufshcd_sl_intr - Interrupt service routine * @hba: per adapter instance * @intr_status: contains interrupts generated by the controller + * @time_limit: time limit in jiffies to not exceed executing command comp= letion * * Return: - * IRQ_HANDLED - If interrupt is valid - * IRQ_NONE - If invalid interrupt + * IRQ_HANDLED - If interrupt is valid + * IRQ_WAKE_THREAD - If further interrupt processing should be delegated = to the + * thread + * IRQ_NONE - If invalid interrupt */ -static irqreturn_t ufshcd_sl_intr(struct ufs_hba *hba, u32 intr_status) +static irqreturn_t ufshcd_sl_intr(struct ufs_hba *hba, u32 intr_status, + unsigned long time_limit) { irqreturn_t retval =3D IRQ_NONE; =20 @@ -7007,7 +7076,7 @@ static irqreturn_t ufshcd_sl_intr(struct ufs_hba *hba= , u32 intr_status) retval |=3D ufshcd_tmc_handler(hba); =20 if (intr_status & UTP_TRANSFER_REQ_COMPL) - retval |=3D ufshcd_transfer_req_compl(hba); + retval |=3D ufshcd_transfer_req_compl(hba, time_limit); =20 if (intr_status & MCQ_CQ_EVENT_STATUS) retval |=3D ufshcd_handle_mcq_cq_events(hba); @@ -7016,15 +7085,25 @@ static irqreturn_t ufshcd_sl_intr(struct ufs_hba *h= ba, u32 intr_status) } =20 /** - * ufshcd_threaded_intr - Threaded interrupt service routine + * ufshcd_intr_helper - hardirq and threaded interrupt service routine * @irq: irq number * @__hba: pointer to adapter instance + * @time_limit: time limit in jiffies to not exceed during execution + * + * Interrupts are initially served from hardirq context with a time limit,= but + * if there is more work to be done than can be completed before the limit + * expires, remaining work is delegated to the IRQ thread. This helper doe= s the + * bulk of the work in either case - if @time_limit is set, it is being ru= n from + * hardirq context, otherwise from the threaded interrupt handler. * * Return: - * IRQ_HANDLED - If interrupt is valid - * IRQ_NONE - If invalid interrupt + * IRQ_HANDLED - If interrupt was fully handled + * IRQ_WAKE_THREAD - If further interrupt processing should be delegated = to the + * thread + * IRQ_NONE - If invalid interrupt */ -static irqreturn_t ufshcd_threaded_intr(int irq, void *__hba) +static irqreturn_t ufshcd_intr_helper(int irq, void *__hba, + unsigned long time_limit) { u32 last_intr_status, intr_status, enabled_intr_status =3D 0; irqreturn_t retval =3D IRQ_NONE; @@ -7038,15 +7117,22 @@ static irqreturn_t ufshcd_threaded_intr(int irq, vo= id *__hba) * if the reqs get finished 1 by 1 after the interrupt status is * read, make sure we handle them by checking the interrupt status * again in a loop until we process all of the reqs before returning. + * This is done until the time limit is exceeded, at which point further + * processing is delegated to the threaded handler. */ - while (intr_status && retries--) { + while (intr_status && !(retval & IRQ_WAKE_THREAD) && retries--) { enabled_intr_status =3D intr_status & ufshcd_readl(hba, REG_INTERRUPT_ENABLE); ufshcd_writel(hba, intr_status, REG_INTERRUPT_STATUS); if (enabled_intr_status) - retval |=3D ufshcd_sl_intr(hba, enabled_intr_status); + retval |=3D ufshcd_sl_intr(hba, enabled_intr_status, + time_limit); =20 intr_status =3D ufshcd_readl(hba, REG_INTERRUPT_STATUS); + + if (intr_status && time_limit && time_after_eq(jiffies, + time_limit)) + retval |=3D IRQ_WAKE_THREAD; } =20 if (enabled_intr_status && retval =3D=3D IRQ_NONE && @@ -7063,6 +7149,20 @@ static irqreturn_t ufshcd_threaded_intr(int irq, voi= d *__hba) return retval; } =20 +/** + * ufshcd_threaded_intr - Threaded interrupt service routine + * @irq: irq number + * @__hba: pointer to adapter instance + * + * Return: + * IRQ_HANDLED - If interrupt was fully handled + * IRQ_NONE - If invalid interrupt + */ +static irqreturn_t ufshcd_threaded_intr(int irq, void *__hba) +{ + return ufshcd_intr_helper(irq, __hba, 0); +} + /** * ufshcd_intr - Main interrupt service routine * @irq: irq number @@ -7070,20 +7170,37 @@ static irqreturn_t ufshcd_threaded_intr(int irq, vo= id *__hba) * * Return: * IRQ_HANDLED - If interrupt is valid - * IRQ_WAKE_THREAD - If handling is moved to threaded handled + * IRQ_WAKE_THREAD - If handling is moved to threaded handler * IRQ_NONE - If invalid interrupt */ static irqreturn_t ufshcd_intr(int irq, void *__hba) { struct ufs_hba *hba =3D __hba; + unsigned long time_limit =3D jiffies + + usecs_to_jiffies(HARDIRQ_TIMELIMIT); =20 - /* Move interrupt handling to thread when MCQ & ESI are not enabled */ - if (!hba->mcq_enabled || !hba->mcq_esi_enabled) - return IRQ_WAKE_THREAD; + /* + * Directly handle interrupts when MCQ & ESI are enabled since MCQ + * ESI handlers do the hard job. + */ + if (hba->mcq_enabled && hba->mcq_esi_enabled) + return ufshcd_sl_intr(hba, + ufshcd_readl(hba, REG_INTERRUPT_STATUS) & + ufshcd_readl(hba, REG_INTERRUPT_ENABLE), + 0); =20 - /* Directly handle interrupts since MCQ ESI handlers does the hard job */ - return ufshcd_sl_intr(hba, ufshcd_readl(hba, REG_INTERRUPT_STATUS) & - ufshcd_readl(hba, REG_INTERRUPT_ENABLE)); + /* + * Otherwise handle interrupt in hardirq context until the time limit + * expires, at which point the remaining work will be completed in + * interrupt thread context. + */ + if (!time_limit) + /* + * To deal with jiffies wrapping, we just add one so that other + * code can reliably detect if a time limit was requested. + */ + time_limit++; + return ufshcd_intr_helper(irq, __hba, time_limit); } =20 static int ufshcd_clear_tm_cmd(struct ufs_hba *hba, int tag) @@ -7516,7 +7633,7 @@ static int ufshcd_eh_device_reset_handler(struct scsi= _cmnd *cmd) __func__, pos); } } - __ufshcd_transfer_req_compl(hba, pending_reqs & ~not_cleared_mask); + __ufshcd_transfer_req_compl(hba, pending_reqs & ~not_cleared_mask, 0); =20 out: hba->req_abort_count =3D 0; @@ -7672,7 +7789,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd) dev_err(hba->dev, "%s: cmd was completed, but without a notifying intr, tag =3D %d", __func__, tag); - __ufshcd_transfer_req_compl(hba, 1UL << tag); + __ufshcd_transfer_req_compl(hba, 1UL << tag, 0); goto release; } =20 --=20 2.50.1.487.gc89ff58d15-goog