From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A96B72080FE for ; Tue, 10 Dec 2024 10:54:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828072; cv=none; b=XrVwhpdX5nBcHrg6MrAQW1LEwJexK6mz5Yw2bDPpwlafMzeXRYTiTL5sCAmzOUH35LO+rfH4LVaUKksv50I7TScGS5G2YyOH5X7pyjUdeQtJzOe6sPeAG9IX7V/mq4n2082944R0mYdE4MXRTnpKMHyvn0e7JfhuAeDz5fULLlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828072; c=relaxed/simple; bh=CCuhclzq1Bj2x4zFpyOxb3Seo48oZFBuzLTkJ5AkbGA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W4E61cJ9j+cBC6cXSzzX7n0al8GnH/cGEAJN/33OslYv6PU6eZLh8iTQsLFxRnfR6Kw6WPJHPsLG5G8J/3jXL45Lwj68uFCdWlWinlUvp68qtqMTTf+64oD9Or1z1j1WYPaiiTK7N+D1/NSCUe0PChO/+UnBQp9DBPJ1IpakwTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=C3yprvHr; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="C3yprvHr" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-7273967f2f0so1424928b3a.1 for ; Tue, 10 Dec 2024 02:54:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828070; x=1734432870; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XBgnV26LOPK7kV/TRK60xkNTdvFSpncDHyQsHKHEa1Q=; b=C3yprvHrp7c1A1o0dvCt8MPzmW5tFB7a8fNsO96aD+4qY6P7I6sI4cgddO2zhYRLmg 5mMzQj/DAqAYd8p7QPIbftb3K8KEO/RJRvSWAvawsDDERkqMta/cJ4PkDhlGMNNGvm0e Ok0zbWuzfGYeJB6Q920FiS2t/3085yUyE2gQw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828070; x=1734432870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XBgnV26LOPK7kV/TRK60xkNTdvFSpncDHyQsHKHEa1Q=; b=QiG0XPeJZVuNQ8K9xTu+3/+mwcScM9A7AeAlZigxeVNknPttyyUYQXMM4i1sfXBA4S oPXSdTcL6gU2+qyPPaNUBepZR+nfMaKZpi2Uh1eyiiCnV7QcjvddZe9iA5INL7cWPhiN 2uuN2cqzAfH5cezvrorEcCJBmZJ3q+pNDc+LWh5G/Jo/smwzcyvm0CeggrMNsnOEYxH2 srV85xBMiIkQu3fPxGUg4AatLbB/aL9WSb6LX1E1prC6CPROWEoutwEUr7QtfUtdWqMe XTNHf1BHdIde6QS27IDz9uOt7JGxQ3Rupnt/7dzruAg3kJ6zcglPJnw5GBruYBuSiuMd 7eOA== X-Forwarded-Encrypted: i=1; AJvYcCVFQo+1gw9u+vx5Pa3K+/FvjLy7p8dUraOXBUy4Wxp64UK3BjUuE6liM/jID77sTERf53mojiZYjWBkCBY=@vger.kernel.org X-Gm-Message-State: AOJu0YyScueDa72fYyAJuPQZWGvEz+eDOkApTF8u07pWkdWqTEhW+dlG A1gVppcKsH47MfYU9vNrocfr6d9gAZtdCH2K3LYt4MV8ACYRDHx5fVIT1oM0pg== X-Gm-Gg: ASbGnctwzdfrJCh8g9SRSL+vSyjKptK/cXU23ZHeTaO3UrWtgDirL+/0AR793RVkHrn Z9foAbuI3w/PYnZC9teLRDbPW/U7XGlqcoo2swR4T1XIjmCYQwLGpXxDcNCgov3fyBBYCQjN1Xf Mt60IaFjN35voj6ozCZtlp4M6BfowxOzMz8MXBPaXweGJUbfZ1L7BI+LqMK5GdUxAPgZ+3HQxFz NsCL37FSrOwDuf6YvK3cWrQ5Z8TevJosnY3XaaN5wy3We2Tf9bgE5Z2 X-Google-Smtp-Source: AGHT+IHN5FQB1uPxNfGbJsfWaRhVvwe4RKqTEDg2FsXlyWluGVdmlf9tuZmfWudvDjFX+ZffT2ip/g== X-Received: by 2002:a05:6a00:b82:b0:725:ef4b:de33 with SMTP id d2e1a72fcca58-7273c7feaf9mr6117335b3a.0.1733828070002; Tue, 10 Dec 2024 02:54:30 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7fd37745f99sm5583605a12.10.2024.12.10.02.54.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:29 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 1/6] zram: cond_resched() in writeback loop Date: Tue, 10 Dec 2024 19:53:55 +0900 Message-ID: <20241210105420.1888790-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Writeback loop can run for quite a while (depending on wb device performance, compression algorithm and the number of entries we writeback), so we need to do cond_resched() there, similarly to what we do in recompress loop. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 5de7f30d0aa6..172546aa1fce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -889,6 +889,8 @@ static ssize_t writeback_store(struct device *dev, next: zram_slot_unlock(zram, index); release_pp_slot(zram, pps); + + cond_resched(); } =20 if (blk_idx) --=20 2.47.1.613.gc27f4b7a9f-goog From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66DB52080FE for ; Tue, 10 Dec 2024 10:54:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828076; cv=none; b=PBI55+f6edqdLnr+zAYBnwifWdB8aABAWzwHO2V58JkSRvx2mwrM+XgKb8gGCG+gF5Z60GbCtRwQXc8Kl37IakZP4O3rB65v+DzUKOMwoVsGNOJwyB6/Ef70Oycoef11ceR/QLPZBn26MFgle4qb8/gDfBDm5JgYnkgTj97PFC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828076; c=relaxed/simple; bh=lE+1kRH1U/7IVQo6hTz7tDbC7zT0gIo9rITcVmUZ18A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FG+wVsQhxoPAV8WJi+opw1WrwuNQ1klEgQF7clv80m7YLiFagR1lI7OM8X15f+/WzCv1jS+4t2jewYBYff6dXwAChscC/tmZ2U+18yeLTApC/nqdlRA5PSYcJcT0eg9Zwu4ZyMo2q62urCHvkxaZk7NKhzKNoljnt2yY8lXNqlU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=UeBfT/C6; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="UeBfT/C6" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21619108a6bso27809865ad.3 for ; Tue, 10 Dec 2024 02:54:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828075; x=1734432875; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATNOyPUUs3gPB5Gzn7rO553POCCitam43iQCzU3DuAw=; b=UeBfT/C6NzRz3sU0+Uii8ZZl3fyuQhipmnSJjsMcSH+kSwPJxqUIOrKPipeF1bRX9h X08xuwGsKlIJJ94ogtM1eExs8ii9+5ER/c+1/YOzltaJy6RtY1En9GQOxm9RFlnfVINJ PKW+xcOiqufHlrLy32qxSuT2b4M9viavA884g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828075; x=1734432875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATNOyPUUs3gPB5Gzn7rO553POCCitam43iQCzU3DuAw=; b=BD5Wf6/qzg/N69EAQeQv4KCv36u7+Rhi7B8Az/fKrazM0L8Vg+953yvzM6g2g34S9f 5V5PUHWiMK1AmDBqreefr4cgo9nx9ju8sqNnKcI8bEKecxIHqtQgie916zzgDoC98VQC MrD+IHEugthJJMNZxxqmlbWtMEJdkBhoCOr4o3mfO8KK5Iv1wlj/I0gq8nlNlfhl5Rx3 pFKjTkjCrjCwY/fpAsk6AYsSEUOj9Pwfp5P47jgrQHp3u81Rmj+4wgqJf4U8elTr0bPH ocJ832gnDfud4cYduQrRmYzamur0fGBbY7lqNhErLwZF56o2FePYHGrxC94kZm4rPvpJ rZrQ== X-Forwarded-Encrypted: i=1; AJvYcCVrexj06y9B3XC4tHsYM3Z/erpkGr3MIZxePyhGVy6lxsOS48aWLTacjne+qj14ZXBXP/eWAIlsRqbjr1w=@vger.kernel.org X-Gm-Message-State: AOJu0YyRoP9x1Nwq/+XAGs1lAQoD7RBKCQaf0H0PnlMZkvTXmFhrQ6K/ 20cuRHouAg2GhUFCarQAj2+TcW1WwQ3MVPVojLM5ellj1eHQ0rjK6JNb7+lXFA== X-Gm-Gg: ASbGnctG39r8qOVgOCzbRnNqAhpJVs2g26vHgbIISuqpbCp8yRKQe4x0k2iYyW5+dcj 8OSIOnZRC85PeWWET6PQwUb/heiEw4bPpiIA2032c93FUhygYyxIzyj+lA0jaR88PM5DPZMbH/1 1HFlQQJptTUdcn35qG2mro6tzujjA+xDBJXcSGJIFA2ZjXhIMcvSOCM2uBc2g4mhlvF9zsPnmFf CVr+u9jroz6Rjf1vgDwJYTPbugorEWTqvnb3KVH7Kqd0hLx8skByohZ X-Google-Smtp-Source: AGHT+IHwqm8mlfj88tt2oDbXl2G2KOx142k+H+H2T7ahDyGRc12eSz/W7yAD73omKWT7c7y3DLD5LQ== X-Received: by 2002:a17:902:cec2:b0:216:4263:133f with SMTP id d9443c01a7336-2166a0bc05bmr51425815ad.57.1733828074786; Tue, 10 Dec 2024 02:54:34 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-2167504c525sm9825525ad.12.2024.12.10.02.54.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:34 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 2/6] zram: free slot memory early during write Date: Tue, 10 Dec 2024 19:53:56 +0900 Message-ID: <20241210105420.1888790-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the current implementation entry's previously allocated memory is released in the very last moment, when we already have allocated a new memory for new data. This, basically, temporarily increases memory usage for no good reason. For example, consider the case when both old (stale) and new entry data are incompressible so such entry will temporarily use two physical pages - one for stale (old) data and one for new data. We can release old memory as soon as we get a write request for entry. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 172546aa1fce..174b4053189b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1647,6 +1647,11 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) unsigned long element =3D 0; enum zram_pageflags flags =3D 0; =20 + /* First, free memory allocated to this slot (if any) */ + zram_slot_lock(zram, index); + zram_free_page(zram, index); + zram_slot_unlock(zram, index); + mem =3D kmap_local_page(page); if (page_same_filled(mem, &element)) { kunmap_local(mem); @@ -1742,13 +1747,7 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); out: - /* - * Free memory associated with this sector - * before overwriting unused sectors. - */ zram_slot_lock(zram, index); - zram_free_page(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); atomic64_inc(&zram->stats.huge_pages); --=20 2.47.1.613.gc27f4b7a9f-goog From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F47C2153C3 for ; Tue, 10 Dec 2024 10:54:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828081; cv=none; b=qX+jjcaLfGp/hZdnEK8vQCLrcFNrgfzLEba3/04ZAqgsZIh0JnUcbHb3l0cfxwaDzyxCFHRNN2dEy0JAv5dlI+SRPA3aE81GEN/hyr+UNhAdl2Qd1ayo0iNZmcc1KDmwC5pdVDzVVBPQElTD/LnYj/05yJbK1E63WgAac4pSx88= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828081; c=relaxed/simple; bh=wx8WBB+LSpRt7NmJATPrFzDkMQWmhlNgUIErDvjyDlQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FDhyhsa8x+XT3SjFvdxsnyCVw0zKiu6vObUcQkx8zqADb2T2g5VIt6TmxMJC40VOGDhUX58RpP/dg85SeerPfvhLgsBhOIh/EF7x+IX3gj8XyBZDf1zQiMpx67V+k1KonINC/OrrorXlKRzX4VqlzITsAxQYS79m9bo9/djGwPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HxTs3O4o; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HxTs3O4o" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-725ee27e905so2181366b3a.2 for ; Tue, 10 Dec 2024 02:54:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828079; x=1734432879; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iKwO52T3yZJ2DDhYKob9Oz5PyVqqD2cyOj11BfMuDq8=; b=HxTs3O4olFPjXNsQmlQGvwinB1CoesiavUQScBHAEnM1M444bHMxjJW+qOCL2t7K3u BLe80EpFysr5sciAxP8k6BYMdJjBJJQCzDwbtRSwqiSNi7hlASd2OSfbbySaqCe+8H/O +hVfkffq6PowWMlc3z8IVggtchV1LwuN9POXw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828079; x=1734432879; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iKwO52T3yZJ2DDhYKob9Oz5PyVqqD2cyOj11BfMuDq8=; b=TjEZYplg+/MANIHdGa6h7Cwikp11OJBi2aa7Y/n0SuZvrvuh/ehEbRu8rR9pUiWL/L W8Hiw/0zAYF2/mkSJnzJprMCYqYKYNdoEL4gLOHo3xErSHm0cIoE2kY1XWub0kkhadnQ kXMYQnBiFr3rq26BlaLuFvMNe+cf7xSaiMl+yqReIB43RAPQdvHPvWGTEjET25owi0kM AEJSApUllvbD9bfX/cQ4DA3eB7CWNbZ4AdhgjDoAsO/QhuG1h2ztAry+MGZZigYk6Gxd d2PQX61uEcFQnqmrMFsX5pADNycmiBJ2W0narnw7w659E0bnKq4i9ZkIak9xMdHME3tf 5sbA== X-Forwarded-Encrypted: i=1; AJvYcCX7U6Le6/LDKsKcIpnTOv9dKMy5xJB1Az64Pp4jacSJT2TusztArqdG4uuMVCyTBcIsPXZt1ZXvt2LUDfo=@vger.kernel.org X-Gm-Message-State: AOJu0YxXqVrq7kfoK+5s4v2UrEWC7SwVP4/qmneXlZUwdy2xpqDHic2B sSwqcsKLX7WB5ccXltzlDy8N3y4Cz/mQV1+gXtp4Qkb4XZ3hWL9kZLF7gkbHpA== X-Gm-Gg: ASbGncuU53kEJQHF0uxJdosKV4tt10a7iZJaQrwLLVYMoTj3l7jOOOlOC2eMdABgShw 6DN53C1zvsvrNstdbQr+bNHW3Tx2biX4MdqoWhJ8JfRGArEAAfZCOTc6YLfi81oZetFBp4S6WOM QBMWxhNPxMuO6R4Ot5/QztkKDcXjfNen4mFse9mrYpJOx0V/Uq20AtLxmNqXDA6EtniB/XrpZ0q /rCL1cz1mb06wnE1OPPfTDwm99p7xpNLN2HssRoNxEB58NRqXFSLVJd X-Google-Smtp-Source: AGHT+IErRXJCOaXLeRUpG4Hl6rEasN4Tvyr0d9JMgPgdvJdwPKXGSzgkjjF0vi2lsxTdmpa1OqeSmQ== X-Received: by 2002:a05:6a00:138a:b0:728:e906:e446 with SMTP id d2e1a72fcca58-728e906e6a4mr346027b3a.24.1733828078853; Tue, 10 Dec 2024 02:54:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-725e4c2defasm4447992b3a.120.2024.12.10.02.54.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 3/6] zram: remove entry element member Date: Tue, 10 Dec 2024 19:53:57 +0900 Message-ID: <20241210105420.1888790-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Element is in the same anon union as handle and hence holds the same value, which makes code below sort of confusing handle =3D zram_get_handle() if (!handle) element =3D zram_get_element() Element doesn't really simplify the code, let's just remove it. We already re-purpose handle to store the block id a written back page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 23 +++++------------------ drivers/block/zram/zram_drv.h | 5 +---- 2 files changed, 6 insertions(+), 22 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 174b4053189b..f68916527846 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -112,17 +112,6 @@ static void zram_clear_flag(struct zram *zram, u32 ind= ex, zram->table[index].flags &=3D ~BIT(flag); } =20 -static inline void zram_set_element(struct zram *zram, u32 index, - unsigned long element) -{ - zram->table[index].element =3D element; -} - -static unsigned long zram_get_element(struct zram *zram, u32 index) -{ - return zram->table[index].element; -} - static size_t zram_get_obj_size(struct zram *zram, u32 index) { return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1); @@ -879,7 +868,7 @@ static ssize_t writeback_store(struct device *dev, =20 zram_free_page(zram, index); zram_set_flag(zram, index, ZRAM_WB); - zram_set_element(zram, index, blk_idx); + zram_set_handle(zram, index, blk_idx); blk_idx =3D 0; atomic64_inc(&zram->stats.pages_stored); spin_lock(&zram->wb_limit_lock); @@ -1502,7 +1491,7 @@ static void zram_free_page(struct zram *zram, size_t = index) =20 if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); - free_block_bdev(zram, zram_get_element(zram, index)); + free_block_bdev(zram, zram_get_handle(zram, index)); goto out; } =20 @@ -1546,12 +1535,10 @@ static int zram_read_from_zspool(struct zram *zram,= struct page *page, =20 handle =3D zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { - unsigned long value; void *mem; =20 - value =3D handle ? zram_get_element(zram, index) : 0; mem =3D kmap_local_page(page); - zram_fill_page(mem, PAGE_SIZE, value); + zram_fill_page(mem, PAGE_SIZE, handle); kunmap_local(mem); return 0; } @@ -1597,7 +1584,7 @@ static int zram_read_page(struct zram *zram, struct p= age *page, u32 index, */ zram_slot_unlock(zram, index); =20 - ret =3D read_from_bdev(zram, page, zram_get_element(zram, index), + ret =3D read_from_bdev(zram, page, zram_get_handle(zram, index), parent); } =20 @@ -1756,7 +1743,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) =20 if (flags) { zram_set_flag(zram, index, flags); - zram_set_element(zram, index, element); + zram_set_handle(zram, index, element); } else { zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 134be414e210..db78d7c01b9a 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -62,10 +62,7 @@ enum zram_pageflags { =20 /* Allocated for each disk page */ struct zram_table_entry { - union { - unsigned long handle; - unsigned long element; - }; + unsigned long handle; unsigned int flags; spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME --=20 2.47.1.613.gc27f4b7a9f-goog From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9129D210F6E for ; Tue, 10 Dec 2024 10:54:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828085; cv=none; b=uTc5BvJezrenYeBiI6KowgXH9sCjNESbKkOKeeJieKtO8HGEgqX7BqXb2m+Dxfz9u+bvu60Cmxm9AlhWd4wURnaTLqkQmLEQMu2ADNknbEKNDWh/ARyJNLyqVhOG3B7ZjiW9bgy4bNhXxzR2sn7Bez2LOnv+ukxvDZjLTIyHLg0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828085; c=relaxed/simple; bh=ZXBlYIaSsAAGDts7TVkJFRHDXa87VVCI/N5cH9om97w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tHBdcnhirHHkzUI8WGXDnEc3Q27o56aHEamzh7uNO3xqV5O4g8LkQ/drsiBEBrxgyKTnM0Moxmb0Caq4wjIuznWeAEmyRNlUq920k4Xc29vBgLn35T2ZkShkfnYrkj4tJAQmkwkYkUcTDUEtpLretN0racgd0DCe2EjsjI9Ok1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=CNeuadsl; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CNeuadsl" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2161eb94cceso24188235ad.2 for ; Tue, 10 Dec 2024 02:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828083; x=1734432883; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5nJ15+aC3WpUqIo+C9NRk5n2cCREumGOutUYIZdyZx0=; b=CNeuadsliFBXGDGsoKl91pq0yIqqPKVfPYCOluz8uuAkQcZW+fotkM2sIFk+ZyvidF bUh41rLpuGJgPz6UDVYm2ISKSE9kOiJJu1T6zAfkSIBWf3iWaHDjYDQlGD0I3RdWMuQN MJufXlJ7mAaDBpYoJX3H+nJzoaf8/8wNOlMV4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828083; x=1734432883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5nJ15+aC3WpUqIo+C9NRk5n2cCREumGOutUYIZdyZx0=; b=WF0vDFCcEQ5edKTemgxzf082f5+IBUckR2LyJjqzP/vjuLtt4B1CAAA2ppg84CxW/4 06K3rSd+Mz9FS0hMKkeHWnhWu6ubmiemwSLehWE2nqVj6uVXad47qiYPu3qzLvEC1IUD NUibBQbJiuWbWruTEcLQIYnp5GUzdx3Dj2nUG4GwrS8lV6N6YG1SZfaFI1B0YnQljmLZ G1JIOeVUuO2cas19gM1lkAHoOIl7N2y6bvkg5D+soUILUwOUpO1A/bfrx4ESRWfhZcf0 TEYsN8tGvadrlxP5Aji1/skMaT2tcO83INYdFNr/HggLSI5QCB+yhHmVAeddGCZ1TyDn Oysg== X-Forwarded-Encrypted: i=1; AJvYcCVVgTK4ZEYhjKcPpdgt3Dt8+lsKTy8BV8zvQO9uK+rP4RUos15eGG8KGSFVranzWJrQBmIA6WeZareVyxs=@vger.kernel.org X-Gm-Message-State: AOJu0YxTuNDI2kpk6SF+5yTKMC5II0J6e2dBBegyAGDwd4+gZ24iglFJ 8pkKOPxqI5e1JNi4B1pREQ2xhKz9m2KcUhhDkCYXvk+EEbHc+Ma7crcE1eAQrw== X-Gm-Gg: ASbGnctTnmgadA6eB7lx69hkSHrbYXzg8Na+ayvuH5ZCJplljbbpH7TgIDG1IFNWlcH nfeIl9MF5PMbIp8RlEfYhrNzGhs09hnw5RqnCLO1xiy10Qn8p3rZaFx2bAhh5yaxvO7yn3Khmdq ZtkA2/XBRsDYmnf7/717GBq+ohACtOR2kdW3FPU+BdRbA+1WOVyGEE92ZEtLTHI7QBAbdOVkQeQ NaSXA4kn3cghYVGTI1daMD8f7LrWWmLnRnUCyrWtvw6j58B/pG/BHLS X-Google-Smtp-Source: AGHT+IE2oedNmDoWlCU5nbMKOQmVbtBNZZvZVd+r8RjHbSMRaGXyNC66mAg+I+IlsrFSqlWIjfLTRA== X-Received: by 2002:a17:902:cf11:b0:216:2a5a:89d3 with SMTP id d9443c01a7336-2162a5a8cedmr157330425ad.25.1733828082922; Tue, 10 Dec 2024 02:54:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21653b0d4f5sm30196395ad.70.2024.12.10.02.54.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 4/6] zram: factor out ZRAM_SAME write Date: Tue, 10 Dec 2024 19:53:58 +0900 Message-ID: <20241210105420.1888790-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Handling of ZRAM_SAME now uses a goto to the final stages of zram_write_page() plus it introduces a branch and flags variable, which is not making the code any simpler. In reality, we can handle ZRAM_SAME immediately when we detect such pages and remove a goto and a branch. Factor out ZRAM_SAME handling into a separate routine to simplify zram_write_page(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 37 ++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f68916527846..22c6ab363ae6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1622,6 +1622,20 @@ static int zram_bvec_read(struct zram *zram, struct = bio_vec *bvec, return zram_read_page(zram, bvec->bv_page, index, bio); } =20 +static int write_same_filled_page(struct zram *zram, unsigned long fill, + u32 index) +{ + zram_slot_lock(zram, index); + zram_set_flag(zram, index, ZRAM_SAME); + zram_set_handle(zram, index, fill); + zram_slot_unlock(zram, index); + + atomic64_inc(&zram->stats.same_pages); + atomic64_inc(&zram->stats.pages_stored); + + return 0; +} + static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; @@ -1632,7 +1646,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) void *src, *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; - enum zram_pageflags flags =3D 0; + bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ zram_slot_lock(zram, index); @@ -1640,14 +1654,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) zram_slot_unlock(zram, index); =20 mem =3D kmap_local_page(page); - if (page_same_filled(mem, &element)) { - kunmap_local(mem); - /* Free memory associated with this sector now. */ - flags =3D ZRAM_SAME; - atomic64_inc(&zram->stats.same_pages); - goto out; - } + same_filled =3D page_same_filled(mem, &element); kunmap_local(mem); + if (same_filled) + return write_same_filled_page(zram, element, index); =20 compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); @@ -1733,7 +1743,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); -out: + zram_slot_lock(zram, index); if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); @@ -1741,13 +1751,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) atomic64_inc(&zram->stats.huge_pages_since); } =20 - if (flags) { - zram_set_flag(zram, index, flags); - zram_set_handle(zram, index, element); - } else { - zram_set_handle(zram, index, handle); - zram_set_obj_size(zram, index, comp_len); - } + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 /* Update stats */ --=20 2.47.1.613.gc27f4b7a9f-goog From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B51B2153F7 for ; Tue, 10 Dec 2024 10:54:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828089; cv=none; b=BFPKPf2A2jK+bqJeG2MZGxx/A5xaEY+6Mk1CrmdPHGKo+EjI7apFHfgydoaU2PM0GzAYAfmebUH+RC+foRRTR3TQ+N1ISVXRG7tlevCqIKh42pIuyUwoK7//iwYLVxOJuBpieDyLgAJnLC70Zgrpro4glaQXqX5DNXm3e5GGhIc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828089; c=relaxed/simple; bh=smdNTw+vMDFHaR4IrLNgF/FN+FFxR5KWARqhFQRNng8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tprQH87Rd9jLp4ZWs3ht/tgjRaY3PzOuJMkUgA2UaN7Da0Lz0zpm25wd52FqAaI7/7Hn3PRiTP0IKXwMeKNHap6zoXnrgBcbDkyAckvpZi86cNV7u5IczpIY6pqT08u/fusV8r5wJpBLnYQb2yrLGuwyS2fOO6ksR8IZoDL2izs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=jxRGvEpu; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="jxRGvEpu" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-7ea8de14848so3353371a12.2 for ; Tue, 10 Dec 2024 02:54:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828087; x=1734432887; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZTnZa6n/DE3ID57RWg7Mxn9pl+7nCgdulUS+gVS34eY=; b=jxRGvEpu4LBeCklThPL6djmj556XVLOZeDLPTDCaZdpI2pP5VIENtUynlTz8GcCiNt Sa0ofqdLiEJCuxgMQLjutuEEPznLVEsdp1rE9qjwtLMJLU5N8lula5rXJ79GSwvr2GX+ FzKkoqjn+X2GQSb0XEvdZpJBHA0SGIEmNX9CE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828087; x=1734432887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZTnZa6n/DE3ID57RWg7Mxn9pl+7nCgdulUS+gVS34eY=; b=POmT5RyO9c2U1KpmSQwPKCHVXW2W8M97zDZvljMFXonFEU6dZBee24vasBrIWy71u8 BgaSAoBTs38kDJ/1XRNUd7Vka2zvsLGjJuK+9ldofnA/cgo2C9nrv/WuwqLvAr9FVRTf 6JfT8yl613WitbzvOQc/xL/Pdp86tHoOn8RotpU8fj/qN1lzbo+YxcnrOWWjK6Eckicq r48ktZRwUGgx9f8zK/xtcERjrap676KzpcO5oMmsQm70CrYikQ/cmThYdFYLiXxkC1m+ 2bnWwnfHAPgmQ9oMAG4F35YWFvyGVI5GMj2vm2x2D644eRCkv3mmIp3URnJZoD45OGYs jb+Q== X-Forwarded-Encrypted: i=1; AJvYcCXScKx38S20MkiVkKWf5tLZHNz28mZ+JbUdJxYOZ2aca7SZwiVEiRhK8y9pZq165b36Bk7UG+bLfEW2mdI=@vger.kernel.org X-Gm-Message-State: AOJu0YyULTb/2se4Uxlg3kfUy4kbhkx97dZ5CwiIse1eB8IwYtEHysph XiEGDfIUxvMsUiWofkvyp0fePktoUDnXA8qr9K2IjyMlYbyesNe3rD9SqH1tYQ== X-Gm-Gg: ASbGncvBH/nlooqyJNAqD12hArsB6OBzfqY1ViHdsgb5iIOxbAcyadWKQF3QzahTZ8c vFaEgxW0LUcD2MqBWKwGcUrWTMf24j42imshHzyzGnjU78wh9Ve9AEsPZ3FDjGIIwUvFcU1UWpH 4NZz0+17ucMuwmq/lh37EMvXIA3Oct2e5e+286pOKaU6pIIWPkRQybl42O/hIUb43/+yUxzLpn5 aHQWFh93OF75ZqlIGy5j8+OKRk4LrAX1Ny2wi18aBZhwHWj5ZW0tw8U X-Google-Smtp-Source: AGHT+IGqA0mE3qai7JdNAnzJ9k3Kt0lR0L33eJet3ghDIIUHhAO2rfR5leTsMMFtRxiqpgC0YeFauw== X-Received: by 2002:a05:6a20:1596:b0:1e1:ae68:d900 with SMTP id adf61e73a8af0-1e1b1b1bb34mr7365110637.22.1733828086890; Tue, 10 Dec 2024 02:54:46 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7fd4846418csm3871862a12.28.2024.12.10.02.54.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:46 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 5/6] zram: factor out ZRAM_HUGE write Date: Tue, 10 Dec 2024 19:53:59 +0900 Message-ID: <20241210105420.1888790-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zram_write_page() handles: ZRAM_SAME pages (which was already factored out) stores, regular page stores and ZRAM_HUGE pages stores. ZRAM_HUGE handling adds a significant amount of complexity. Instead, we can handle ZRAM_HUGE in a separate function. This allows us to simplify zs_handle allocations slow-path, as it now does not handle ZRAM_HUGE case. ZRAM_HUGE zs_handle allocation, on the other hand, can now drop __GFP_KSWAPD_RECLAIM because we handle ZRAM_HUGE in preemptible context (outside of local-lock scope). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 139 ++++++++++++++++++++-------------- 1 file changed, 83 insertions(+), 56 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 22c6ab363ae6..18263e4c208e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -132,6 +132,27 @@ static inline bool zram_allocated(struct zram *zram, u= 32 index) zram_test_flag(zram, index, ZRAM_WB); } =20 +static inline void update_used_max(struct zram *zram, const unsigned long = pages) +{ + unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); + + do { + if (cur_max >=3D pages) + return; + } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, + &cur_max, pages)); +} + +static bool zram_can_store_page(struct zram *zram) +{ + unsigned long alloced_pages; + + alloced_pages =3D zs_get_total_pages(zram->mem_pool); + update_used_max(zram, alloced_pages); + + return !zram->limit_pages || alloced_pages <=3D zram->limit_pages; +} + #if PAGE_SIZE !=3D 4096 static inline bool is_partial_io(struct bio_vec *bvec) { @@ -266,18 +287,6 @@ static struct zram_pp_slot *select_pp_slot(struct zram= _pp_ctl *ctl) } #endif =20 -static inline void update_used_max(struct zram *zram, - const unsigned long pages) -{ - unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); - - do { - if (cur_max >=3D pages) - return; - } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, - &cur_max, pages)); -} - static inline void zram_fill_page(void *ptr, unsigned long len, unsigned long value) { @@ -1636,14 +1645,55 @@ static int write_same_filled_page(struct zram *zram= , unsigned long fill, return 0; } =20 +static int write_incompressible_page(struct zram *zram, struct page *page, + u32 index) +{ + unsigned long handle; + void *src, *dst; + + /* + * This function is called from preemptible context so we don't need + * to do optimistic and fallback to pessimistic handle allocation, + * like we do for compressible pages. + */ + handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle)) + return PTR_ERR((void *)handle); + + if (!zram_can_store_page(zram)) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zs_free(zram->mem_pool, handle); + return -ENOMEM; + } + + dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + src =3D kmap_local_page(page); + memcpy(dst, src, PAGE_SIZE); + kunmap_local(src); + zs_unmap_object(zram->mem_pool, handle); + + zram_slot_lock(zram, index); + zram_set_flag(zram, index, ZRAM_HUGE); + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, PAGE_SIZE); + zram_slot_unlock(zram, index); + + atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); + atomic64_inc(&zram->stats.huge_pages); + atomic64_inc(&zram->stats.huge_pages_since); + atomic64_inc(&zram->stats.pages_stored); + + return 0; +} + static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long alloced_pages; unsigned long handle =3D -ENOMEM; unsigned int comp_len =3D 0; unsigned int last_comp_len =3D 0; - void *src, *dst, *mem; + void *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; bool same_filled; @@ -1661,10 +1711,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) =20 compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); - src =3D kmap_local_page(page); + mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, - src, &comp_len); - kunmap_local(src); + mem, &comp_len); + kunmap_local(mem); =20 if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); @@ -1673,13 +1723,16 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return ret; } =20 - if (comp_len >=3D huge_class_size) - comp_len =3D PAGE_SIZE; - if (last_comp_len && (last_comp_len !=3D comp_len)) { zs_free(zram->mem_pool, handle); handle =3D -ENOMEM; } + + if (comp_len >=3D huge_class_size) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + return write_incompressible_page(zram, page, index); + } + /* * handle allocation has 2 paths: * a) fast path is executed with preemption disabled (for @@ -1695,37 +1748,22 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) */ if (IS_ERR_VALUE(handle)) handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + __GFP_KSWAPD_RECLAIM | + __GFP_NOWARN | + __GFP_HIGHMEM | + __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) return PTR_ERR((void *)handle); =20 - if (comp_len !=3D PAGE_SIZE) { - last_comp_len =3D comp_len; - goto compress_again; - } - /* - * If the page is not compressible, you need to acquire the - * lock and execute the code below. The zcomp_stream_get() - * call is needed to disable the cpu hotplug and grab the - * zstrm buffer back. It is necessary that the dereferencing - * of the zstrm variable below occurs correctly. - */ - zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); + goto compress_again; } =20 - alloced_pages =3D zs_get_total_pages(zram->mem_pool); - update_used_max(zram, alloced_pages); - - if (zram->limit_pages && alloced_pages > zram->limit_pages) { + if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_free(zram->mem_pool, handle); return -ENOMEM; @@ -1733,30 +1771,19 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) =20 dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 - src =3D zstrm->buffer; - if (comp_len =3D=3D PAGE_SIZE) - src =3D kmap_local_page(page); - memcpy(dst, src, comp_len); - if (comp_len =3D=3D PAGE_SIZE) - kunmap_local(src); - + memcpy(dst, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); - atomic64_add(comp_len, &zram->stats.compr_data_size); =20 zram_slot_lock(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { - zram_set_flag(zram, index, ZRAM_HUGE); - atomic64_inc(&zram->stats.huge_pages); - atomic64_inc(&zram->stats.huge_pages_since); - } - zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 /* Update stats */ atomic64_inc(&zram->stats.pages_stored); + atomic64_add(comp_len, &zram->stats.compr_data_size); + return ret; } =20 --=20 2.47.1.613.gc27f4b7a9f-goog From nobody Sun Dec 14 13:56:14 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11356212D84 for ; Tue, 10 Dec 2024 10:54:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828092; cv=none; b=tRpuixcCa68eI7/CuwVMesoFMipSpXqtwdPH2zOILthc2qq6Uj1dMCUmrSAswqFwiV2511DJJpqbp4FsDaWNnOfkcBU8IxGbfharhIikEV/zCLSb9Xa/NDwPvKRJaaQSWuUJSkaCN/Bmy2k29fnYo/zLLF1a2IcxJ/fxZQYW7qw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733828092; c=relaxed/simple; bh=CEN/kcVfPFI0hWfQlsFCYytcqPd64w8A0GpCq1HazDU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j3UF6wBvpBpNKLdSsSBhq56yF3U8Y9MXLsLuMk+qjf7kU0w55Q3+Oac2ajlAWFlG7Q/m+aFpZTvoFHIjun42ptgYWSAtZOZObRtSZOXpv+WDr+40NQwcTJ4g3crU7a91SpfwGRxYm47/6xLZGQsSPWnXXfYmM9C855q/C4L4+z0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=DA6Yt8B4; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DA6Yt8B4" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7fbbe0fb0b8so3806413a12.0 for ; Tue, 10 Dec 2024 02:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1733828090; x=1734432890; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=S4tnLIiWtc+PvEy1ThvlwDe1w/SkjAwqNX6TrEifnoo=; b=DA6Yt8B4ylGxxFPLwgXadQZ9YGVDn+sydPUcP9CADgXuSFG2rAjiCuDUPHYFvlHj1t 47T2F970xTJh+bQRiYIZOvYEdElDRgxm1HVD5HTA5kyR1JWMeHrf0tROzP7u8UmVP8Dx gXJm1m8KwuYLX0kZGpE0q6Nd0IviYRPCV7P9A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733828090; x=1734432890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S4tnLIiWtc+PvEy1ThvlwDe1w/SkjAwqNX6TrEifnoo=; b=dSi7QnYCpE3KCFKjHi3sZ91FMy/6wjxK9J5dgLU2KwM1dql47xwiRcHjpOTuarDQ4G wT6A5M1mW/PGmbEO6MrAGnzcilNSMLK+CTC9eJ5mWGOb917anXd29EVHuSKQNc3YPhsk tZtoQTNe63FXGuYfN9qxyJfZxd7igYCd/gDcB8g0Amz/N+0w5cZ7f2e9ol3Bt2Gsfjnw /E1vnHV7QtbplWSvL76XxgoE+xn5KZKHVOCynzroh9TfLmVuFmBT2a7WrRDNXENFzO5q cQKbzZN7s4Wj4AWT+srCyHf/m53Rw548Gv/pmgU5UQAvrirE73qjHCybQjuej+v547sb ncXQ== X-Forwarded-Encrypted: i=1; AJvYcCU2hxDEOjIu+UFGS7gccw4PpyMrNEG6Ma2C43DeWtrC81A9V4JwiJq9SfkCBkdCsg3KRMKg2k0/OMFbOYA=@vger.kernel.org X-Gm-Message-State: AOJu0YzoFWKbY3DGDL2vGa39XE9UIhS4ylPHJ36DgqYZdQpbNRnr9jH3 8CTCSYjk42b/T1acnv2uJlWmc/8SL5/x2/x3ItDiLXA/35Sck7WonM3JZtJN1A== X-Gm-Gg: ASbGncsVDfCPwhyzTcFgDc0TvLaMqm+71JNXVBhFseAY0UyXvMKkcXHs3CSTcWD0BHs CVo7MhxjBdVxBphRkkyYozGPIA+ouxgBOrQu8hav0MxY+sYszxhsG0LEIlkZfmM1KmRwji9Am4d qhAPnsrByr7G/L0TAFAvtZ4XfaxgzwtI6DErkImxxCoALVyw/J9YXU7VozBl5tPuzG3p7drjR8l BJ5DztxsStjre/6s3yLOCCfBl7JAFFqAsljLIe/NUr+ZVNS/CekG427 X-Google-Smtp-Source: AGHT+IEtUH17pL5ZO8X4frDDTYwgT73IZVyYi0OGKQpyREJpE6VTQ3CiShhX4tdqrx/2yoLwt/OWVA== X-Received: by 2002:a05:6a20:4322:b0:1e1:b0b7:b059 with SMTP id adf61e73a8af0-1e1b1a7512emr5388144637.3.1733828090489; Tue, 10 Dec 2024 02:54:50 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4d97:9dbf:1a3d:bc59]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-725e701c042sm4140803b3a.200.2024.12.10.02.54.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Dec 2024 02:54:50 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 6/6] zram: factor out different page types read Date: Tue, 10 Dec 2024 19:54:00 +0900 Message-ID: <20241210105420.1888790-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog In-Reply-To: <20241210105420.1888790-1-senozhatsky@chromium.org> References: <20241210105420.1888790-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similarly to write, split the page read code into ZRAM_HUGE read, ZRAM_SAME read and compressed page read to simplify the code. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 85 +++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 33 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 18263e4c208e..e4a7191ec13c 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1528,54 +1528,73 @@ static void zram_free_page(struct zram *zram, size_= t index) zram_set_obj_size(zram, index, 0); } =20 -/* - * Reads (decompresses if needed) a page from zspool (zsmalloc). - * Corresponding ZRAM slot should be locked. - */ -static int zram_read_from_zspool(struct zram *zram, struct page *page, +static int read_same_filled_page(struct zram *zram, struct page *page, u32 index) { - struct zcomp_strm *zstrm; + void *mem; + + mem =3D kmap_local_page(page); + zram_fill_page(mem, PAGE_SIZE, zram_get_handle(zram, index)); + kunmap_local(mem); + return 0; +} + +static int read_incompressible_page(struct zram *zram, struct page *page, + u32 index) +{ unsigned long handle; - unsigned int size; void *src, *dst; - u32 prio; - int ret; =20 handle =3D zram_get_handle(zram, index); - if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { - void *mem; + src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + dst =3D kmap_local_page(page); + copy_page(dst, src); + kunmap_local(dst); + zs_unmap_object(zram->mem_pool, handle); =20 - mem =3D kmap_local_page(page); - zram_fill_page(mem, PAGE_SIZE, handle); - kunmap_local(mem); - return 0; - } + return 0; +} =20 - size =3D zram_get_obj_size(zram, index); +static int read_compressed_page(struct zram *zram, struct page *page, u32 = index) +{ + struct zcomp_strm *zstrm; + unsigned long handle; + unsigned int size; + void *src, *dst; + int ret, prio; =20 - if (size !=3D PAGE_SIZE) { - prio =3D zram_get_priority(zram, index); - zstrm =3D zcomp_stream_get(zram->comps[prio]); - } + handle =3D zram_get_handle(zram, index); + size =3D zram_get_obj_size(zram, index); + prio =3D zram_get_priority(zram, index); =20 + zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); - if (size =3D=3D PAGE_SIZE) { - dst =3D kmap_local_page(page); - copy_page(dst, src); - kunmap_local(dst); - ret =3D 0; - } else { - dst =3D kmap_local_page(page); - ret =3D zcomp_decompress(zram->comps[prio], zstrm, - src, size, dst); - kunmap_local(dst); - zcomp_stream_put(zram->comps[prio]); - } + dst =3D kmap_local_page(page); + ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); + kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); + zcomp_stream_put(zram->comps[prio]); + return ret; } =20 +/* + * Reads (decompresses if needed) a page from zspool (zsmalloc). + * Corresponding ZRAM slot should be locked. + */ +static int zram_read_from_zspool(struct zram *zram, struct page *page, + u32 index) +{ + if (zram_test_flag(zram, index, ZRAM_SAME) || + !zram_get_handle(zram, index)) + return read_same_filled_page(zram, page, index); + + if (!zram_test_flag(zram, index, ZRAM_HUGE)) + return read_compressed_page(zram, page, index); + else + return read_incompressible_page(zram, page, index); +} + static int zram_read_page(struct zram *zram, struct page *page, u32 index, struct bio *parent) { --=20 2.47.1.613.gc27f4b7a9f-goog