From nobody Sat Nov 30 04:27:16 2024 Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F6DA189F2A for ; Thu, 12 Sep 2024 07:04:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124666; cv=none; b=ox5TWyJacWhmoF8Beo/Hnv1k+rZ/b+w5f5sn1HlzS9eHIQ7d2omO0rTd7KSyRkGgdxobafKELvZ62tSKxXVjq5oxIfnUzZhGzcKlXbrDZB3pV+uQkjQtEa8+gBmR3Ej51ebZ7GBoeU+5skuQGygS8Ydw/jo4E/7K0RMWHhV/wnU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124666; c=relaxed/simple; bh=cMrFZFeYshEhBFOfKSSKpG6P8VFIuhAwYxrlWBPSmzs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tErpTQUPDtRZF8uPvK8XVAhY90LkCtIb6bhdH7lEBeAHui+V1AypFAyM8RCeir6OGBadcIlJ6ZtFa9CfZG7064WmtiTJ/O+vMCjmdgeZxGD4F93UJCNiRZRIvk2jUBh4fprTn+zFmghnuMB5IkR7oP6ja6f756BlsKo3fkgEiGU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=R8HULKJZ; arc=none smtp.client-ip=209.85.160.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="R8HULKJZ" Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-277efdcfa63so269872fac.2 for ; Thu, 12 Sep 2024 00:04:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124663; x=1726729463; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AwMu+l4YrL6K6neHxIk3gwGHEtDDHlzEJ+d/igKF6jE=; b=R8HULKJZCjJOh+d0J4jYm2LPlvnUN/TlvXO9vIAZMOuZWQ3i5RdbAagfPQXjMgfq+V pkNrqSQzPGF0U7mMloYKQnW2JiY6ND+KmdX7xeMfnD17WRK/4fHTc8r4+Zqvzvdj7I/3 UnM30pd0IaGbHw8fSEo3fgeP4DMUOAA8y63JI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124663; x=1726729463; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AwMu+l4YrL6K6neHxIk3gwGHEtDDHlzEJ+d/igKF6jE=; b=GIlHPaEWIQ/QOfUBo/69ZJY++IrAMQ0XTSXkaksZoqqNKFf148uzGFUScVvELO8B+g /G32TJAHUMWD7kLZRz/KxWfO55lQKbRrPaJsktyTHDyCZEnXxHS++E3cuxO+t94z1pfY QYeRYOzEcROk6c20NZynTc6VP+7rixNtstls670/tIxMjzqbPSwr9WVCGUa1Ojzocgc2 YWtzb0RLXurtMqNBvT/UvogNbYeSxGE3TzI01NN7seKGMh8uPhJr6+sTPokB02tvARJD Ou76oOT5vnVRvfzxZuBTRDgR/hMtdGKchq3DGt0VZO1uznDkBdrFFBbcpc1smWXXVxU6 sYIQ== X-Gm-Message-State: AOJu0YyyVDeB5t+G5xV15TcVSNrPAflcgdmey7GHJJK3SX6ecsSDqIym mnBROiuEXaK1kCI6UD2UdZFuzDOsYVVdEuF/4VZlNXNcSyj0QYnX2+rscqYCdQ== X-Google-Smtp-Source: AGHT+IGwcigeArG87HZCWXsUWOzefcpatw1EYMe7A1pLPrG4o9yaF5072wBhWhAl9kCLBq1zno9Uuw== X-Received: by 2002:a05:6870:1606:b0:273:da6b:9a3d with SMTP id 586e51a60fabf-27c3f313a5amr984354fac.22.1726124663688; Thu, 12 Sep 2024 00:04:23 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:23 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 1/6] zram: introduce ZRAM_PP_SLOT flag Date: Thu, 12 Sep 2024 16:02:16 +0900 Message-ID: <20240912070413.4179924-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This flag indicates that the slot was selected as a candidate slot for post-processing (pp) and was assigned to a pp bucket. It does not necessarily mean that the slot is currently under post-processing, but may mean so. The slot can loose its PP_SLOT flag, while still being in the pp-bucket, if it's accessed or slot_free-ed. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 2 ++ drivers/block/zram/zram_drv.h | 1 + 2 files changed, 3 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ee2a279c5f25..d170bf6cdcd8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -178,6 +178,7 @@ static inline u32 zram_get_priority(struct zram *zram, = u32 index) static void zram_accessed(struct zram *zram, u32 index) { zram_clear_flag(zram, index, ZRAM_IDLE); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D ktime_get_boottime(); #endif @@ -1354,6 +1355,7 @@ static void zram_free_page(struct zram *zram, size_t = index) zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); =20 zram_set_priority(zram, index, 0); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); =20 if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index cfc8c059db63..914cb6629969 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -48,6 +48,7 @@ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_UNDER_WB, /* page is under writeback */ + ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ ZRAM_IDLE, /* not accessed page since last idle marking */ ZRAM_INCOMPRESSIBLE, /* none of the algorithms could compress it */ --=20 2.46.0.598.g6f2099f65c-goog From nobody Sat Nov 30 04:27:16 2024 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9DF718C920 for ; Thu, 12 Sep 2024 07:04:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124670; cv=none; b=J2wLdeeknwq8BOATCKOhRisVitXNLbTB9b28B2CyDbpCllgBFbqwL85X4b3TRBNX3d/t5+FT7ge1guxbI/m7NPL78OzUjonA0Pxk0WGYLfb2+D4V1re1FwhFWKF6NHsnAVia6G/KfdbvM6vrS8aV4o5VHVbLLQ4G1ka69iMHKBo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124670; c=relaxed/simple; bh=tJIwMwWYOHuMMebMB2tQEB+hQA+Iobdc9qsuNS6ewxo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ks6rkAT3eukamEa2thhH3+OrQGARLBHWEnPiG9H4LcUxWWVvk0t960N4jD39R3dupAIQFzYAGQJESy7zSASmIGju0IjZRAMzm6nAMhpE1YW55Y1jfM67wUt+64NleQtDfkI/Az6grhI0cy0tV1rqp9D5nCyCooKYV82XfWu5lU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=TGV4Zc/R; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="TGV4Zc/R" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-71781f42f75so470251b3a.1 for ; Thu, 12 Sep 2024 00:04:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124668; x=1726729468; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ivRYctFtAHjZnxxMEveMODYRdfT+M4UTh4BuUvLN5kA=; b=TGV4Zc/R71zknBu7IEgmo1CvOoo3Tyr6Pv6F5eSUW6jP1cQBD0U6miBq5xwb424VhN gEREuYBi50qCG2Q989Y8pI6VU3mRV2Ego2mVbwmHCG6WXIgTEYT0hV+2Tm0lcMcCA8ua wvRbtHK/YmbDP1HTHBb8HrIgCBjyzcGzCASTM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124668; x=1726729468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ivRYctFtAHjZnxxMEveMODYRdfT+M4UTh4BuUvLN5kA=; b=XX+GAnK8UEeRbAgWVsjR9nhWZVVJtSZnHwwDRu1Kt0XJrrCr/unkjPLzLcy7rAoOwl G4Qa+DZYm5j41ikn3TglfSIa/78+UDx5KMr3xCppI7O/dL2s0bPJl4iaIt0b6HwSXfTs SYdlPqSG8q2m61cEQQAvm7jGx2sp24G7JRh0tnu0/PFbo9YIVna0s8oD9wchaTUHqqBB V7P5ermD8MTwtEzGN6GcTOZtKt/wGaP6Lvp3vqTCr0n5bCBUgTFi3cMdS//jLhKLfKR0 WFt813EcGLImr2G50nYH7MgSzKq1QQJg6fV0MvTLY/DRT+PAC2j5Q8YTYwhGrJeZZWVn YteQ== X-Gm-Message-State: AOJu0YyD2tDc/VtRK6tTK/oVb+Gl4l7by+1agpxc0CF1JKgcAEPy+QKx E40qcaS8A/FOjFeX14sMcFIwMQ5YXsAW3AY4ZL44lDOq36wBsDNnVx64BjPATA== X-Google-Smtp-Source: AGHT+IEQoVR8mYge+xgG6bMMFiRWvL7ZNtijIlLe+Em8CHNjt4EJO+5x9k12b2SUxBys0IPLfx3luw== X-Received: by 2002:a05:6a20:3f24:b0:1cf:7bc0:3e14 with SMTP id adf61e73a8af0-1cf7bc045a1mr1007889637.17.1726124668127; Thu, 12 Sep 2024 00:04:28 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:27 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 2/6] zram: permit only one post-processing operation at a time Date: Thu, 12 Sep 2024 16:02:17 +0900 Message-ID: <20240912070413.4179924-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both recompress and writeback soon will unlock slots during processing, which makes things too complex wrt possible race-conditions. We still want to clear PP_SLOT in slot_free, because this is how we figure out that slot that was selected for post-processing has been released under us and when we start post-processing we check if slot still has PP_SLOT set. At the same time, theoretically, we can have something like this: CPU0 CPU1 recompress scan slots set PP_SLOT unlock slot slot_free clear PP_SLOT allocate PP_SLOT writeback scan slots set PP_SLOT unlock slot select PP-slot test PP_SLOT So recompress will not detect that slot has been re-used and re-selected for concurrent writeback post-processing. Make sure that we only permit on post-processing operation at a time. So now recompress and writeback post-processing don't race against each other, we only need to handle slot re-use (slot_free and write), which is handled individually by each pp operation. Having recompress and writeback competing for the same slots is not exactly good anyway (can't imagine anyone doing that). Signed-off-by: Sergey Senozhatsky --- Documentation/admin-guide/blockdev/zram.rst | 2 ++ drivers/block/zram/zram_drv.c | 16 ++++++++++++++++ drivers/block/zram/zram_drv.h | 1 + 3 files changed, 19 insertions(+) diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 678d70d6e1c3..714a5171bfc0 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -47,6 +47,8 @@ The list of possible return codes: -ENOMEM zram was not able to allocate enough memory to fulfil your needs. -EINVAL invalid input has been provided. +-EAGAIN re-try operation later (e.g. when attempting to run recompress + and writeback simultaneously). =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 If you use 'echo', the returned value is set by the 'echo' utility, diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d170bf6cdcd8..37622268104e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -627,6 +627,12 @@ static ssize_t writeback_store(struct device *dev, goto release_init_lock; } =20 + /* Do not permit concurrent post-processing actions. */ + if (atomic_xchg(&zram->pp_in_progress, 1)) { + up_read(&zram->init_lock); + return -EAGAIN; + } + if (!zram->backing_dev) { ret =3D -ENODEV; goto release_init_lock; @@ -753,6 +759,7 @@ static ssize_t writeback_store(struct device *dev, free_block_bdev(zram, blk_idx); __free_page(page); release_init_lock: + atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); =20 return ret; @@ -1883,6 +1890,12 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 + /* Do not permit concurrent post-processing actions. */ + if (atomic_xchg(&zram->pp_in_progress, 1)) { + up_read(&zram->init_lock); + return -EAGAIN; + } + if (algo) { bool found =3D false; =20 @@ -1950,6 +1963,7 @@ static ssize_t recompress_store(struct device *dev, __free_page(page); =20 release_init_lock: + atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); return ret; } @@ -2146,6 +2160,7 @@ static void zram_reset_device(struct zram *zram) zram->disksize =3D 0; zram_destroy_comps(zram); memset(&zram->stats, 0, sizeof(zram->stats)); + atomic_set(&zram->pp_in_progress, 0); reset_bdev(zram); =20 comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); @@ -2383,6 +2398,7 @@ static int zram_add(void) zram->disk->fops =3D &zram_devops; zram->disk->private_data =3D zram; snprintf(zram->disk->disk_name, 16, "zram%d", device_id); + atomic_set(&zram->pp_in_progress, 0); =20 /* Actual capacity set using sysfs (/sys/block/zram/disksize */ set_capacity(zram->disk, 0); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 914cb6629969..73a9d47d76ba 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -140,5 +140,6 @@ struct zram { #ifdef CONFIG_ZRAM_MEMORY_TRACKING struct dentry *debugfs_dir; #endif + atomic_t pp_in_progress; }; #endif --=20 2.46.0.598.g6f2099f65c-goog From nobody Sat Nov 30 04:27:16 2024 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06F6E18DF62 for ; Thu, 12 Sep 2024 07:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124673; cv=none; b=fYzh2Ilff25LtoP9IY80VzoF3uGdWfBjSRh5xJ63QMVCI7ptRH6xhjf+vZeWKJMeDITCEga2mMfi+2g+7ApEPwVJ1JyyYPA5MgC+gHD3sbMiXwAxI4POa+dmlNe1gJEJhv3/HTMrSF1ONeQ7iyXj6S87EA8vfr6GzRAD8O/ajQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124673; c=relaxed/simple; bh=BO28K1tejO2J8iIccf1acOz7SJT41KgUnTvtCvkdYbI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fz4fLgvpdF/XBgiiICYCwsHVPI11Tv7w3347HI4OXlenuivXRLAnseKaI2eWkGhjJA0sMXbMt7gj7iC1YuOoAXpkURMU+ZL7swYSlF8QcvaOg3msQhS1G2oFfjuLSJYtg5BSX9j66HgL+4joGOQsCRgBDiQ393UkWqJkTlU0GqE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=DWj4fNhr; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DWj4fNhr" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-7db1f13b14aso608309a12.1 for ; Thu, 12 Sep 2024 00:04:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124671; x=1726729471; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eOqqiFl7+PSHLNk87BV7Qg9ZI8tpG2YDnz3wr/rQglc=; b=DWj4fNhrtANFNQG+wM6T+uSaTnsGV5SztjwhyEiWbYXqRUTtAKzvgW20SJ/CFpbEXT Xg6DvWOTJIM7WPHi5yj8xr6nq4S9CSBFdWZbEwR5TGxfSXXMlUBkbcAxpBrJ3/JPxQX/ CHmfPQ4uCL1EpEWrI0h81g02aIseJ5bSyBjCM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124671; x=1726729471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eOqqiFl7+PSHLNk87BV7Qg9ZI8tpG2YDnz3wr/rQglc=; b=T1lCbpebaOUdsl5JeQGm+pQtVNEmmLvneSGRcv41z6ipaiAfwo2jIJNDz4NkGhGP/3 YmZdbMqBomVKCIifAEuv/DnII04JsQLVGIw1K7nebf6hpvxBVAyFQJH0sTRgl4e4ctEC x/y5FhHk7qZN8O329ceUESAyztl57QwgO5VPX1Z9gTU87UkMOgdXQSZUnnvPvM+t6Fd6 RveWBJLYrqRNQ9d4diB4J+Qm5XTBpg4GpQwOqbHr16xN0LmXPByqAOn9Mh+WyM1IQvdd T782DOT8W+FHB7JdOj3bkf6i6ec1lp2s7PSycPEdDvBZhNSzKXV2mTBZRoiZjpEJHpQh zYHg== X-Gm-Message-State: AOJu0Yy2/VXQO6jqhJZR3KDA6sGfIHFcuJqmvimegqbRK56e4yEjqGj1 Nc2LnZueF0j/aZdU+OFxVIeAS31YQVYj3nzb3HhLGaZWWUoXzd10LCvkENwq9/1VWVjaUM27p+S Dfw== X-Google-Smtp-Source: AGHT+IHgHHJJpvWkXs+lLUw6m3794dcKU0lYPZPpF8x2WWIgyhnyYBGEV/+2pMGcMkUcGk0f30KHjw== X-Received: by 2002:a05:6a20:7b07:b0:1cf:d746:22d0 with SMTP id adf61e73a8af0-1cfd7463553mr842276637.1.1726124671131; Thu, 12 Sep 2024 00:04:31 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:30 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 3/6] zram: rework recompress target selection strategy Date: Thu, 12 Sep 2024 16:02:18 +0900 Message-ID: <20240912070413.4179924-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Target slot selection for recompression is just a simple iteration over zram->table entries (stored pages) from slot 0 to max slot. Given that zram->table slots are written in random order and are not sorted by size, a simple iteration over slots selects suboptimal targets for recompression. This is not a problem if we recompress every single zram->table slot, but we never do that in reality. In reality we limit the number of slots we can recompress (via max_pages parameter) and hence proper slot selection becomes very important. The strategy is quite simple, suppose we have two candidate slots for recompression, one of size 48 bytes and one of size 2800 bytes, and we can recompress only one, then it certainly makes more sense to pick 2800 entry for recompression. Because even if we manage to compress 48 bytes objects even further the savings are going to be very small. Potential savings after good re-compression of 2800 bytes objects are much higher. This patch reworks slot selection and introduces the strategy described above: among candidate slots always select the biggest ones first. For that the patch introduces zram_pp_ctl (post-processing) structure which holds NUM_PP_BUCKETS pp buckets of slots. Slots are assigned to a particular group based on their sizes - the larger the size of the slot the higher the group index. This, basically, sorts slots by size in liner time (we still perform just one iteration over zram->table slots). When we select slot for recompression we always first lookup in higher pp buckets (those that hold the largest slots). Which achieves the desired behavior. TEST =3D=3D=3D=3D A very simple demonstration: zram is configured with zstd, and zstd with dict as a recompression stream. A limited (max 4096 pages) recompression is performed then, with a log of sizes of slots that were recompressed. You can see that patched zram selects slots for recompression in significantly different manner, which leads to higher memory savings (see column #2 of mm_stat output). BASE ---- *** initial state of zram device /sys/block/zram0/mm_stat 1750994944 504491413 514203648 0 514203648 1 0 3420= 4 34204 *** recompress idle max_pages=3D4096 /sys/block/zram0/mm_stat 1750994944 504262229 514953216 0 514203648 1 0 3420= 4 34204 Sizes of selected objects for recompression: ... 45 58 24 226 91 40 24 24 24 424 2104 93 2078 2078 2078 959 154 ... PATCHED ------- *** initial state of zram device /sys/block/zram0/mm_stat 1750982656 504492801 514170880 0 514170880 1 0 3420= 4 34204 *** recompress idle max_pages=3D4096 /sys/block/zram0/mm_stat 1750982656 503716710 517586944 0 514170880 1 0 3420= 4 34204 Sizes of selected objects for recompression: ... 3680 3694 3667 3590 3614 3553 3537 3548 3550 3542 3543 3537 ... Note, pp-slots are not strictly sorted, there is a PP_BUCKET_SIZE_RANGE variation of sizes within particular bucket. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 187 +++++++++++++++++++++++++++++----- 1 file changed, 160 insertions(+), 27 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 37622268104e..6688f70b2140 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -184,6 +184,99 @@ static void zram_accessed(struct zram *zram, u32 index) #endif } =20 +#ifdef CONFIG_ZRAM_MULTI_COMP +struct zram_pp_slot { + unsigned long index; + struct list_head entry; +}; + +/* + * A post-processing bucket is, essentially, a size class, this defines + * the range (in bytes) of pp-slots sizes in particular bucket. + */ +#define PP_BUCKET_SIZE_RANGE 64 +#define NUM_PP_BUCKETS ((PAGE_SIZE / PP_BUCKET_SIZE_RANGE) + 1) + +struct zram_pp_ctl { + struct list_head pp_buckets[NUM_PP_BUCKETS]; +}; + +static struct zram_pp_ctl *init_pp_ctl(void) +{ + struct zram_pp_ctl *ctl; + u32 idx; + + ctl =3D kmalloc(sizeof(*ctl), GFP_KERNEL); + if (!ctl) + return NULL; + + for (idx =3D 0; idx < NUM_PP_BUCKETS; idx++) + INIT_LIST_HEAD(&ctl->pp_buckets[idx]); + return ctl; +} + +static void release_pp_slot(struct zram *zram, struct zram_pp_slot *pps) +{ + list_del_init(&pps->entry); + + zram_slot_lock(zram, pps->index); + zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); + zram_slot_unlock(zram, pps->index); + + kfree(pps); +} + +static void release_pp_ctl(struct zram *zram, struct zram_pp_ctl *ctl) +{ + u32 idx; + + if (!ctl) + return; + + for (idx =3D 0; idx < NUM_PP_BUCKETS; idx++) { + while (!list_empty(&ctl->pp_buckets[idx])) { + struct zram_pp_slot *pps; + + pps =3D list_first_entry(&ctl->pp_buckets[idx], + struct zram_pp_slot, + entry); + release_pp_slot(zram, pps); + } + } + + kfree(ctl); +} + +static void place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, + struct zram_pp_slot *pps) +{ + u32 idx; + + idx =3D zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; + list_add(&pps->entry, &ctl->pp_buckets[idx]); + + zram_set_flag(zram, pps->index, ZRAM_PP_SLOT); +} + +static struct zram_pp_slot *select_pp_slot(struct zram_pp_ctl *ctl) +{ + struct zram_pp_slot *pps =3D NULL; + s32 idx =3D NUM_PP_BUCKETS - 1; + + /* The higher the bucket id the more optimal slot post-processing is */ + while (idx > 0) { + pps =3D list_first_entry_or_null(&ctl->pp_buckets[idx], + struct zram_pp_slot, + entry); + if (pps) + break; + + idx--; + } + return pps; +} +#endif + static inline void update_used_max(struct zram *zram, const unsigned long pages) { @@ -1657,6 +1750,52 @@ static int zram_bvec_write(struct zram *zram, struct= bio_vec *bvec, } =20 #ifdef CONFIG_ZRAM_MULTI_COMP +#define RECOMPRESS_IDLE (1 << 0) +#define RECOMPRESS_HUGE (1 << 1) + +static int scan_slots_for_recompress(struct zram *zram, u32 mode, + struct zram_pp_ctl *ctl) +{ + unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; + struct zram_pp_slot *pps =3D NULL; + unsigned long index; + + for (index =3D 0; index < nr_pages; index++) { + if (!pps) + pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); + if (!pps) + return -ENOMEM; + + INIT_LIST_HEAD(&pps->entry); + + zram_slot_lock(zram, index); + if (!zram_allocated(zram, index)) + goto next; + + if (mode & RECOMPRESS_IDLE && + !zram_test_flag(zram, index, ZRAM_IDLE)) + goto next; + + if (mode & RECOMPRESS_HUGE && + !zram_test_flag(zram, index, ZRAM_HUGE)) + goto next; + + if (zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_SAME) || + zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + goto next; + + pps->index =3D index; + place_pp_slot(zram, ctl, pps); + pps =3D NULL; +next: + zram_slot_unlock(zram, index); + } + + kfree(pps); + return 0; +} + /* * This function will decompress (unless it's ZRAM_HUGE) the page and then * attempt to compress it using provided compression algorithm priority @@ -1664,7 +1803,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, * * Corresponding ZRAM slot should be locked. */ -static int zram_recompress(struct zram *zram, u32 index, struct page *page, +static int recompress_slot(struct zram *zram, u32 index, struct page *page, u64 *num_recomp_pages, u32 threshold, u32 prio, u32 prio_max) { @@ -1807,20 +1946,17 @@ static int zram_recompress(struct zram *zram, u32 i= ndex, struct page *page, return 0; } =20 -#define RECOMPRESS_IDLE (1 << 0) -#define RECOMPRESS_HUGE (1 << 1) - static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); - unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; + struct zram_pp_ctl *ctl =3D NULL; + struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; - unsigned long index; struct page *page; ssize_t ret; =20 @@ -1922,36 +2058,32 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 + ctl =3D init_pp_ctl(); + if (!ctl) { + ret =3D -ENOMEM; + goto release_init_lock; + } + + scan_slots_for_recompress(zram, mode, ctl); + ret =3D len; - for (index =3D 0; index < nr_pages; index++) { + while ((pps =3D select_pp_slot(ctl))) { int err =3D 0; =20 if (!num_recomp_pages) break; =20 - zram_slot_lock(zram, index); - - if (!zram_allocated(zram, index)) - goto next; - - if (mode & RECOMPRESS_IDLE && - !zram_test_flag(zram, index, ZRAM_IDLE)) + zram_slot_lock(zram, pps->index); + if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; =20 - if (mode & RECOMPRESS_HUGE && - !zram_test_flag(zram, index, ZRAM_HUGE)) - goto next; - - if (zram_test_flag(zram, index, ZRAM_WB) || - zram_test_flag(zram, index, ZRAM_UNDER_WB) || - zram_test_flag(zram, index, ZRAM_SAME) || - zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) - goto next; - - err =3D zram_recompress(zram, index, page, &num_recomp_pages, - threshold, prio, prio_max); + err =3D recompress_slot(zram, pps->index, page, + &num_recomp_pages, threshold, + prio, prio_max); next: - zram_slot_unlock(zram, index); + zram_slot_unlock(zram, pps->index); + release_pp_slot(zram, pps); + if (err) { ret =3D err; break; @@ -1963,6 +2095,7 @@ static ssize_t recompress_store(struct device *dev, __free_page(page); =20 release_init_lock: + release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); return ret; --=20 2.46.0.598.g6f2099f65c-goog From nobody Sat Nov 30 04:27:16 2024 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAE0818C920 for ; Thu, 12 Sep 2024 07:04:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124676; cv=none; b=Hrzkv76vaQxnOny9DotXEyyL4iLxA3DHRAdDojizx53ygN9PCpKR1ZZRvU3wL7kG/Ima1emdkesIbjL3eXvjNhox4RvKuBd+cd2KdosG6oYSE2P0ZrDS8GR0BQ/2VKxIjLsJ4pwQ3YEcm9U+sLWds7wtQbMMj+xaPtqa9OBCmW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124676; c=relaxed/simple; bh=JRLRXgyQqo6xYv+WAjtOW/iCMpNcasMB/RXY7828h68=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NmdJ7AHUAxgK1JjIRGzymIUYRcB9Fv0iAlDfNQQzhcoUbLMj6ZhC4iSrUSXl26gkDQ+nTCi8j7fuLiNH2P3H6Xdcgo2aiArHWSDzWvluo/3XWl5PcLWdoeVD94A1BalNa5XJz/rQmJByhomNB4Np0QONAhg1ZgBbhYsH+9htQaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=dHNkcg95; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="dHNkcg95" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2054e22ce3fso7034165ad.2 for ; Thu, 12 Sep 2024 00:04:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124673; x=1726729473; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AUUG9N1lEhl3OV7ZgcZTdELGeL30tImYKIZa/xNBV7o=; b=dHNkcg95HIg9ylgNLF8WmTxt4iFJYJi1ZMfwEIAvx+V6PP4f9KnXaPgqDXbN5i7ZxG u1QRkLuRNjMe1NzkDMqYYphwEKy0mYcWC/i9FUB2K4lKP4MoI/yPkPD8I3tJlxd+hncJ Sq/Cp1Tz6L+jFl5rUprTmQZKhkwz+N4aGjbHM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124673; x=1726729473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AUUG9N1lEhl3OV7ZgcZTdELGeL30tImYKIZa/xNBV7o=; b=QIgXSMH6rlznpHKLreNSemIMndcGeZTlVA1Ymz6BfxHvEIkTSXA9wZKBrzZcvtpTAv LAVtyN8qAC0Aqp24dT/jPDDM9JNflLbX0YdX0cIGyd0/wcygpN4b3HKbjbiKZti7oW5Z jVKIV090I18f7AqGtWVYkSyomuEuNQzF32C+EvJxlaRwtDNc3nKR0LpbN3L/olpzo3HH llN3JxYT3hORlYjY7BO5M4YVdjsMw/YQuYrDkqhGaAkCRlgwESoR+74SI/0BvThWz15A TCKb/4DXRAKtepeuBxVTVXMSZZsJUftSRiCQwr7T9da2PGJfsIBScnHy8iIl5jv/XpZX OXgg== X-Gm-Message-State: AOJu0YykSjJvvIyfzSBVsCGpTLuDdgQ08R0H1UAwKAqhONcWmBnW8uzt 3SjlKG7wQ2awbwLOwYw0tFu7wNI+EklvsbPDGc8Cd4VWIa6g1IPWc9j+zliw3Q== X-Google-Smtp-Source: AGHT+IGHKU6bIxwpDal5Gc9HvLXRfjOUaJyo/V427lSJ+H6KcbsvLVA6//YtNRJ1bvmOCDlAce7MTw== X-Received: by 2002:a05:6a20:b598:b0:1cf:2858:be5b with SMTP id adf61e73a8af0-1cf764c44b1mr1946403637.50.1726124673386; Thu, 12 Sep 2024 00:04:33 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:33 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 4/6] zram: rework writeback target selection strategy Date: Thu, 12 Sep 2024 16:02:19 +0900 Message-ID: <20240912070413.4179924-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Writeback suffers from the same problem as recompression did before - target slot selection for writeback is just a simple iteration over zram->table entries (stored pages) which selects suboptimal targets for writeback. This is especially problematic for writeback, because we uncompress objects before writeback so each of them takes 4K out of limited writeback storage. For example, when we take a 48 bytes slot and store it as a 4K object to writeback device we only save 48 bytes of memory (release from zsmalloc pool). We naturally want to pick the largest objects for writeback, because then each writeback will release the largest amount of memory. This patch applies the same solution and strategy as for recompression target selection: pp control (post-process) with 16 buckets of candidate pp slots. Slots are assigned to pp buckets based on sizes - the larger the slot the higher the group index. This gives us sorted by size lists of candidate slots (in linear time), so that among post-processing candidate slots we always select the largest ones first and maximize the memory saving. TEST =3D=3D=3D=3D A very simple demonstration: zram is configured with a writeback device. A limited writeback (wb_limit 2500 pages) is performed then, with a log of sizes of slots that were written back. You can see that patched zram selects slots for recompression in significantly different manner, which leads to higher memory savings (see column #2 of mm_stat output). BASE ---- *** initial state of zram device /sys/block/zram0/mm_stat 1750327296 619765836 631902208 0 631902208 1 0 3427= 8 34278 *** writeback idle wb_limit 2500 /sys/block/zram0/mm_stat 1750327296 617622333 631578624 0 631902208 1 0 3427= 8 34278 Sizes of selected objects for writeback: ... 193 349 46 46 46 46 852 1002 543 162 107 49 34 34 34 ... PATCHED ------- *** initial state of zram device /sys/block/zram0/mm_stat 1750319104 619760957 631992320 0 631992320 1 0 3427= 8 34278 *** writeback idle wb_limit 2500 /sys/block/zram0/mm_stat 1750319104 612672056 626135040 0 631992320 1 0 3427= 8 34278 Sizes of selected objects for writeback: ... 3667 3580 3581 3580 3581 3581 3581 3231 3211 3203 3231 3246 ... Note, pp-slots are not strictly sorted, there is a PP_BUCKET_SIZE_RANGE variation of sizes within particular bucket. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 83 +++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6688f70b2140..8ba994a5f30e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -184,7 +184,7 @@ static void zram_accessed(struct zram *zram, u32 index) #endif } =20 -#ifdef CONFIG_ZRAM_MULTI_COMP +#if defined CONFIG_ZRAM_WRITEBACK || defined CONFIG_ZRAM_MULTI_COMP struct zram_pp_slot { unsigned long index; struct list_head entry; @@ -681,11 +681,57 @@ static void read_from_bdev_async(struct zram *zram, s= truct page *page, #define IDLE_WRITEBACK (1<<1) #define INCOMPRESSIBLE_WRITEBACK (1<<2) =20 +static int scan_slots_for_writeback(struct zram *zram, u32 mode, + unsigned long nr_pages, + unsigned long index, + struct zram_pp_ctl *ctl) +{ + struct zram_pp_slot *pps =3D NULL; + + for (; nr_pages !=3D 0; index++, nr_pages--) { + if (!pps) + pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); + if (!pps) + return -ENOMEM; + + INIT_LIST_HEAD(&pps->entry); + + zram_slot_lock(zram, index); + if (!zram_allocated(zram, index)) + goto next; + + if (zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_SAME)) + goto next; + + if (mode & IDLE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_IDLE)) + goto next; + if (mode & HUGE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_HUGE)) + goto next; + if (mode & INCOMPRESSIBLE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + goto next; + + pps->index =3D index; + place_pp_slot(zram, ctl, pps); + pps =3D NULL; +next: + zram_slot_unlock(zram, index); + } + + kfree(pps); + return 0; +} + static ssize_t writeback_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { struct zram *zram =3D dev_to_zram(dev); unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; + struct zram_pp_ctl *ctl =3D NULL; + struct zram_pp_slot *pps; unsigned long index =3D 0; struct bio bio; struct bio_vec bio_vec; @@ -737,7 +783,15 @@ static ssize_t writeback_store(struct device *dev, goto release_init_lock; } =20 - for (; nr_pages !=3D 0; index++, nr_pages--) { + ctl =3D init_pp_ctl(); + if (!ctl) { + ret =3D -ENOMEM; + goto release_init_lock; + } + + scan_slots_for_writeback(zram, mode, nr_pages, index, ctl); + + while ((pps =3D select_pp_slot(ctl))) { spin_lock(&zram->wb_limit_lock); if (zram->wb_limit_enable && !zram->bd_wb_limit) { spin_unlock(&zram->wb_limit_lock); @@ -754,25 +808,10 @@ static ssize_t writeback_store(struct device *dev, } } =20 + index =3D pps->index; zram_slot_lock(zram, index); - if (!zram_allocated(zram, index)) - goto next; - - if (zram_test_flag(zram, index, ZRAM_WB) || - zram_test_flag(zram, index, ZRAM_SAME) || - zram_test_flag(zram, index, ZRAM_UNDER_WB)) - goto next; - - if (mode & IDLE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_IDLE)) - goto next; - if (mode & HUGE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_HUGE)) - goto next; - if (mode & INCOMPRESSIBLE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) goto next; - /* * Clearing ZRAM_UNDER_WB is duty of caller. * IOW, zram_free_page never clear it. @@ -786,6 +825,8 @@ static ssize_t writeback_store(struct device *dev, zram_clear_flag(zram, index, ZRAM_UNDER_WB); zram_clear_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); + + release_pp_slot(zram, pps); continue; } =20 @@ -804,6 +845,8 @@ static ssize_t writeback_store(struct device *dev, zram_clear_flag(zram, index, ZRAM_UNDER_WB); zram_clear_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); + + release_pp_slot(zram, pps); /* * BIO errors are not fatal, we continue and simply * attempt to writeback the remaining objects (pages). @@ -846,12 +889,14 @@ static ssize_t writeback_store(struct device *dev, spin_unlock(&zram->wb_limit_lock); next: zram_slot_unlock(zram, index); + release_pp_slot(zram, pps); } =20 if (blk_idx) free_block_bdev(zram, blk_idx); __free_page(page); release_init_lock: + release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); =20 --=20 2.46.0.598.g6f2099f65c-goog From nobody Sat Nov 30 04:27:16 2024 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C1E718E047 for ; Thu, 12 Sep 2024 07:04:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124677; cv=none; b=O728whVRsm9V8N+ppZ8+3KtrCcpFajNldx3P4J8Y5W7lza9BpUzUBsNTkdpadZA+Y3JbgouKrubcKc/XtxV6TBn9DpHQWWhDG5QJxZP5PwZfezB9W2mIXuqivFA8GA6EOesAigaSKJuOJ9p2OPzBBebKspFttpPqqOd5YBPNw0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124677; c=relaxed/simple; bh=I+kuo6MDSFh1Bhzr51TroqFkgroBeul0T3itlQe+qCA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YIt0I0/68Pe0HoNu22B8+cvtzb4MKYiAaPFJfQH/DPaWjBHU6LM5ESh/2JC+jzloMvutKl+UDjdk2nxFEPGpaleePzJKds41hZybOGBYDFH1tdxsXTBDrdQ7zr8Yp/lbsd5nzLE9JAwgECeI8GJ6jmcMSNrnwAR53AAI/2kEWaM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RqXcIgSZ; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RqXcIgSZ" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-718d8d6af8fso405593b3a.3 for ; Thu, 12 Sep 2024 00:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124676; x=1726729476; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7GFCqjG6t4HanjxEsiLCHQQ2pA5kAkIPxQoDQLNprJM=; b=RqXcIgSZxPOonklYNgILlIqCEcolsR4UebiC07cbEnKY3U7M0i9+uqvuhOcV0zJWN2 wKTPiGdJ8X1pCz+6lXJznap1UEftOVKiRW3KlT2pLouw2VqQ/iTrzbZd+T7wqyjlWOSd ryoW9EJi184JTqeMhi8e5aA9S/FurEpsuz/UQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124676; x=1726729476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7GFCqjG6t4HanjxEsiLCHQQ2pA5kAkIPxQoDQLNprJM=; b=woU2vIn2kiO6kbtueDzu6vK/giqfkyymCWR1L/8IMDFIKu9OkCd+gg1kNogcO5MrBr UCOFBx+uvlJFPV6WWMzNHeJTI5lpFkS/2BbVEAHTGTQmXYk1Y4/jMSl5MBmc8rotYSOH GCcWHP1hj6t0izWjjYcQbww6ZX8ykHbmzr+QaKJ6ISqGKV5fC3pT1VaOvU7V28ryvl1R u+zdO0tkoTSOUk796Ezkx6gsJPY6eOI3NdGiBDBJy+0h+7upRiZ9pX3ABBWGBQSKoFzA 3+w7V0q/0KeRLmiEHXDMH/srzIvc8zFQsx/MBJL50LAFesfAw9jDiHFjPHCpyF+V+DDs 7b8w== X-Gm-Message-State: AOJu0YxneiFsDoWXe+keDtor+wo0lJ4yvTQ85swrxFbls0zCALLRwuZx tTDV4S+M/ppzEC2wiQYUB2DOrj/2ipU9zPYhearx/vK/ix3J2TIAp9tDTl4BR3phJsNUU2sdZLG 9Og== X-Google-Smtp-Source: AGHT+IEaKcYVQXn74aOSr6pjLq105JrvxJyT1w4oVchYDkTP4Fh4o5Pz/q6SBVXlcd94V2F5VeDrbQ== X-Received: by 2002:a05:6300:404c:b0:1cf:4491:b6b7 with SMTP id adf61e73a8af0-1cf75e79c7amr2271175637.3.1726124675929; Thu, 12 Sep 2024 00:04:35 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:35 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 5/6] zram: do not mark idle slots that cannot be idle Date: Thu, 12 Sep 2024 16:02:20 +0900 Message-ID: <20240912070413.4179924-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ZRAM_SAME slots cannot be post-processed (writeback or recompress) so do not mark them ZRAM_IDLE. Same with ZRAM_WB slots, they cannot be ZRAM_IDLE because they are not in zsmalloc pool anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8ba994a5f30e..e48e2deec685 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -392,17 +392,28 @@ static void mark_idle(struct zram *zram, ktime_t cuto= ff) /* * Do not mark ZRAM_UNDER_WB slot as ZRAM_IDLE to close race. * See the comment in writeback_store. + * + * Also do not mark ZRAM_SAME slots as ZRAM_IDLE, because no + * post-processing (recompress, writeback) happens to the + * ZRAM_SAME slot. + * + * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ zram_slot_lock(zram, index); - if (zram_allocated(zram, index) && - !zram_test_flag(zram, index, ZRAM_UNDER_WB)) { + if (!zram_allocated(zram, index) || + zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_UNDER_WB) || + zram_test_flag(zram, index, ZRAM_SAME)) { + zram_slot_unlock(zram, index); + continue; + } + #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME - is_idle =3D !cutoff || ktime_after(cutoff, - zram->table[index].ac_time); + is_idle =3D !cutoff || + ktime_after(cutoff, zram->table[index].ac_time); #endif - if (is_idle) - zram_set_flag(zram, index, ZRAM_IDLE); - } + if (is_idle) + zram_set_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); } } --=20 2.46.0.598.g6f2099f65c-goog From nobody Sat Nov 30 04:27:16 2024 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF24818EFF4 for ; Thu, 12 Sep 2024 07:04:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124680; cv=none; b=h6Jl0AfUbTuJDGMtCnnyY4WpHrJAnq1axmlEDYpVdtsMk9Feh+4nR9OwOnd+2wlrKUj8GPEXczK+RXsikqXpuGUrybk5Dql7kVQsFDsLyG3ciYKfQu8FQ6433fhGEQAjE3Bi8QIeg8c2nnbfwlXvJIgAWUmmHZECFSqidf2i1GA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726124680; c=relaxed/simple; bh=ESMGWutE/ueYLMAj2JiYG7tbgPqYxMU1kdGf4uO7PX4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IQ7EdFQ3IP3dkW8xUV0I/26ANlSqp4CkjZJGLUi2E+/6ALrqP/Sn9flqdya1I9rcK6VbOItW3BZY9VIFOlM9Hy/C40sfbCISYn00dCJI7U4ecXSrJ3p+yZoCyYuYRgS0e7eMGQtAN6w3dV7fd3+G+882FzdfrXtHaU+Iohq43k0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=c/J+729U; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="c/J+729U" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-71923d87be4so374880b3a.0 for ; Thu, 12 Sep 2024 00:04:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726124678; x=1726729478; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8uxEhnmpnm3I1DwDVMOxLe1iKdHvwPreOeTeNXmi03A=; b=c/J+729UD21RfytljelW0kL9AAYZy6LsaSU2zfntjK5YDhPdXZfekNB35CDglqajFG PCP35pitHuTgPhnONWkXJZRhSN9HCdi17s2/xAaVEBrvtbdhdfpm4H34BRArr1x+OdIq M6J1X4Pm5cgMyA3ACRZc5AWxZ/kfaCiiH0DvU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726124678; x=1726729478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8uxEhnmpnm3I1DwDVMOxLe1iKdHvwPreOeTeNXmi03A=; b=mA86NW71DjMSqIVJxUPtTNByo6yyzgCzuotoCgZefdTD0YE+Wgmgloqh4Elg/RyrI+ Uqc83M9asf8dptn4cXxtkXuMfoieZBTgru4zDuFTWbFrauKiYaTLmNYnUg8kSs2hZaTf AbUssCi4UCWcofSkGEFT6dz7OFVZ7VoKSooW8TXP6KzNlooClULtX4vP5i+dR9nHdzZu 16la7Ar01mrAzMFCZYINvbNN3E26pSRU+Zg1+EAjaTv8srzS71l+JZQmyITfd+Ehm6W9 AupdwjrqB9raS4T3skNNrf5PQjxw7g41IZ1/D2V6Fn8qL4a/yjGYlndKScDhe3Rzv/Yt f5aQ== X-Gm-Message-State: AOJu0YyEZwEXPgyWfICbEZ+GQhzZ80hSZvMSMFAqswq4CabMkgtLYp9W 0uxoZb6BPSDA4lTERqm2+OGiz8DAv6q7bYnoitaLEGiiDwyNW6YE23BNTL9ozA== X-Google-Smtp-Source: AGHT+IHxbY/zTNbD1OSJJmig6eMIYQ1euNqHzW8qtk2WIE0dG64pvGxq38P5tJES6spF7EYJEfVe7A== X-Received: by 2002:a05:6a00:a96:b0:717:8eca:95 with SMTP id d2e1a72fcca58-719261d0951mr3207502b3a.17.1726124678268; Thu, 12 Sep 2024 00:04:38 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:2fb2:d683:de5d:c3cb]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7db1fbb5a23sm1101543a12.31.2024.09.12.00.04.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 00:04:38 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 6/6] zram: reshuffle zram_free_page() flags operations Date: Thu, 12 Sep 2024 16:02:21 +0900 Message-ID: <20240912070413.4179924-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.598.g6f2099f65c-goog In-Reply-To: <20240912070413.4179924-1-senozhatsky@chromium.org> References: <20240912070413.4179924-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop some redundant zram_test_flag() calls and re-order zram_clear_flag() calls. Plus two small trivial coding style fixes. No functional changes. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e48e2deec685..8f01dc1fc796 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1499,20 +1499,17 @@ static void zram_free_page(struct zram *zram, size_= t index) #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D 0; #endif - if (zram_test_flag(zram, index, ZRAM_IDLE)) - zram_clear_flag(zram, index, ZRAM_IDLE); + + zram_clear_flag(zram, index, ZRAM_IDLE); + zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); + zram_set_priority(zram, index, 0); =20 if (zram_test_flag(zram, index, ZRAM_HUGE)) { zram_clear_flag(zram, index, ZRAM_HUGE); atomic64_dec(&zram->stats.huge_pages); } =20 - if (zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) - zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); - - zram_set_priority(zram, index, 0); - zram_clear_flag(zram, index, ZRAM_PP_SLOT); - if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); free_block_bdev(zram, zram_get_element(zram, index)); @@ -1536,13 +1533,12 @@ static void zram_free_page(struct zram *zram, size_= t index) zs_free(zram->mem_pool, handle); =20 atomic64_sub(zram_get_obj_size(zram, index), - &zram->stats.compr_data_size); + &zram->stats.compr_data_size); out: atomic64_dec(&zram->stats.pages_stored); zram_set_handle(zram, index, 0); zram_set_obj_size(zram, index, 0); - WARN_ON_ONCE(zram->table[index].flags & - ~(1UL << ZRAM_UNDER_WB)); + WARN_ON_ONCE(zram->table[index].flags & ~(1UL << ZRAM_UNDER_WB)); } =20 /* --=20 2.46.0.598.g6f2099f65c-goog