From nobody Sat Dec 28 06:50:22 2024 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB8B619D098 for ; Wed, 18 Dec 2024 11:46:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522416; cv=none; b=LZgAT3UI4o2oy4ToPWBt8iS7Rg7FV3QTs+WKvsuBnZFzKhT8Kaghldwepr9elParmlqti2bjzfki+bcJW/egkU9ccyl2osRLC/RVmo/4rz1oNiTz1gdVAFUq4a2vgIoLhyWGcI2WhKf45vOJJXqaz2SeE+cVF4Uw45WQZYo1pEs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522416; c=relaxed/simple; bh=pzeupsIj270sR+W/PqVPJ7qfiPRVogiciKRbC2tMwiQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FJsZA3zmG+ymDMBzgX9pj5fJNuQNrtl37rB5fe1UiptYJQODwcmoOtk7LDsVRG/UJT/q8lGogogtAdVFuDkxRHPiiVA3cmL5JOHaTs4qGZy2PgR4HWQJlBinhzasei/23uHAYy/uPehMc7rB+ESM9V3/UCeSKjtLYN/8keerm78= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XmLiElrz; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XmLiElrz" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21654fdd5daso53350365ad.1 for ; Wed, 18 Dec 2024 03:46:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522414; x=1735127214; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=XmLiElrz6TTqesLSn7d54r5apIeKpobOGPH/Fnac2f2h471ccTFOVKmHG/SzbBnsLc UKpSl2yxXyuwSeXlVtaHmXcacMBa8u05xiJyYhalhVRe/l66DWd4kYdsFG0ZUOA+qScS SXBsiKp/PtqkrkCum2LcNsp7Ufjva1bfT2/wEGTG4uOYdmNGhY02Jo64k/XQqm6fEQzX LrTwezpQAIKDJPE7wOgH2e7/xUlD+V68KXUlvdd2cS5huL98vOGIxv4+C33jJey2I0mR mv8BTIZaOUI0BkqkOLm7oVUTwUhzwLTm1CXjsDWZ/zAwWd+Q8OYboAEhoGwZum7XHoD1 p0UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522414; x=1735127214; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=TZjrxNR4SmEO/4Ic6juilg8sSmGm/a2O3VT4Rmm6d5HDOnc0wSp5X9FGCe+eQ2MQWs SFe3ARuHDBtllqdIQe+EoSHMaHrjdl7SB6jFVNlK8+E906hOLVdGxUCmlV9U0Cyqyj2J CKj+EechOhvGi1KuA65ApLMF7fCM+v2NH95b0hk3vPPYHduEcDOCp6SD8alCiDPYZnt4 2hknf2Cj49sWwhD0hftYLZPOb78eZa1GyElH2gfyrFLGHA61TIi993scOLxh618pvj5i enmrro7HtvyG9foqOUv5JNrusyFqCxS7gq+mhuF205NxrTVqkG3qkQsTYM21j+7No769 VgFA== X-Forwarded-Encrypted: i=1; AJvYcCU7+6AOEbR3VRv4nc+6CALBLiMPAU6QQMyQsZVhxtT5mhd29BZOfIoe3lZ0u1hIdZxXDS0+lKxgy0EOVhw=@vger.kernel.org X-Gm-Message-State: AOJu0Yx6MKSILwMA+8blzp9046Aeh5TYb7mqY8hDnSEOBsx3YE4eXRLL 9LQcm0UyzNdDF/T6Te8Db6AsYsCTZQLgiyRbI/zxmEYHasCFunNs X-Gm-Gg: ASbGncuBm+j0eDmBzBfHK4zYRFCq7Ex6+mu1V1huCahdTTZpJgortXGl7hqx2GAHDwb JuGIUQTLBRe0N5Zb90S66V2wbtpuH377VXpXS4CfM8cNe2/Omk3pknPuRcJ+CdoMe+OscNwHoa5 kPpioUlbrsKer2gDDEuOynWF0mIgxvZLZuhrJBAAYB/D4CqP8LxYNj/4yNXUj8zz/aGtgB1umCX YVNGKR3/O1C/2Wuhn7C7vUSY4Ulbr+y3xUBiKNsVyNNBisUHkkY31sh/M4vwiECufeG/Sah5oja eCT0CmYcQw0= X-Google-Smtp-Source: AGHT+IHeEFfNZXB8eHhDobzJ6aD+/g4JcIAsmjLr3dsquieQlQtdJw7HvtuvZYuwp308/geLA7elvA== X-Received: by 2002:a17:903:2348:b0:215:9642:4d6d with SMTP id d9443c01a7336-218d6fe1d79mr29573335ad.17.1734522414061; Wed, 18 Dec 2024 03:46:54 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:46:53 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 1/4] mm, memcontrol: avoid duplicated memcg enable check Date: Wed, 18 Dec 2024 19:46:30 +0800 Message-ID: <20241218114633.85196-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song mem_cgroup_uncharge_swap() includes a mem_cgroup_disabled() check, so the caller doesn't need to check that. Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..79900a486ed1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4609,7 +4609,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entr= y, unsigned int nr_pages) * correspond 1:1 to page and swap slot lifetimes: we charge the * page to memory here, and uncharge swap when the slot is freed. */ - if (!mem_cgroup_disabled() && do_memsw_account()) { + if (do_memsw_account()) { /* * The swap entry might not get freed for a long time, * let's not wait for it. The page already received a --=20 2.47.1 From nobody Sat Dec 28 06:50:22 2024 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 771F21AA1D1 for ; Wed, 18 Dec 2024 11:46:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522419; cv=none; b=apGHgecJvZCRUXIDaS+V6+51NBp+6m+gqiRwb8uT7w7fyrdpuUXJg8UbwXqHbtGJ/eEfuAltZXDkFDpYrTKi+PaLzQJPrLw3FgycwpWMXvCE+fpFJnOyfiPMIyRgSe+0SEC9ANaQX6MVBR1nT69d6+y9Y0oOk5o7KGw0PFWYOCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522419; c=relaxed/simple; bh=hOZ5NlNngeRjH8z8G2TqenUNXh7uBbdu8kbCPeD0Q80=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OROIZY9tJp67yTuo/WdfDqRmF8BNvGZT48uJW7wV7HXxUZJY2ocHcuTkKz6YYlCTMKLMLpHHy/nucrjjQpt95MYv9NM5VhL2YkIAbqzir72WL8RhPsWINcREfXjp2OtpDkPA7P52nyVZA7FxrdmUJxV7vE7N5E1uo9a5OCcd4b0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JQMXYgp3; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JQMXYgp3" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-21644aca3a0so75550845ad.3 for ; Wed, 18 Dec 2024 03:46:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522418; x=1735127218; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=JQMXYgp3eoebul49Mf7+Y0x28nz+zA4SjsYH3CGrtRE99c/3RqLfYZULANSd1jQFRt aaZKEnSGQRa7eMNv1t/UgHtIrh7g4NOxHILznFo5ceeZ7djSQ4kbcHRn1OzHckcVzSYR SE+KTNdk3ID3Aeo9sOyA5wxtGI5O9gkm4+5GqkH9FXL1hrK3bJfTmNhgGLC0m+hWbfrK pRTEtYIepKG8UY5lATaue0Vm7X4X0j+b2+zZSa75Vk7/ygt9I7lxPYDdN4FQXQaWwpWx PfZTmqfPsmmybp1tb6fdkYrHhZconf+v8WgVdhqe1c6RXIDkIgAetQmShBNR9yCBHXXQ qCWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522418; x=1735127218; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=fYoARxSODb8jTG/cGWbenYxqFsxlNqDZVhe8I85UtYgAq9OB+UqIMrx/5uV1vHulXF F0NwEklE8Lz+GaLK3J3BZhEj+iVRz0VaG+wvAhbW5JcjzlvbIAOiPWkdBob1W3VD4nvd cju46nd7jaC6Hn8B6tVjR8Q+US1fFyVk0IhA2krtDM6tF/mY2CjmEDe7DnTrEPdmiL1W KcRghTHZaWiwBUPpL4+V6MdbQL4/2HEBIW9YGd6d+N4R7jcFT6jowdfs1zrI6N6jma46 Lig2Xzs/lvB5ZgEowxvuwXnf9/e9/u4VEmVSwhUJNe1Ml2svRTsTy1PjFOT0tbunMlPA GpGw== X-Forwarded-Encrypted: i=1; AJvYcCWu2eqE7re3oShyiGMQ5HvOjMTUaAo8re3Ls3nLm77BzD2Ok3dpc2z0piY+vRsuX/hidUYEjs4hU0WLHPo=@vger.kernel.org X-Gm-Message-State: AOJu0YzNHoPLZ604b2aWt7GFRyflNxYo+hl3hixo4aQ3IcmULeFN/+Uo FV9Kt74Zx0Sql6b5VWjfnHJLfRAjJ1e+jTKvdlmdcmZRm9emAt10 X-Gm-Gg: ASbGncuWYb6BV7zu3wE8Cg4x25jB1yxcnTocOdPZ2FxCq+EycTqHI3hTFQdTxE9cePw skumpIqPUYy3U37W+Y740e6D4LRgbs8d+B4hb9DQqqsmQ3isC57tfi+nZaNQR7rTLWBeNXTw5Mz FmhobazS767JjvImLZQIMGFb7ETNMDjN1Nhx4q1mazdd4pOmlFpxKxrdug2ZU4kYwxIm3kZeHzx aH9cESJtUK7L7GPsiX6K+vr4Yk0XokEJlkLWyhEhn2cVIIz0IVXPKcprkiGKbfioO3w3G9ithJv 5WsrJICWPZ4= X-Google-Smtp-Source: AGHT+IFDiEO8Z+4jJiOJGyVQCnx8m44fCY5Sa4Cm3pyPpkeXJ0qEbZhG2rc7F1TYGzL+U0+D2iRi5w== X-Received: by 2002:a17:90b:2745:b0:2ee:8ea0:6b9c with SMTP id 98e67ed59e1d1-2f2e91d7dddmr4670423a91.12.1734522417699; Wed, 18 Dec 2024 03:46:57 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:46:57 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 2/4] mm/swap_cgroup: remove swap_cgroup_cmpxchg Date: Wed, 18 Dec 2024 19:46:31 +0800 Message-ID: <20241218114633.85196-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song This function is never used after commit 6b611388b626 ("memcg-v1: remove charge move code"). Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- include/linux/swap_cgroup.h | 2 -- mm/swap_cgroup.c | 29 ----------------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h index ae73a87775b3..d521ad1c4164 100644 --- a/include/linux/swap_cgroup.h +++ b/include/linux/swap_cgroup.h @@ -6,8 +6,6 @@ =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) =20 -extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new); extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short i= d, unsigned int nr_ents); extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent); diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index f63d1aa072a1..1770b076f6b7 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -45,35 +45,6 @@ static struct swap_cgroup *lookup_swap_cgroup(swp_entry_= t ent, return &ctrl->map[offset]; } =20 -/** - * swap_cgroup_cmpxchg - cmpxchg mem_cgroup's id for this swp_entry. - * @ent: swap entry to be cmpxchged - * @old: old id - * @new: new id - * - * Returns old id at success, 0 at failure. - * (There is no mem_cgroup using 0 as its id) - */ -unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new) -{ - struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned long flags; - unsigned short retval; - - sc =3D lookup_swap_cgroup(ent, &ctrl); - - spin_lock_irqsave(&ctrl->lock, flags); - retval =3D sc->id; - if (retval =3D=3D old) - sc->id =3D new; - else - retval =3D 0; - spin_unlock_irqrestore(&ctrl->lock, flags); - return retval; -} - /** * swap_cgroup_record - record mem_cgroup for a set of swap entries * @ent: the first swap entry to be recorded into --=20 2.47.1 From nobody Sat Dec 28 06:50:22 2024 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D4951ACEAE for ; Wed, 18 Dec 2024 11:47:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522423; cv=none; b=JmhQ/IhlVesWsUMHVQcdEL/dGxjuZKxPLI8N/rqicJpfbft9O5vVQ4YaFtWsn9DSGemfj2RuHhByj97TF1h/1S/NAkup5i2wMNrly1FUOxTq6/PlVZQ12nMxJfjLljuQ1sPa3oA5mcTIgNaVlNYBzx3UzywM1z31muXBH+u7ZpY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522423; c=relaxed/simple; bh=iO77dclC7icaLUgCLpicVvYq/fLvHBaS3sC4soxvpTs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iBTogdLhjvEjZ6yfmBGQxYzLkJdalecIMOdWj/wsp6uZJPfXOFYtWxk5aX0urk4+zYonY3RoyVK5YIdFh5U6u4tllRhvjGk/NO/me8eLZehSdQyd++PTsx9PAOZcyoqM9460ln10e5inEeQluUYTD+mMbq0XjwhXIq2Y/GINXTc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OcdcnwHb; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OcdcnwHb" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2f43da61ba9so152065a91.2 for ; Wed, 18 Dec 2024 03:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522421; x=1735127221; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=OcdcnwHbCzyEpgCimBeYCfU2SS/eHKfPM4/CO85Oz9q6wc82OvjS0m5axor9EeO0Xq Ytt7VlFDlYPkn7E6KvMEdbXOyXGNAAt+P2IVwlCebVQ5PMcsbwW4eNlc/zkWGJAVv4gv 7obQD+O9oJrU6xqRrIX3jHGvDbUpY4MbQCNRY/V5MqC01xhwOSrWIJCZqduIdTJBbJdn m8a0SF1xrechO5ehSpz8sz8GRsejKW1+nBR2G8LSYTVsYslA9nvlMP7R7v8BmLv5es8a DOmTEAIrg81Gjnoklk92IwUAoJPTEWWu/eThpqMS8RxPvIVuj6FmeN2Tv9xnAPhkSX4y wrNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522421; x=1735127221; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=B7icpzLk+r1+Sxmu90ovDmP7gxxFSqKedVUncXzTsr+41zqQOvrpbNrEAuwJOcivQy WMfNNbeXUyJYqKuSER/fJXczclMohFiFMvtFdsoridRC0n4IoQXcNoeoT+QpvfF2PPXr IyjBIa4E0jCZuWTG4dTTRUUKNXM47woIyOjNfhOAi1lwUbI5KGrqqjEj6Uur2ZDhHNKu ge0sY4vxAvSbDw2TQ+t6UXZa7uFMcizoIk4x9a65Pbd/pkCeg5nvErCvCuf8JVZucdj6 6Wgb86rtbmNHVKqbjsBfRmBl4Kq9O3p2P7c/GVOEVtUGzWTuK1mUYRJQrCdnDkxcFAFf n1lg== X-Forwarded-Encrypted: i=1; AJvYcCU6vXqn6lr1UImXIjRBC2vd5szuivQlQprJCJ46wGO1mbP2DjJ75JKMljyWTZqezbN5Crt4Rzyqxk6LLLY=@vger.kernel.org X-Gm-Message-State: AOJu0Ywul7LlvZPQKay3IuYzna6uOhgFhW0Q0juCeUxaFWQS7CSyUIKu O5qKWSuiOZnL5EAnLMzyWOWovcVnPsvAFnvpnnGO4ub7jcV7lhHE X-Gm-Gg: ASbGncsBRTCm7oLGnHMGcNOFSE2JVXnqC87ErcmwjLN4ZpTPA2pW+uNe3rFwWXAAdDu VikEpI7cM71JFg40K25p9+i3XgGLTu0OhOFoe8tZA3WXjEWjVczc/N/aaDc9IgKZaQrfMMqRLJn 7MUFNX4/2Hi9/VfBqWLobM48H4iRfwE1MJ9fe+Y+K4iGlQQPNt0xKQ1n87a0RRNacu5Nvopvc1s It2IktyK2gP6MdSBPBmJyggskVNaBqHB+LfUb8Ck7dGdHpEtIHWAppxooWxktPbH1snpcLLAq8k TgTirY9wCc4= X-Google-Smtp-Source: AGHT+IEO+JWDMXQW6FsZJE3+1Wvh/7poieUhJ+CwA1N0IXPZ0ViHOqmG3ylHlvEdt5u9FJAhum58XA== X-Received: by 2002:a17:90b:5206:b0:2ee:c04a:4281 with SMTP id 98e67ed59e1d1-2f2e91a903cmr3823034a91.6.1734522421476; Wed, 18 Dec 2024 03:47:01 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:47:01 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 3/4] mm/swap_cgroup: remove global swap cgroup lock Date: Wed, 18 Dec 2024 19:46:32 +0800 Message-ID: <20241218114633.85196-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song commit e9e58a4ec3b1 ("memcg: avoid use cmpxchg in swap cgroup maintainance") replaced the cmpxchg/xchg with a global irq spinlock because some archs doesn't support 2 bytes cmpxchg/xchg. Clearly this won't scale well. And as commented in swap_cgroup.c, this lock is not needed for map synchronization. Emulation of 2 bytes xchg with atomic cmpxchg isn't hard, so implement it to get rid of this lock. Introduced two helpers for doing so and they can be easily dropped if a generic 2 byte xchg is support. Testing using 64G brd and build with build kernel with make -j96 in 1.5G memory cgroup using 4k folios showed below improvement (6 test run): Before this series: Sys time: 10782.29 (stdev 42.353886) Real time: 171.49 (stdev 0.595541) After this commit: Sys time: 9617.23 (stdev 37.764062), -10.81% Real time: 159.65 (stdev 0.587388), -6.90% With 64k folios and 2G memcg: Before this series: Sys time: 8176.94 (stdev 26.414712) Real time: 141.98 (stdev 0.797382) After this commit: Sys time: 7358.98 (stdev 54.927593), -10.00% Real time: 134.07 (stdev 0.757463), -5.57% Sequential swapout of 8G 64k zero folios with madvise (24 test run): Before this series: 5461409.12 us (stdev 183957.827084) After this commit: 5420447.26 us (stdev 196419.240317) Sequential swapin of 8G 4k zero folios (24 test run): Before this series: 19736958.916667 us (stdev 189027.246676) After this commit: 19662182.629630 us (stdev 172717.640614) Performance is better or at least not worse for all tests above. Signed-off-by: Kairui Song Reviewed-by: Roman Gushchin --- mm/swap_cgroup.c | 77 ++++++++++++++++++++++++++++++------------------ 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index 1770b076f6b7..cf0445cb35ed 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -7,19 +7,20 @@ =20 static DEFINE_MUTEX(swap_cgroup_mutex); =20 +/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t)= */ +#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short)) +#define ID_SHIFT (BITS_PER_TYPE(unsigned short)) +#define ID_MASK (BIT(ID_SHIFT) - 1) struct swap_cgroup { - unsigned short id; + atomic_t ids; }; =20 struct swap_cgroup_ctrl { struct swap_cgroup *map; - spinlock_t lock; }; =20 static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; =20 -#define SC_PER_PAGE (PAGE_SIZE/sizeof(struct swap_cgroup)) - /* * SwapCgroup implements "lookup" and "exchange" operations. * In typical usage, this swap_cgroup is accessed via memcg's charge/uncha= rge @@ -30,19 +31,35 @@ static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWA= PFILES]; * SwapCache(and its swp_entry) is under lock. * - When called via swap_free(), there is no user of this entry and no r= ace. * Then, we don't need lock around "exchange". - * - * TODO: we can push these buffers out to HIGHMEM. */ -static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, - struct swap_cgroup_ctrl **ctrlp) +static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, + pgoff_t offset) { - pgoff_t offset =3D swp_offset(ent); - struct swap_cgroup_ctrl *ctrl; + unsigned int shift =3D (offset % ID_PER_SC) * ID_SHIFT; + unsigned int old_ids =3D atomic_read(&map[offset / ID_PER_SC].ids); =20 - ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; - if (ctrlp) - *ctrlp =3D ctrl; - return &ctrl->map[offset]; + BUILD_BUG_ON(!is_power_of_2(ID_PER_SC)); + BUILD_BUG_ON(sizeof(struct swap_cgroup) !=3D sizeof(atomic_t)); + + return (old_ids >> shift) & ID_MASK; +} + +static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map, + pgoff_t offset, + unsigned short new_id) +{ + unsigned short old_id; + struct swap_cgroup *sc =3D &map[offset / ID_PER_SC]; + unsigned int shift =3D (offset % ID_PER_SC) * ID_SHIFT; + unsigned int new_ids, old_ids =3D atomic_read(&sc->ids); + + do { + old_id =3D (old_ids >> shift) & ID_MASK; + new_ids =3D (old_ids & ~(ID_MASK << shift)); + new_ids |=3D ((unsigned int)new_id) << shift; + } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids)); + + return old_id; } =20 /** @@ -58,21 +75,19 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsi= gned short id, unsigned int nr_ents) { struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned short old; - unsigned long flags; pgoff_t offset =3D swp_offset(ent); pgoff_t end =3D offset + nr_ents; + unsigned short old, iter; + struct swap_cgroup *map; =20 - sc =3D lookup_swap_cgroup(ent, &ctrl); + ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; + map =3D ctrl->map; =20 - spin_lock_irqsave(&ctrl->lock, flags); - old =3D sc->id; - for (; offset < end; offset++, sc++) { - VM_BUG_ON(sc->id !=3D old); - sc->id =3D id; - } - spin_unlock_irqrestore(&ctrl->lock, flags); + old =3D __swap_cgroup_id_lookup(map, offset); + do { + iter =3D __swap_cgroup_id_xchg(map, offset, id); + VM_BUG_ON(iter !=3D old); + } while (++offset !=3D end); =20 return old; } @@ -85,9 +100,13 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsi= gned short id, */ unsigned short lookup_swap_cgroup_id(swp_entry_t ent) { + struct swap_cgroup_ctrl *ctrl; + if (mem_cgroup_disabled()) return 0; - return lookup_swap_cgroup(ent, NULL)->id; + + ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; + return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent)); } =20 int swap_cgroup_swapon(int type, unsigned long max_pages) @@ -98,14 +117,16 @@ int swap_cgroup_swapon(int type, unsigned long max_pag= es) if (mem_cgroup_disabled()) return 0; =20 - map =3D vcalloc(max_pages, sizeof(struct swap_cgroup)); + BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=3D + sizeof(struct swap_cgroup)); + map =3D vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + sizeof(struct swap_cgroup)); if (!map) goto nomem; =20 ctrl =3D &swap_cgroup_ctrl[type]; mutex_lock(&swap_cgroup_mutex); ctrl->map =3D map; - spin_lock_init(&ctrl->lock); mutex_unlock(&swap_cgroup_mutex); =20 return 0; --=20 2.47.1 From nobody Sat Dec 28 06:50:22 2024 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BAD91ACEC7 for ; Wed, 18 Dec 2024 11:47:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522427; cv=none; b=N/fgmd6qvZx2QhqQaA+WIh6V3M6p2W4L7IwaJJxL6y7gJpKBMbMd1fUJFwGrnKKzCZNdIaHzCmUoZTJEjQ8oemMVXR/B54g1CpOzBw5Dha7v8sqV088GmQvmtPknpWAsWRqEoL5fya1lqBhKZCIjkr0Hxnr9vTicrh687RHe4kk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522427; c=relaxed/simple; bh=rG7e3gQGWAgP3Yj+QVhTZVB7c+i50XKZWLdjrhNDj7U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OU7vd5YxnJ68sTHUY7t+KdAGkT4C6e9DIw4haNcq1TH07OaojpKIhLiNOb4mMBWru0zP+mfIT9rauABKMIZpZeI2rWcvmK2LUEtslTYEbUpYQ2LRicBsbgP9Kq9iJ4dF/fz+5FInc/ZfLiIG+Q+oBUb0LaBS8soPZoS8DuUqrHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YoeUEH54; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YoeUEH54" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-215770613dbso43181315ad.2 for ; Wed, 18 Dec 2024 03:47:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522425; x=1735127225; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=e+ni2sm3wk+IEpHJsvCxGYROd6D/iUDHnxMXaHI/HCw=; b=YoeUEH54G2x4O87IDpySf8GkOP9rzUS4/JxhgVY4xxHEPIcwFqxb5F2ISR6ioodGG2 ir1es5zkQFpoLmFWapqvoquKET0Iq7+CjNPpnxrMDLPrpL4dp42liVStTQx9b1Iet8eD FcYI0onATJumvKi1HDsipjrWmns3viQOJKLL6g20ESxQgo+qhnGH9tar/l6yeOH1w2CM rM6CbWCiZyKB/SfQ4GVGQ4ELjIuCQK24dnUh+L64ZYmAZHDDjWjeXldOfXJ3kHCI5Xd3 lDYR3YtMnZnI2rF0BKg+iREA4Nn6hHxpSbKojPXDqZY3l6x4UmmXa8VqDdY3lGgqlIer u5eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522425; x=1735127225; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=e+ni2sm3wk+IEpHJsvCxGYROd6D/iUDHnxMXaHI/HCw=; b=NG9RtU44064ehVPdBDWgWYPlXgOXNY5MXW0Yrs1UqJaPPOU4uZBwSDbO/icW/n15td oRGROO6mbM46mf4PL//oKICs/2siEGoMqWYP3xzA38eyCcIjeJL5OAFQ5gkJ/1VPZz3G YdCwAu8/zeFPmhzL8PlhpvJQ+/GdATpc2S0vkL8dmFJ1dgsVtQoIgK907B1aegfwp+xe Ca9YfucbspmxSf/rUzB+2wCCJsBsF4wsCKNvoXAu4ng12bUCFMnxe+EsqqClrhDW+5OI R+CSQJejhsN4lpeZczItqejJDFKZDIk+9xoGZZT0JLktGVDdYL6OaQ1Uhw6FSksG/l5Z p4hg== X-Forwarded-Encrypted: i=1; AJvYcCV5pk2USn1C2mrpyAxW2xuSKy7fXPPkJNF/QW14eJuNjru4Q51LoqxAywIdBInfLRkUDPsbksiix0Vp5qo=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7MlJZdyjcdERjJw29VOHfohP5Ps+UqwrRfACPc2L3HbNNgHAb CUTEIIRya60OjeRtIby31053kR2R/zxMot9K0BanBkNWU4ePwHxR X-Gm-Gg: ASbGncvpeOEdVSdx2FGwGUejo25FNpDS5zyg04x3rgwYw9uxR73ZRLRovsqbcXkkimp E7aoKEsn7/g1U4KBJLfPltbqBNKkD0V/mqSKEhCewh9Mwg8wosRxyGh2K7ps3BHdRDDA4DSVirJ 5F8Td0rZKEkOgFR1/NYZk7EeVNSMUxTv1w0h8YY3Ki3Z9um7k30bZGtSTqPwvQVhtc8Znqj+rwI dtjAdQwifQkGUHfIsS/6Nu6GD1EM92eo8UkELN78T1lB//vMeUeF1Z/x23py0oAdN3dxGGZ8HJ0 4R5OXHOa6Rk= X-Google-Smtp-Source: AGHT+IFfwnmvDXUoevR3JMwJ83hwhWaRbCVWe3lKgTz6e75Uv7wy2N9Yq4LbPCY/3wUaTXBc06OKTQ== X-Received: by 2002:a17:903:183:b0:215:9894:5679 with SMTP id d9443c01a7336-218d6c4bc26mr35002555ad.0.1734522425286; Wed, 18 Dec 2024 03:47:05 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.47.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:47:04 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 4/4] mm/swap_cgroup: decouple swap cgroup recording and clearing Date: Wed, 18 Dec 2024 19:46:33 +0800 Message-ID: <20241218114633.85196-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song The current implementation of swap cgroup tracking is a bit complex and fragile: On charging path, swap_cgroup_record always records an actual memcg id, and it depends on the caller to make sure all entries passed in must belong to one single folio. As folios are always charged or uncharged as a whole, and always charged and uncharged in order, swap_cgroup doesn't need an extra lock. On uncharging path, swap_cgroup_record always sets the record to zero. These entries won't be charged again until uncharging is done. So there is no extra lock needed either. Worth noting that swap cgroup clearing may happen without folio involved, eg. exiting processes will zap its page table without swapin. The xchg/cmpxchg provides atomic operations and barriers to ensure no tearing or synchronization issue of these swap cgroup records. It works but quite error-prone. Things can be much clear and robust by decoupling recording and clearing into two helpers. Recording takes the actual folio being charged as argument, and clearing always set the record to zero, and refine the debug sanity checks to better reflect their usage Benchmark even showed a very slight improvement as it saved some extra arguments and lookups: make -j96 with defconfig on tmpfs in 1.5G memory cgroup using 4k folios: Before: sys 9617.23 (stdev 37.764062) After : sys 9541.54 (stdev 42.973976) make -j96 with defconfig on tmpfs in 2G memory cgroup using 64k folios: Before: sys 7358.98 (stdev 54.927593) After : sys 7337.82 (stdev 39.398956) Suggested-by: Chris Li Signed-off-by: Kairui Song --- include/linux/swap_cgroup.h | 12 ++++--- mm/memcontrol.c | 13 +++----- mm/swap_cgroup.c | 66 +++++++++++++++++++++++-------------- 3 files changed, 55 insertions(+), 36 deletions(-) diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h index d521ad1c4164..b5ec038069da 100644 --- a/include/linux/swap_cgroup.h +++ b/include/linux/swap_cgroup.h @@ -6,8 +6,8 @@ =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) =20 -extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short i= d, - unsigned int nr_ents); +extern void swap_cgroup_record(struct folio *folio, swp_entry_t ent); +extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_e= nts); extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent); extern int swap_cgroup_swapon(int type, unsigned long max_pages); extern void swap_cgroup_swapoff(int type); @@ -15,8 +15,12 @@ extern void swap_cgroup_swapoff(int type); #else =20 static inline -unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, - unsigned int nr_ents) +void swap_cgroup_record(struct folio *folio, swp_entry_t ent) +{ +} + +static inline +unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents) { return 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 79900a486ed1..ca1ea84b4bce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4973,7 +4973,6 @@ void mem_cgroup_swapout(struct folio *folio, swp_entr= y_t entry) { struct mem_cgroup *memcg, *swap_memcg; unsigned int nr_entries; - unsigned short oldid; =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); @@ -5000,11 +4999,10 @@ void mem_cgroup_swapout(struct folio *folio, swp_en= try_t entry) /* Get references for the tail pages, too */ if (nr_entries > 1) mem_cgroup_id_get_many(swap_memcg, nr_entries - 1); - oldid =3D swap_cgroup_record(entry, mem_cgroup_id(swap_memcg), - nr_entries); - VM_BUG_ON_FOLIO(oldid, folio); mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); =20 + swap_cgroup_record(folio, entry); + folio_unqueue_deferred_split(folio); folio->memcg_data =3D 0; =20 @@ -5035,7 +5033,6 @@ int __mem_cgroup_try_charge_swap(struct folio *folio,= swp_entry_t entry) unsigned int nr_pages =3D folio_nr_pages(folio); struct page_counter *counter; struct mem_cgroup *memcg; - unsigned short oldid; =20 if (do_memsw_account()) return 0; @@ -5064,10 +5061,10 @@ int __mem_cgroup_try_charge_swap(struct folio *foli= o, swp_entry_t entry) /* Get references for the tail pages, too */ if (nr_pages > 1) mem_cgroup_id_get_many(memcg, nr_pages - 1); - oldid =3D swap_cgroup_record(entry, mem_cgroup_id(memcg), nr_pages); - VM_BUG_ON_FOLIO(oldid, folio); mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); =20 + swap_cgroup_record(folio, entry); + return 0; } =20 @@ -5081,7 +5078,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, un= signed int nr_pages) struct mem_cgroup *memcg; unsigned short id; =20 - id =3D swap_cgroup_record(entry, 0, nr_pages); + id =3D swap_cgroup_clear(entry, nr_pages); rcu_read_lock(); memcg =3D mem_cgroup_from_id(id); if (memcg) { diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index cf0445cb35ed..be39078f255b 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -21,17 +21,6 @@ struct swap_cgroup_ctrl { =20 static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; =20 -/* - * SwapCgroup implements "lookup" and "exchange" operations. - * In typical usage, this swap_cgroup is accessed via memcg's charge/uncha= rge - * against SwapCache. At swap_free(), this is accessed directly from swap. - * - * This means, - * - we have no race in "exchange" when we're accessed via SwapCache beca= use - * SwapCache(and its swp_entry) is under lock. - * - When called via swap_free(), there is no user of this entry and no r= ace. - * Then, we don't need lock around "exchange". - */ static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, pgoff_t offset) { @@ -63,29 +52,58 @@ static unsigned short __swap_cgroup_id_xchg(struct swap= _cgroup *map, } =20 /** - * swap_cgroup_record - record mem_cgroup for a set of swap entries + * swap_cgroup_record - record mem_cgroup for a set of swap entries. + * These entries must belong to one single folio, and that folio + * must be being charged for swap space (swap out), and these + * entries must not have been charged + * + * @folio: the folio that the swap entry belongs to + * @ent: the first swap entry to be recorded + */ +void swap_cgroup_record(struct folio *folio, swp_entry_t ent) +{ + unsigned int nr_ents =3D folio_nr_pages(folio); + struct swap_cgroup *map; + pgoff_t offset, end; + unsigned short old; + + offset =3D swp_offset(ent); + end =3D offset + nr_ents; + map =3D swap_cgroup_ctrl[swp_type(ent)].map; + + do { + old =3D __swap_cgroup_id_xchg(map, offset, + mem_cgroup_id(folio_memcg(folio))); + VM_BUG_ON(old); + } while (++offset !=3D end); +} + +/** + * swap_cgroup_clear - clear mem_cgroup for a set of swap entries. + * These entries must be being uncharged from swap. They either + * belongs to one single folio in the swap cache (swap in for + * cgroup v1), or no longer have any users (slot freeing). + * * @ent: the first swap entry to be recorded into - * @id: mem_cgroup to be recorded * @nr_ents: number of swap entries to be recorded * - * Returns old value at success, 0 at failure. - * (Of course, old value can be 0.) + * Returns the existing old value. */ -unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, - unsigned int nr_ents) +unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents) { - struct swap_cgroup_ctrl *ctrl; pgoff_t offset =3D swp_offset(ent); pgoff_t end =3D offset + nr_ents; - unsigned short old, iter; struct swap_cgroup *map; + unsigned short old, iter =3D 0; =20 - ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; - map =3D ctrl->map; + offset =3D swp_offset(ent); + end =3D offset + nr_ents; + map =3D swap_cgroup_ctrl[swp_type(ent)].map; =20 - old =3D __swap_cgroup_id_lookup(map, offset); do { - iter =3D __swap_cgroup_id_xchg(map, offset, id); + old =3D __swap_cgroup_id_xchg(map, offset, 0); + if (!iter) + iter =3D old; VM_BUG_ON(iter !=3D old); } while (++offset !=3D end); =20 @@ -119,7 +137,7 @@ int swap_cgroup_swapon(int type, unsigned long max_page= s) =20 BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=3D sizeof(struct swap_cgroup)); - map =3D vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + map =3D vzalloc(DIV_ROUND_UP(max_pages, ID_PER_SC) * sizeof(struct swap_cgroup)); if (!map) goto nomem; --=20 2.47.1