From nobody Thu Apr 25 08:13:57 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:ab0:590e:0:0:0:0:0 with SMTP id n14csp1071730uad; Fri, 29 Jul 2022 11:14:55 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tHkTG/pf95IfkJvH4tZEe8obkGuHnbp/FqxK7yCe9UEapSLICmwz2Mr6eb+ACSzNEY1a4X X-Received: by 2002:a05:622a:40f:b0:31f:83c:e17d with SMTP id n15-20020a05622a040f00b0031f083ce17dmr4504471qtx.165.1659118495658; Fri, 29 Jul 2022 11:14:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659118495; cv=none; d=google.com; s=arc-20160816; b=MQApI1iFqZJWWunDS7/TC4Pi2Kx12l5AAcrdW1QRTxpQFlwv9iTb6cLCQBYHezg9CW Br47h1qg1S1A4zF18NhPt72hxoE2bI/ZxOulTx27bmxSYUp9V7s5w9FPSrjD2UqHrMUZ uBmVE3N4RBXlvfx+ZFjvtD/q2robKW4tV4FWfmqGgVFLJiYek04xBP8/JHNvoK2wSK4j t/z7GH/4u8e9MnLHBiRWFC+oCfccI7DoxtCMYEilFpOZTIMd+SFXRnMvcPEmzyE7hqVs tmuf9FonXujUctsQjVHgSojaJEnsT511djnW6qammEj89pEIDxbk5jE1DGklEl8shsqM IGIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=bOxjCT/a5P87RNatFo2lM2GkXg3NSBmm+aMpbukibtQ=; b=Qnm8YZso62fww3P3/CW+QUpzWDwUsCaBZjYMD8aYeaunkIvDCIgxWKuwsNRURm/nqy KqN3SoL/P2re46oOJdcmFfvMkrcQtoI7iKG1Zns/b9qQ5LHo46DXtXIo258FerZqvDLT jMLPYWcQfTpX1tB1rowPsXpyJO3pa0QAVfEG+sGWBCpR1SOwXu5U0QV5ixE28eCcZxdi 2GhFDpX983IyMlReMH6T9tuQVvrRyJ3e7FtVYurSbsBMBEsKldnXHlfs1kPFu/JQq7EG pMQELTXquH5qDbGBWJIVmLLWFTPGUkRAR9disK8XT6KbL418Mw7RQiKLhH2OriIOVDxw H5Vw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ce6+Gj9s; spf=pass (google.com: domain of mptcp+bounces-6077-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6077-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id e14-20020a0562141d0e00b0047305aae947si1425564qvd.345.2022.07.29.11.14.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jul 2022 11:14:55 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-6077-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ce6+Gj9s; spf=pass (google.com: domain of mptcp+bounces-6077-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6077-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 76FBB1C2090F for ; Fri, 29 Jul 2022 18:14:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 25BD04C8B; Fri, 29 Jul 2022 18:14:54 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99C604C77 for ; Fri, 29 Jul 2022 18:14:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659118490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bOxjCT/a5P87RNatFo2lM2GkXg3NSBmm+aMpbukibtQ=; b=ce6+Gj9sAri6MYA9D15MbdjXb/VE59Jql54+pvitwH1K//GMl2Ml3CzXoqA0eoZgq2o/uP bzre56njwbsahYiV06Xo1457xNKUCbK7wYTG+CWbzEc30goXxs8C0nrvZ2s3D5kOZBz0WW ObpLHWpgawntMewM7iIZ7Oim5VxMatQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-275-UADs1iQBMZeQpZG6MmSVpQ-1; Fri, 29 Jul 2022 14:14:49 -0400 X-MC-Unique: UADs1iQBMZeQpZG6MmSVpQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0CE7C1C05139 for ; Fri, 29 Jul 2022 18:14:49 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.192.205]) by smtp.corp.redhat.com (Postfix) with ESMTP id 91EBC1121314 for ; Fri, 29 Jul 2022 18:14:48 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [PATCH v2 mptcp-next 1/4] mptcp: move RCVPRUNE event later Date: Fri, 29 Jul 2022 20:14:38 +0200 Message-Id: <81be75dff29adfdc136035168f5bcd6d1378b0b4.1659117128.git.pabeni@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" This clean the code a bit, and avoid skipping msk receive buffer update on some weird corner case. Signed-off-by: Paolo Abeni --- net/mptcp/protocol.c | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 970da88cd04f..5af3d591a20b 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -642,6 +642,10 @@ static bool __mptcp_move_skbs_from_subflow(struct mptc= p_sock *msk, } } =20 + /* over limit? can't append more skbs to msk */ + if (__mptcp_rmem(sk) > sk_rbuf) + return true; + pr_debug("msk=3D%p ssk=3D%p", msk, ssk); tp =3D tcp_sk(ssk); do { @@ -786,8 +790,6 @@ static bool move_skbs_to_msk(struct mptcp_sock *msk, st= ruct sock *ssk) void mptcp_data_ready(struct sock *sk, struct sock *ssk) { struct mptcp_subflow_context *subflow =3D mptcp_subflow_ctx(ssk); - struct mptcp_sock *msk =3D mptcp_sk(sk); - int sk_rbuf, ssk_rbuf; =20 /* The peer can send data while we are shutting down this * subflow at msk destruction time, but we must avoid enqueuing @@ -796,20 +798,9 @@ void mptcp_data_ready(struct sock *sk, struct sock *ss= k) if (unlikely(subflow->disposable)) return; =20 - ssk_rbuf =3D READ_ONCE(ssk->sk_rcvbuf); - sk_rbuf =3D READ_ONCE(sk->sk_rcvbuf); - if (unlikely(ssk_rbuf > sk_rbuf)) - sk_rbuf =3D ssk_rbuf; - - /* over limit? can't append more skbs to msk, Also, no need to wake-up*/ - if (__mptcp_rmem(sk) > sk_rbuf) { - MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_RCVPRUNED); - return; - } - /* Wake-up the reader only for in-sequence data */ mptcp_data_lock(sk); - if (move_skbs_to_msk(msk, ssk)) + if (move_skbs_to_msk(mptcp_sk(sk), ssk)) sk->sk_data_ready(sk); =20 mptcp_data_unlock(sk); --=20 2.35.3 From nobody Thu Apr 25 08:13:57 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:ab0:590e:0:0:0:0:0 with SMTP id n14csp1071737uad; Fri, 29 Jul 2022 11:14:56 -0700 (PDT) X-Google-Smtp-Source: AGRyM1urkRpVb24WBLxtocOjdDo4IFKIQ9POAb7bQils3gqVROb7ub8o68pfSYcZbH1W1CBh2c4H X-Received: by 2002:a05:620a:2491:b0:6b5:e53d:25dc with SMTP id i17-20020a05620a249100b006b5e53d25dcmr3665427qkn.540.1659118496680; Fri, 29 Jul 2022 11:14:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659118496; cv=none; d=google.com; s=arc-20160816; b=jhqq9SgG0uJVkct3rc8XiJg1sr1z8gdXbphBpTPiyX4bG5wrpwx1/obggf9u3EgCSo QiZBMaqvIj85ra7skrooYaL/lEwNXBISHTFRbtNBG+axJurOAJj1kKAkgfopvb6H40Rq 8CMWQsjYTLHNOm/AykUQVlqniCkbb8DDg8ap/tt3ajEjaHlenlBlQePW4rsY/KtIc1ng 3gJQviGvDY3yvRINvjdi7+RYB2XcRPh0IZmj4qaFoEGpHzPM/dinbW1m+Xe1sRNG25XB YASxMdsUy4YqUxCg6L1S3GgrFlspgH7NoKWWsuNK11NW8sZAMBWxPyzAMY9aTuvzhPby 8eeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=0cgJGv6YYOKjInMCrAks3Onl/Q3rxx/NYNKZdjKTGbI=; b=gpBRh3QS+pSvjXOdwdHjen0TlPvxw/9kzZrvSvU+N6RxFdBScQmSAV2FTA95WOntSY flTg/ewFIhpUjb5SI0KGq9Yq12QbrEdj2qjS+BAPs5r+12xlEW5dop9Ymx4agnUsRHIB LIu80WDjc70o40NCB9LwNyViccuan3u2O80/gZEGsC0C+MTgFP4WQL3/E/ewIvXcdb2H 3MDf4IOGleJxgtK2TOmhpeAD2Rr1W+V2Y2yeTBaafVKqT5jG5VeltEKGFNCXZsNDBD3E ka+u2gq+b57q+znD2RMb59q4mNoG+XIUjXy4aYa8vCMZMigFotea5hMha5kSbi4V14Ko Shsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JFu9lN5q; spf=pass (google.com: domain of mptcp+bounces-6078-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6078-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id u13-20020a05620a0c4d00b006b5fe583195si2921681qki.708.2022.07.29.11.14.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jul 2022 11:14:56 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-6078-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JFu9lN5q; spf=pass (google.com: domain of mptcp+bounces-6078-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6078-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 8289D1C2093E for ; Fri, 29 Jul 2022 18:14:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 937684C77; Fri, 29 Jul 2022 18:14:54 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 592EA4C87 for ; Fri, 29 Jul 2022 18:14:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659118491; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0cgJGv6YYOKjInMCrAks3Onl/Q3rxx/NYNKZdjKTGbI=; b=JFu9lN5qttApv8YrD0DX1brbGbHNzGZ6gqcF/UqIm//BjnmXbUqLqD9QEqCXxlHu7jXhwA tPi3/SSqEDOW2PFVxCnv4JVF0xWZ0eQJBhwyYQhRXpIQgvsGFVkNQFAEcjqqQ8KpX84eIf ADUJW8J8Ynnjo5Bih9th2LAJGuAsDUg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-611-9j-fXJcNPaiOzJ0QlT8vEg-1; Fri, 29 Jul 2022 14:14:50 -0400 X-MC-Unique: 9j-fXJcNPaiOzJ0QlT8vEg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C0864800124 for ; Fri, 29 Jul 2022 18:14:49 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.192.205]) by smtp.corp.redhat.com (Postfix) with ESMTP id 51B251121314 for ; Fri, 29 Jul 2022 18:14:49 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [PATCH v2 mptcp-next 2/4] mptcp: more accurate receive buffer updates Date: Fri, 29 Jul 2022 20:14:39 +0200 Message-Id: <1da024c8ed9da4fcf78aa2d73e00c1f9e4d3e98f.1659117128.git.pabeni@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" Currently mptcp_cleanup_rbuf() makes a significant effort to avoid acquiring the subflow socket lock, estimating if the tcp level cleanup could actually send an ack. Such estimate is a bit rough when accounting for receive window change, as it consider the msk available buffer space instead of the announced mptcp-level window. Let's consider the announced window instead, mirroring closely the plain TCP implementation. We need to lockless access a bunch of additional, tcp fields. Signed-off-by: Paolo Abeni --- include/net/mptcp.h | 2 ++ net/ipv4/tcp.c | 3 +++ net/mptcp/options.c | 1 - net/mptcp/protocol.c | 14 +++++++++++--- net/mptcp/protocol.h | 6 +++++- 5 files changed, 21 insertions(+), 5 deletions(-) diff --git a/include/net/mptcp.h b/include/net/mptcp.h index 7af7fd48acc7..4b6c66b73bf4 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -136,6 +136,7 @@ static inline bool rsk_drop_req(const struct request_so= ck *req) return tcp_rsk(req)->is_mptcp && tcp_rsk(req)->drop_req; } =20 +void mptcp_receive_window(const struct sock *ssk, int *rwnd); void mptcp_space(const struct sock *ssk, int *space, int *full_space); bool mptcp_syn_options(struct sock *sk, const struct sk_buff *skb, unsigned int *size, struct mptcp_out_options *opts); @@ -284,6 +285,7 @@ static inline bool mptcp_skb_can_collapse(const struct = sk_buff *to, return true; } =20 +static inline void mptcp_receive_window(const struct sock *ssk, int *rwnd)= { } static inline void mptcp_space(const struct sock *ssk, int *s, int *fs) { } static inline void mptcp_seq_show(struct seq_file *seq) { } =20 diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 970e9a2cca4a..d0c0100f74f3 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1606,6 +1606,9 @@ void tcp_cleanup_rbuf(struct sock *sk, int copied) if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) { __u32 rcv_window_now =3D tcp_receive_window(tp); =20 + if (sk_is_mptcp(sk)) + mptcp_receive_window(sk, &rcv_window_now); + /* Optimize, __tcp_select_window() is not cheap. */ if (2*rcv_window_now <=3D tp->window_clamp) { __u32 new_window =3D __tcp_select_window(sk); diff --git a/net/mptcp/options.c b/net/mptcp/options.c index 30d289044e71..563ef8fe5a85 100644 --- a/net/mptcp/options.c +++ b/net/mptcp/options.c @@ -604,7 +604,6 @@ static bool mptcp_established_options_dss(struct sock *= sk, struct sk_buff *skb, } opts->ext_copy.use_ack =3D 1; opts->suboptions =3D OPTION_MPTCP_DSS; - WRITE_ONCE(msk->old_wspace, __mptcp_space((struct sock *)msk)); =20 /* Add kind/length/subtype/flag overhead if mapping is not populated */ if (dss_size =3D=3D 0) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 5af3d591a20b..17e2dbe43639 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -535,6 +535,14 @@ static void mptcp_subflow_cleanup_rbuf(struct sock *ss= k) unlock_sock_fast(ssk, slow); } =20 +void mptcp_receive_window(const struct sock *ssk, int *rwnd) +{ + const struct mptcp_subflow_context *subflow =3D mptcp_subflow_ctx(ssk); + const struct sock *sk =3D subflow->conn; + + *rwnd =3D __mptcp_receive_window(mptcp_sk(sk)); +} + static bool mptcp_subflow_could_cleanup(const struct sock *ssk, bool rx_em= pty) { const struct inet_connection_sock *icsk =3D inet_csk(ssk); @@ -550,13 +558,13 @@ static bool mptcp_subflow_could_cleanup(const struct = sock *ssk, bool rx_empty) =20 static void mptcp_cleanup_rbuf(struct mptcp_sock *msk) { - int old_space =3D READ_ONCE(msk->old_wspace); + int cur_window =3D __mptcp_receive_window(msk); struct mptcp_subflow_context *subflow; struct sock *sk =3D (struct sock *)msk; - int space =3D __mptcp_space(sk); + int new_window =3D __mptcp_space(sk); bool cleanup, rx_empty; =20 - cleanup =3D (space > 0) && (space >=3D (old_space << 1)); + cleanup =3D new_window > 0 && new_window >=3D (cur_window << 1); rx_empty =3D !__mptcp_rmem(sk); =20 mptcp_for_each_subflow(msk, subflow) { diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 1bc9f6e77ddd..1e603f28f1db 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -260,7 +260,6 @@ struct mptcp_sock { int rmem_fwd_alloc; struct sock *last_snd; int snd_burst; - int old_wspace; u64 recovery_snd_nxt; /* in recovery mode accept up to this seq; * recovery related fields are under data_lock * protection @@ -336,6 +335,11 @@ static inline int __mptcp_rmem(const struct sock *sk) return atomic_read(&sk->sk_rmem_alloc) - READ_ONCE(mptcp_sk(sk)->rmem_rel= eased); } =20 +static inline int __mptcp_receive_window(const struct mptcp_sock *msk) +{ + return atomic64_read(&msk->rcv_wnd_sent) - READ_ONCE(msk->ack_seq); +} + static inline int __mptcp_space(const struct sock *sk) { return tcp_win_from_space(sk, READ_ONCE(sk->sk_rcvbuf) - __mptcp_rmem(sk)= ); --=20 2.35.3 From nobody Thu Apr 25 08:13:57 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:ab0:590e:0:0:0:0:0 with SMTP id n14csp1071748uad; Fri, 29 Jul 2022 11:14:57 -0700 (PDT) X-Google-Smtp-Source: AGRyM1uP5J4mPbSG2UXApp3G6RjofYj68RbD1ueG0qgHIMtKKthc1C3SJZOktvjZzIARuLuzqX2d X-Received: by 2002:a63:18a:0:b0:41b:475c:6c08 with SMTP id 132-20020a63018a000000b0041b475c6c08mr3983799pgb.77.1659118497791; Fri, 29 Jul 2022 11:14:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659118497; cv=none; d=google.com; s=arc-20160816; b=OIlQJbthehovv5gs/9rxnzUd0SlV2OO+SF9tSmdrseR9TPIL/68/M4NyRJh7YwMBzK esYzwjeUEi3gPBRyEZUu5+FCV1cOS68XFvfR27jwDaLQI9N8Fjhq9oPaQ5dAHbISrbXd TmpwJghLW5PkkQqGFpBrZtPI7fSjMTDCFVzyiGaNcaj7QM4ADsafDMod8YztS38CL6kb OPtPJzZ0EMdhQq10SXBcu319sEKqwyIoAc49MDaipFOEAyw/R6PJWytRohwnk7ceiU7l Vf/lUOtkwOnhoHb+1syDz2EcICL1SnXd/9RPWqsRgWei3Yrx9G9iToFAY6RTG9uVTcPb w6GQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=GaButeQXGlT9DY7XbRreNiBMhcnzrjF6WGFOjidMj4A=; b=Ix9YLaAKgoFx02lEXp3u4+13nftriH3HJK+fyHe4Hdp1U7ErdAQ1tqo1WfMwbkGZL1 rGD1MCEeam9PkETUYSWGnjbxJTibIbEQ8j/ss3iuXujlIyENgccljpV4rUh1pEe/lXIz 5djK6WNiOBm9/hqjh0f8TkZxHnnrMAl09V97vjtNiRYx2jw/8+jidIsskzKScDS8FhM/ K5BeaOVj4UnhFUl9dMjcSjGuwind4tCRhD3j5QqhxltgZAGggs5GZ4gt6XhcJoqYPFdR C0fJWqi7xCTeh7YjQYq2T2LiFvzFXlWvXzS3P9r6ZtZYPB2bOcJz0MtcB5YAzBMoFDTO GR9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bzml1Rzy; spf=pass (google.com: domain of mptcp+bounces-6079-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6079-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id q20-20020a170902b11400b0016d7b228298si4728804plr.138.2022.07.29.11.14.57 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jul 2022 11:14:57 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-6079-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bzml1Rzy; spf=pass (google.com: domain of mptcp+bounces-6079-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="mptcp+bounces-6079-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 4E304280C36 for ; Fri, 29 Jul 2022 18:14:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5D5694C87; Fri, 29 Jul 2022 18:14:55 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BD514C85 for ; Fri, 29 Jul 2022 18:14:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659118492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GaButeQXGlT9DY7XbRreNiBMhcnzrjF6WGFOjidMj4A=; b=bzml1RzyXdRRGCg6DzbIjZsc01j3yq8RrgYhZnvFJjOkUFyyfESO5bmcAoHneWTfQyx2GF 5i/S8gxM5X5NWm1eWE6K1TUJ2O3XwHjzB76oKOJg1E7HCPeXx2teCWrQGNpKxf+hiuzKiq IB+xbadFcrSnaheWM5284pktX3z0w/4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-26-PBrTYMjvOy28lp8Lc0dPoQ-1; Fri, 29 Jul 2022 14:14:50 -0400 X-MC-Unique: PBrTYMjvOy28lp8Lc0dPoQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8763285A585 for ; Fri, 29 Jul 2022 18:14:50 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.192.205]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14DB91121314 for ; Fri, 29 Jul 2022 18:14:49 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [PATCH v2 mptcp-next 3/4] mptcp: move msk input path under full msk socket lock Date: Fri, 29 Jul 2022 20:14:40 +0200 Message-Id: <24af8221d74f9923858c926d4acb7d81e37dd4d2.1659117128.git.pabeni@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" After commit c2e6048fa1cf ("mptcp: fix race in release_cb") it's pretty straight forward move the whole MPTCP rx path under the socket lock leveraging the release_cb. We can drop a bunch of spin_lock pairs in the receive functions, use a single receive queue and invoke __mptcp_move_skbs only when subflows ask for it. This will allow more cleanup in the next patch. Signed-off-by: Paolo Abeni Reported-by: kernel test robot --- net/mptcp/protocol.c | 78 +++++++++++++++++++------------------------- net/mptcp/protocol.h | 2 +- 2 files changed, 35 insertions(+), 45 deletions(-) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 17e2dbe43639..b7982b578c86 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -795,7 +795,7 @@ static bool move_skbs_to_msk(struct mptcp_sock *msk, st= ruct sock *ssk) return moved > 0; } =20 -void mptcp_data_ready(struct sock *sk, struct sock *ssk) +void __mptcp_data_ready(struct sock *sk, struct sock *ssk) { struct mptcp_subflow_context *subflow =3D mptcp_subflow_ctx(ssk); =20 @@ -807,10 +807,17 @@ void mptcp_data_ready(struct sock *sk, struct sock *s= sk) return; =20 /* Wake-up the reader only for in-sequence data */ - mptcp_data_lock(sk); if (move_skbs_to_msk(mptcp_sk(sk), ssk)) sk->sk_data_ready(sk); +} =20 +void mptcp_data_ready(struct sock *sk, struct sock *ssk) +{ + mptcp_data_lock(sk); + if (!sock_owned_by_user(sk)) + __mptcp_data_ready(sk, ssk); + else + __set_bit(MPTCP_DEQUEUE, &mptcp_sk(sk)->cb_flags); mptcp_data_unlock(sk); } =20 @@ -1768,16 +1775,22 @@ static int mptcp_sendmsg(struct sock *sk, struct ms= ghdr *msg, size_t len) return copied ? : ret; } =20 -static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk, +static bool __mptcp_move_skbs(struct sock *sk); + +static int __mptcp_recvmsg_mskq(struct sock *sk, struct msghdr *msg, size_t len, int flags, struct scm_timestamping_internal *tss, int *cmsg_flags) { + struct mptcp_sock *msk =3D mptcp_sk(sk); struct sk_buff *skb, *tmp; int copied =3D 0; =20 - skb_queue_walk_safe(&msk->receive_queue, skb, tmp) { + if (skb_queue_empty(&sk->sk_receive_queue) && !__mptcp_move_skbs(sk)) + return 0; + + skb_queue_walk_safe(&sk->sk_receive_queue, skb, tmp) { u32 offset =3D MPTCP_SKB_CB(skb)->offset; u32 data_len =3D skb->len - offset; u32 count =3D min_t(size_t, len - copied, data_len); @@ -1811,7 +1824,7 @@ static int __mptcp_recvmsg_mskq(struct mptcp_sock *ms= k, /* we will bulk release the skb memory later */ skb->destructor =3D NULL; WRITE_ONCE(msk->rmem_released, msk->rmem_released + skb->truesize); - __skb_unlink(skb, &msk->receive_queue); + __skb_unlink(skb, &sk->sk_receive_queue); __kfree_skb(skb); } =20 @@ -1932,16 +1945,9 @@ static void __mptcp_update_rmem(struct sock *sk) WRITE_ONCE(msk->rmem_released, 0); } =20 -static void __mptcp_splice_receive_queue(struct sock *sk) +static bool __mptcp_move_skbs(struct sock *sk) { struct mptcp_sock *msk =3D mptcp_sk(sk); - - skb_queue_splice_tail_init(&sk->sk_receive_queue, &msk->receive_queue); -} - -static bool __mptcp_move_skbs(struct mptcp_sock *msk) -{ - struct sock *sk =3D (struct sock *)msk; unsigned int moved =3D 0; bool ret, done; =20 @@ -1949,37 +1955,29 @@ static bool __mptcp_move_skbs(struct mptcp_sock *ms= k) struct sock *ssk =3D mptcp_subflow_recv_lookup(msk); bool slowpath; =20 - /* we can have data pending in the subflows only if the msk - * receive buffer was full at subflow_data_ready() time, - * that is an unlikely slow path. - */ - if (likely(!ssk)) + if (unlikely(!ssk)) break; =20 slowpath =3D lock_sock_fast(ssk); - mptcp_data_lock(sk); __mptcp_update_rmem(sk); done =3D __mptcp_move_skbs_from_subflow(msk, ssk, &moved); - mptcp_data_unlock(sk); =20 if (unlikely(ssk->sk_err)) __mptcp_error_report(sk); unlock_sock_fast(ssk, slowpath); } while (!done); =20 - /* acquire the data lock only if some input data is pending */ ret =3D moved > 0; if (!RB_EMPTY_ROOT(&msk->out_of_order_queue) || - !skb_queue_empty_lockless(&sk->sk_receive_queue)) { - mptcp_data_lock(sk); + !skb_queue_empty(&sk->sk_receive_queue)) { __mptcp_update_rmem(sk); ret |=3D __mptcp_ofo_queue(msk); - __mptcp_splice_receive_queue(sk); - mptcp_data_unlock(sk); } - if (ret) + if (ret) { + mptcp_cleanup_rbuf(msk); mptcp_check_data_fin((struct sock *)msk); - return !skb_queue_empty(&msk->receive_queue); + } + return ret; } =20 static unsigned int mptcp_inq_hint(const struct sock *sk) @@ -1987,7 +1985,7 @@ static unsigned int mptcp_inq_hint(const struct sock = *sk) const struct mptcp_sock *msk =3D mptcp_sk(sk); const struct sk_buff *skb; =20 - skb =3D skb_peek(&msk->receive_queue); + skb =3D skb_peek(&sk->sk_receive_queue); if (skb) { u64 hint_val =3D msk->ack_seq - MPTCP_SKB_CB(skb)->map_seq; =20 @@ -2033,7 +2031,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msgh= dr *msg, size_t len, while (copied < len) { int bytes_read; =20 - bytes_read =3D __mptcp_recvmsg_mskq(msk, msg, len - copied, flags, &tss,= &cmsg_flags); + bytes_read =3D __mptcp_recvmsg_mskq(sk, msg, len - copied, flags, &tss, = &cmsg_flags); if (unlikely(bytes_read < 0)) { if (!copied) copied =3D bytes_read; @@ -2045,9 +2043,6 @@ static int mptcp_recvmsg(struct sock *sk, struct msgh= dr *msg, size_t len, /* be sure to advertise window change */ mptcp_cleanup_rbuf(msk); =20 - if (skb_queue_empty(&msk->receive_queue) && __mptcp_move_skbs(msk)) - continue; - /* only the master socket status is relevant here. The exit * conditions mirror closely tcp_recvmsg() */ @@ -2074,7 +2069,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msgh= dr *msg, size_t len, /* race breaker: the shutdown could be after the * previous receive queue check */ - if (__mptcp_move_skbs(msk)) + if (__mptcp_move_skbs(sk)) continue; break; } @@ -2111,9 +2106,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msgh= dr *msg, size_t len, } } =20 - pr_debug("msk=3D%p rx queue empty=3D%d:%d copied=3D%d", - msk, skb_queue_empty_lockless(&sk->sk_receive_queue), - skb_queue_empty(&msk->receive_queue), copied); + pr_debug("msk=3D%p rx queue empty=3D%d copied=3D%d", + msk, skb_queue_empty(&sk->sk_receive_queue), copied); if (!(flags & MSG_PEEK)) mptcp_rcv_space_adjust(msk, copied); =20 @@ -2566,7 +2560,6 @@ static int __mptcp_init_sock(struct sock *sk) INIT_LIST_HEAD(&msk->join_list); INIT_LIST_HEAD(&msk->rtx_queue); INIT_WORK(&msk->work, mptcp_worker); - __skb_queue_head_init(&msk->receive_queue); msk->out_of_order_queue =3D RB_ROOT; msk->first_pending =3D NULL; msk->rmem_fwd_alloc =3D 0; @@ -3048,12 +3041,8 @@ void mptcp_destroy_common(struct mptcp_sock *msk, un= signed int flags) mptcp_for_each_subflow_safe(msk, subflow, tmp) __mptcp_close_ssk(sk, mptcp_subflow_tcp_sock(subflow), subflow, flags); =20 - /* move to sk_receive_queue, sk_stream_kill_queues will purge it */ - mptcp_data_lock(sk); - skb_queue_splice_tail_init(&msk->receive_queue, &sk->sk_receive_queue); __skb_queue_purge(&sk->sk_receive_queue); skb_rbtree_purge(&msk->out_of_order_queue); - mptcp_data_unlock(sk); =20 /* move all the rx fwd alloc into the sk_mem_reclaim_final in * inet_sock_destruct() will dispose it @@ -3135,6 +3124,8 @@ static void mptcp_release_cb(struct sock *sk) __mptcp_flush_join_list(sk); if (flags & BIT(MPTCP_PUSH_PENDING)) __mptcp_push_pending(sk, 0); + if ((flags & BIT(MPTCP_DEQUEUE)) && __mptcp_move_skbs(sk)) + sk->sk_data_ready(sk); if (flags & BIT(MPTCP_RETRANSMIT)) __mptcp_retrans(sk); =20 @@ -3383,7 +3374,7 @@ static int mptcp_ioctl(struct sock *sk, int cmd, unsi= gned long arg) return -EINVAL; =20 lock_sock(sk); - __mptcp_move_skbs(msk); + __mptcp_move_skbs(sk); answ =3D mptcp_inq_hint(sk); release_sock(sk); break; @@ -3619,8 +3610,7 @@ static __poll_t mptcp_check_readable(struct mptcp_soc= k *msk) /* Concurrent splices from sk_receive_queue into receive_queue will * always show at least one non-empty queue when checked in this order. */ - if (skb_queue_empty_lockless(&((struct sock *)msk)->sk_receive_queue) && - skb_queue_empty_lockless(&msk->receive_queue)) + if (skb_queue_empty_lockless(&((struct sock *)msk)->sk_receive_queue)) return 0; =20 return EPOLLIN | EPOLLRDNORM; diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 1e603f28f1db..f12a6e80171d 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -125,6 +125,7 @@ #define MPTCP_FLUSH_JOIN_LIST 5 #define MPTCP_CONNECTED 6 #define MPTCP_RESET_SCHEDULER 7 +#define MPTCP_DEQUEUE 8 =20 static inline bool before64(__u64 seq1, __u64 seq2) { @@ -288,7 +289,6 @@ struct mptcp_sock { struct work_struct work; struct sk_buff *ooo_last_skb; struct rb_root out_of_order_queue; - struct sk_buff_head receive_queue; struct list_head conn_list; struct list_head rtx_queue; struct mptcp_data_frag *first_pending; --=20 2.35.3 From nobody Thu Apr 25 08:13:57 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:ab0:590e:0:0:0:0:0 with SMTP id n14csp1071763uad; Fri, 29 Jul 2022 11:14:59 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tge/wlTZoSGSY3E53Q+r4pvPa/If1dRayo9w/IXLl4UmU2v4y6URsiImKdktTHSqt+N4RK X-Received: by 2002:a63:cd4d:0:b0:41a:f9b8:e221 with SMTP id a13-20020a63cd4d000000b0041af9b8e221mr3898889pgj.321.1659118499121; Fri, 29 Jul 2022 11:14:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659118499; cv=none; d=google.com; s=arc-20160816; b=cDTtKUO4G2niRDsk0H3QpIG8VF8wszD8fkOcXT//DteJz0LBvywjcmtZvKtgYHwE8R BYL36UD1QbxxQR3mEBYRPsk8nXUldOuXhk/G61syy4bsgW7MA0v+7DMOVFgwBpZkHMJP u6hZoJkTDy9ThNOxoeDOGGPPxo4vHrzyN6OBnF+ly5ppuiWRyfoutz/w0CuqJpmskCfS 20iIwVqvCYIuxE6RZxiUDb329rwS1mi7e6aX0gWp39JuWCtjvxGjtJEgPNyx+9ZVYRRF ehluB7e3STQ1PPatISvxYfku9MmiqVSBZSkEsmkotGWi2kvrKnV7x/O8mX/XiZKmCf6z Yr/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=bFNE4JGHhubGyGVygNNiX2rA+9CcUBUuFfxxhypH3v8=; b=K+6sHNGBuXZEFRqu46rSPD7hFo0AE69fQ9ZAF4BJ/Oq8WPnebMOmAOcm/akGb0O8OL xOahimFpZw7VIFm49ouG9MwzBnchd/lPQAXGnU3kzEW+XEOLMWSwffF3fCCY8frK02B+ /SrU4ed+IZ2rVFrgpo1Dg0xDbT0+GjomXWx8zwf2Sn3YQ33sImuEE1Dcbj/BSBSFuhY+ RaYfCHL6ofYFHXPm4AUoiGG7y+m+gZWeasnFm5hdMhxXPY3+p3A8IQZXpM/5kFssgJ+N rXj1J7idiESybCLCpifL2tmoYU08/qsupiJN0y4iDuxLtyOCI7dSqCHN4TJcdOMhT6yi SxIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hQxHsnq2; spf=pass (google.com: domain of mptcp+bounces-6080-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 139.178.88.99 as permitted sender) smtp.mailfrom="mptcp+bounces-6080-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id n14-20020a170902d2ce00b0016d981eb1bdsi5350116plc.336.2022.07.29.11.14.58 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jul 2022 11:14:59 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-6080-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hQxHsnq2; spf=pass (google.com: domain of mptcp+bounces-6080-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 139.178.88.99 as permitted sender) smtp.mailfrom="mptcp+bounces-6080-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 80B5B280C4D for ; Fri, 29 Jul 2022 18:14:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B9BC04C85; Fri, 29 Jul 2022 18:14:55 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D53D94C89 for ; Fri, 29 Jul 2022 18:14:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659118492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bFNE4JGHhubGyGVygNNiX2rA+9CcUBUuFfxxhypH3v8=; b=hQxHsnq2CBNOv+7KAuNgmJTmrsVtcSMTTtnETd06QLED/CsIi4CqaLnYdO860XbBI+ktMN gBSvdsMb+Rwz5Eu1dVttvQFDyU4JKYrN5vGzYEkzcL72XZ+iNIZkNsw84nccCVpzgi6GEY 0UQaGrLCXlugmj0ZKfkUTAMcWG08ftQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-326-VkAux13tMiGDR7IrZY61gQ-1; Fri, 29 Jul 2022 14:14:51 -0400 X-MC-Unique: VkAux13tMiGDR7IrZY61gQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4AAC3802D2C for ; Fri, 29 Jul 2022 18:14:51 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.192.205]) by smtp.corp.redhat.com (Postfix) with ESMTP id CC5B71121314 for ; Fri, 29 Jul 2022 18:14:50 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [PATCH v2 mptcp-next 4/4] mptcp: use common helper for rmem memory accounting Date: Fri, 29 Jul 2022 20:14:41 +0200 Message-Id: In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" After the previous patch, updating sk_forward_memory is cheap and we can drop a lot of complexity from the MPTCP memory acconting, using the common helper for that. Signed-off-by: Paolo Abeni --- net/mptcp/protocol.c | 112 ++++--------------------------------------- net/mptcp/protocol.h | 4 +- 2 files changed, 9 insertions(+), 107 deletions(-) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index b7982b578c86..f6ff130f39c1 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -131,11 +131,6 @@ static void mptcp_drop(struct sock *sk, struct sk_buff= *skb) __kfree_skb(skb); } =20 -static void mptcp_rmem_charge(struct sock *sk, int size) -{ - mptcp_sk(sk)->rmem_fwd_alloc -=3D size; -} - static bool mptcp_try_coalesce(struct sock *sk, struct sk_buff *to, struct sk_buff *from) { @@ -152,7 +147,7 @@ static bool mptcp_try_coalesce(struct sock *sk, struct = sk_buff *to, MPTCP_SKB_CB(to)->end_seq =3D MPTCP_SKB_CB(from)->end_seq; kfree_skb_partial(from, fragstolen); atomic_add(delta, &sk->sk_rmem_alloc); - mptcp_rmem_charge(sk, delta); + sk->sk_forward_alloc -=3D delta; return true; } =20 @@ -165,44 +160,6 @@ static bool mptcp_ooo_try_coalesce(struct mptcp_sock *= msk, struct sk_buff *to, return mptcp_try_coalesce((struct sock *)msk, to, from); } =20 -static void __mptcp_rmem_reclaim(struct sock *sk, int amount) -{ - amount >>=3D PAGE_SHIFT; - mptcp_sk(sk)->rmem_fwd_alloc -=3D amount << PAGE_SHIFT; - __sk_mem_reduce_allocated(sk, amount); -} - -static void mptcp_rmem_uncharge(struct sock *sk, int size) -{ - struct mptcp_sock *msk =3D mptcp_sk(sk); - int reclaimable; - - msk->rmem_fwd_alloc +=3D size; - reclaimable =3D msk->rmem_fwd_alloc - sk_unused_reserved_mem(sk); - - /* see sk_mem_uncharge() for the rationale behind the following schema */ - if (unlikely(reclaimable >=3D PAGE_SIZE)) - __mptcp_rmem_reclaim(sk, reclaimable); -} - -static void mptcp_rfree(struct sk_buff *skb) -{ - unsigned int len =3D skb->truesize; - struct sock *sk =3D skb->sk; - - atomic_sub(len, &sk->sk_rmem_alloc); - mptcp_rmem_uncharge(sk, len); -} - -static void mptcp_set_owner_r(struct sk_buff *skb, struct sock *sk) -{ - skb_orphan(skb); - skb->sk =3D sk; - skb->destructor =3D mptcp_rfree; - atomic_add(skb->truesize, &sk->sk_rmem_alloc); - mptcp_rmem_charge(sk, skb->truesize); -} - /* "inspired" by tcp_data_queue_ofo(), main differences: * - use mptcp seqs * - don't cope with sacks @@ -315,25 +272,7 @@ static void mptcp_data_queue_ofo(struct mptcp_sock *ms= k, struct sk_buff *skb) =20 end: skb_condense(skb); - mptcp_set_owner_r(skb, sk); -} - -static bool mptcp_rmem_schedule(struct sock *sk, struct sock *ssk, int siz= e) -{ - struct mptcp_sock *msk =3D mptcp_sk(sk); - int amt, amount; - - if (size <=3D msk->rmem_fwd_alloc) - return true; - - size -=3D msk->rmem_fwd_alloc; - amt =3D sk_mem_pages(size); - amount =3D amt << PAGE_SHIFT; - if (!__sk_mem_raise_allocated(sk, size, amt, SK_MEM_RECV)) - return false; - - msk->rmem_fwd_alloc +=3D amount; - return true; + skb_set_owner_r(skb, sk); } =20 static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk, @@ -350,8 +289,7 @@ static bool __mptcp_move_skb(struct mptcp_sock *msk, st= ruct sock *ssk, skb_ext_reset(skb); skb_orphan(skb); =20 - /* try to fetch required memory from subflow */ - if (!mptcp_rmem_schedule(sk, ssk, skb->truesize)) + if (!sk_rmem_schedule(sk, skb, skb->truesize)) goto drop; =20 has_rxtstamp =3D TCP_SKB_CB(skb)->has_rxtstamp; @@ -372,7 +310,7 @@ static bool __mptcp_move_skb(struct mptcp_sock *msk, st= ruct sock *ssk, if (tail && mptcp_try_coalesce(sk, tail, skb)) return true; =20 - mptcp_set_owner_r(skb, sk); + skb_set_owner_r(skb, sk); __skb_queue_tail(&sk->sk_receive_queue, skb); return true; } else if (after64(MPTCP_SKB_CB(skb)->map_seq, msk->ack_seq)) { @@ -1783,7 +1721,6 @@ static int __mptcp_recvmsg_mskq(struct sock *sk, struct scm_timestamping_internal *tss, int *cmsg_flags) { - struct mptcp_sock *msk =3D mptcp_sk(sk); struct sk_buff *skb, *tmp; int copied =3D 0; =20 @@ -1821,9 +1758,10 @@ static int __mptcp_recvmsg_mskq(struct sock *sk, } =20 if (!(flags & MSG_PEEK)) { - /* we will bulk release the skb memory later */ + /* avoid the indirect call, we know the destructor is sock_wfree */ skb->destructor =3D NULL; - WRITE_ONCE(msk->rmem_released, msk->rmem_released + skb->truesize); + atomic_sub(skb->truesize, &sk->sk_rmem_alloc); + sk_mem_uncharge(sk, skb->truesize); __skb_unlink(skb, &sk->sk_receive_queue); __kfree_skb(skb); } @@ -1933,18 +1871,6 @@ static void mptcp_rcv_space_adjust(struct mptcp_sock= *msk, int copied) msk->rcvq_space.time =3D mstamp; } =20 -static void __mptcp_update_rmem(struct sock *sk) -{ - struct mptcp_sock *msk =3D mptcp_sk(sk); - - if (!msk->rmem_released) - return; - - atomic_sub(msk->rmem_released, &sk->sk_rmem_alloc); - mptcp_rmem_uncharge(sk, msk->rmem_released); - WRITE_ONCE(msk->rmem_released, 0); -} - static bool __mptcp_move_skbs(struct sock *sk) { struct mptcp_sock *msk =3D mptcp_sk(sk); @@ -1959,7 +1885,6 @@ static bool __mptcp_move_skbs(struct sock *sk) break; =20 slowpath =3D lock_sock_fast(ssk); - __mptcp_update_rmem(sk); done =3D __mptcp_move_skbs_from_subflow(msk, ssk, &moved); =20 if (unlikely(ssk->sk_err)) @@ -1968,11 +1893,7 @@ static bool __mptcp_move_skbs(struct sock *sk) } while (!done); =20 ret =3D moved > 0; - if (!RB_EMPTY_ROOT(&msk->out_of_order_queue) || - !skb_queue_empty(&sk->sk_receive_queue)) { - __mptcp_update_rmem(sk); - ret |=3D __mptcp_ofo_queue(msk); - } + ret |=3D __mptcp_ofo_queue(msk); if (ret) { mptcp_cleanup_rbuf(msk); mptcp_check_data_fin((struct sock *)msk); @@ -2562,8 +2483,6 @@ static int __mptcp_init_sock(struct sock *sk) INIT_WORK(&msk->work, mptcp_worker); msk->out_of_order_queue =3D RB_ROOT; msk->first_pending =3D NULL; - msk->rmem_fwd_alloc =3D 0; - WRITE_ONCE(msk->rmem_released, 0); msk->timer_ival =3D TCP_RTO_MIN; =20 msk->first =3D NULL; @@ -2775,8 +2694,6 @@ static void __mptcp_destroy_sock(struct sock *sk) =20 sk->sk_prot->destroy(sk); =20 - WARN_ON_ONCE(msk->rmem_fwd_alloc); - WARN_ON_ONCE(msk->rmem_released); sk_stream_kill_queues(sk); xfrm_sk_free_policy(sk); =20 @@ -3044,11 +2961,6 @@ void mptcp_destroy_common(struct mptcp_sock *msk, un= signed int flags) __skb_queue_purge(&sk->sk_receive_queue); skb_rbtree_purge(&msk->out_of_order_queue); =20 - /* move all the rx fwd alloc into the sk_mem_reclaim_final in - * inet_sock_destruct() will dispose it - */ - sk->sk_forward_alloc +=3D msk->rmem_fwd_alloc; - msk->rmem_fwd_alloc =3D 0; mptcp_token_destroy(msk); mptcp_pm_free_anno_list(msk); mptcp_free_local_addr_list(msk); @@ -3146,8 +3058,6 @@ static void mptcp_release_cb(struct sock *sk) if (__test_and_clear_bit(MPTCP_RESET_SCHEDULER, &msk->cb_flags)) msk->last_snd =3D NULL; } - - __mptcp_update_rmem(sk); } =20 /* MP_JOIN client subflow must wait for 4th ack before sending any data: @@ -3328,11 +3238,6 @@ static void mptcp_shutdown(struct sock *sk, int how) __mptcp_wr_shutdown(sk); } =20 -static int mptcp_forward_alloc_get(const struct sock *sk) -{ - return sk->sk_forward_alloc + mptcp_sk(sk)->rmem_fwd_alloc; -} - static int mptcp_ioctl_outq(const struct mptcp_sock *msk, u64 v) { const struct sock *sk =3D (void *)msk; @@ -3413,7 +3318,6 @@ static struct proto mptcp_prot =3D { .hash =3D mptcp_hash, .unhash =3D mptcp_unhash, .get_port =3D mptcp_get_port, - .forward_alloc_get =3D mptcp_forward_alloc_get, .sockets_allocated =3D &mptcp_sockets_allocated, =20 .memory_allocated =3D &tcp_memory_allocated, diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index f12a6e80171d..d4a267cb7663 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -258,7 +258,6 @@ struct mptcp_sock { u64 ack_seq; atomic64_t rcv_wnd_sent; u64 rcv_data_fin_seq; - int rmem_fwd_alloc; struct sock *last_snd; int snd_burst; u64 recovery_snd_nxt; /* in recovery mode accept up to this seq; @@ -269,7 +268,6 @@ struct mptcp_sock { u64 wnd_end; unsigned long timer_ival; u32 token; - int rmem_released; unsigned long flags; unsigned long cb_flags; unsigned long push_pending; @@ -332,7 +330,7 @@ static inline struct mptcp_sock *mptcp_sk(const struct = sock *sk) */ static inline int __mptcp_rmem(const struct sock *sk) { - return atomic_read(&sk->sk_rmem_alloc) - READ_ONCE(mptcp_sk(sk)->rmem_rel= eased); + return atomic_read(&sk->sk_rmem_alloc); } =20 static inline int __mptcp_receive_window(const struct mptcp_sock *msk) --=20 2.35.3