From nobody Mon Feb 9 02:27:08 2026 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC3547F496; Tue, 12 Mar 2024 14:28:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710253703; cv=none; b=OxYFs15IOrOn1QSaQWnUhpwLtBwOZUVB/iNYakgCRFsr50lAPaYnVrmL4hdRc2peEwYNEUb0oJCUZ1Hdk4KzYU5734GVxVAvHhT53XkjUIlaQxgqxO1D9JbtjbblVGJH2Z6Lbgbi1Y1+YmvO7+xGykO23wKpqQG3Dy9lpOZ86kU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710253703; c=relaxed/simple; bh=448tqIjdGJFxxJ0g+tBbMKYg5dpbKXvzMlxswSFA2HU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O74eLYSKhrjzBMs+lcNDgp54YyiiRTWpBfHR1rKDlhEY3lqMCpx+APXHQWY5cCio6GToHLnMhvDQpQbrmuIhs9u6W59zTpaG9peSGSFLOrLtFbnmO1Feyupj25yBYDrp408KFYlpgn0I9A4BJZBoF1FwDPZBtpoxB/E3bmOu8kk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=TZK9uSkN; arc=none smtp.client-ip=115.124.30.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="TZK9uSkN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1710253698; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=fSelHZESROM0Ka6YP0PC72uoMYqmMaUB1kibaryyfU4=; b=TZK9uSkNWg9EsQvGD9gipbDfac0tqvRLPFNbykGHGPUAIAFN7BXaE/ga4E7+iyG/nk+cqz0MwsUlu6PZrvlmaZh7xsxZphrZshkH1T0JmpTVRNZE98VRCkgLVFZWKMm0oqsN20uiSDI2Zv+DzLM/NJWJtiGewszvB71ytFs654Y= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0W2M60XI_1710253696; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0W2M60XI_1710253696) by smtp.aliyun-inc.com; Tue, 12 Mar 2024 22:28:17 +0800 From: Wen Gu To: wintera@linux.ibm.com, twinkler@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, wenjia@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH net-next v3 09/11] net/smc: attach or detach ghost sndbuf to peer DMB Date: Tue, 12 Mar 2024 22:27:41 +0800 Message-Id: <20240312142743.41406-10-guwen@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240312142743.41406-1-guwen@linux.alibaba.com> References: <20240312142743.41406-1-guwen@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The ghost sndbuf descriptor will be created and attached to peer DMB once peer token is obtained and it will be detach and freed when the connection is freed. Signed-off-by: Wen Gu --- net/smc/af_smc.c | 16 ++++++++++++ net/smc/smc_core.c | 61 +++++++++++++++++++++++++++++++++++++++++++++- net/smc/smc_core.h | 1 + 3 files changed, 77 insertions(+), 1 deletion(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 751a8fdd6eb0..fce7a5b2ce5c 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -1438,6 +1438,14 @@ static int smc_connect_ism(struct smc_sock *smc, } =20 smc_conn_save_peer_info(smc, aclc); + + if (smc_ism_support_dmb_nocopy(smc->conn.lgr->smcd)) { + rc =3D smcd_buf_attach(smc); + if (rc) { + rc =3D SMC_CLC_DECL_MEM; /* try to fallback */ + goto connect_abort; + } + } smc_close_init(smc); smc_rx_init(smc); smc_tx_init(smc); @@ -2542,6 +2550,14 @@ static void smc_listen_work(struct work_struct *work) mutex_unlock(&smc_server_lgr_pending); } smc_conn_save_peer_info(new_smc, cclc); + + if (ini->is_smcd && + smc_ism_support_dmb_nocopy(new_smc->conn.lgr->smcd)) { + rc =3D smcd_buf_attach(new_smc); + if (rc) + goto out_decl; + } + smc_listen_out_connected(new_smc); SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini); goto out_free; diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c index 9b84d5897aa5..fafdb97adfad 100644 --- a/net/smc/smc_core.c +++ b/net/smc/smc_core.c @@ -1149,6 +1149,20 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_= desc, bool is_rmb, } } =20 +static void smcd_buf_detach(struct smc_connection *conn) +{ + struct smcd_dev *smcd =3D conn->lgr->smcd; + u64 peer_token =3D conn->peer_token; + + if (!conn->sndbuf_desc) + return; + + smc_ism_detach_dmb(smcd, peer_token); + + kfree(conn->sndbuf_desc); + conn->sndbuf_desc =3D NULL; +} + static void smc_buf_unuse(struct smc_connection *conn, struct smc_link_group *lgr) { @@ -1192,6 +1206,8 @@ void smc_conn_free(struct smc_connection *conn) if (lgr->is_smcd) { if (!list_empty(&lgr->list)) smc_ism_unset_conn(conn); + if (smc_ism_support_dmb_nocopy(lgr->smcd)) + smcd_buf_detach(conn); tasklet_kill(&conn->rx_tsklet); } else { smc_cdc_wait_pend_tx_wr(conn); @@ -1445,6 +1461,8 @@ static void smc_conn_kill(struct smc_connection *conn= , bool soft) smc_sk_wake_ups(smc); if (conn->lgr->is_smcd) { smc_ism_unset_conn(conn); + if (smc_ism_support_dmb_nocopy(conn->lgr->smcd)) + smcd_buf_detach(conn); if (soft) tasklet_kill(&conn->rx_tsklet); else @@ -2464,12 +2482,18 @@ int smc_buf_create(struct smc_sock *smc, bool is_sm= cd) int rc; =20 /* create send buffer */ + if (is_smcd && + smc_ism_support_dmb_nocopy(smc->conn.lgr->smcd)) + goto create_rmb; + rc =3D __smc_buf_create(smc, is_smcd, false); if (rc) return rc; + +create_rmb: /* create rmb */ rc =3D __smc_buf_create(smc, is_smcd, true); - if (rc) { + if (rc && smc->conn.sndbuf_desc) { down_write(&smc->conn.lgr->sndbufs_lock); list_del(&smc->conn.sndbuf_desc->list); up_write(&smc->conn.lgr->sndbufs_lock); @@ -2479,6 +2503,41 @@ int smc_buf_create(struct smc_sock *smc, bool is_smc= d) return rc; } =20 +int smcd_buf_attach(struct smc_sock *smc) +{ + struct smc_connection *conn =3D &smc->conn; + struct smcd_dev *smcd =3D conn->lgr->smcd; + u64 peer_token =3D conn->peer_token; + struct smc_buf_desc *buf_desc; + int rc; + + buf_desc =3D kzalloc(sizeof(*buf_desc), GFP_KERNEL); + if (!buf_desc) + return -ENOMEM; + + /* The ghost sndbuf_desc describes the same memory region as + * peer RMB. Its lifecycle is consistent with the connection's + * and it will be freed with the connections instead of the + * link group. + */ + rc =3D smc_ism_attach_dmb(smcd, peer_token, buf_desc); + if (rc) + goto free; + + smc->sk.sk_sndbuf =3D buf_desc->len; + buf_desc->cpu_addr =3D + (u8 *)buf_desc->cpu_addr + sizeof(struct smcd_cdc_msg); + buf_desc->len -=3D sizeof(struct smcd_cdc_msg); + conn->sndbuf_desc =3D buf_desc; + conn->sndbuf_desc->used =3D 1; + atomic_set(&conn->sndbuf_space, conn->sndbuf_desc->len); + return 0; + +free: + kfree(buf_desc); + return rc; +} + static inline int smc_rmb_reserve_rtoken_idx(struct smc_link_group *lgr) { int i; diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h index 1f175376037b..d93cf51dbd7c 100644 --- a/net/smc/smc_core.h +++ b/net/smc/smc_core.h @@ -557,6 +557,7 @@ void smc_smcd_terminate(struct smcd_dev *dev, struct sm= cd_gid *peer_gid, void smc_smcd_terminate_all(struct smcd_dev *dev); void smc_smcr_terminate_all(struct smc_ib_device *smcibdev); int smc_buf_create(struct smc_sock *smc, bool is_smcd); +int smcd_buf_attach(struct smc_sock *smc); int smc_uncompress_bufsize(u8 compressed); int smc_rmb_rtoken_handling(struct smc_connection *conn, struct smc_link *= link, struct smc_clc_msg_accept_confirm *clc); --=20 2.32.0.3.g01195cf9f