From nobody Mon Feb 9 22:39:00 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1620993009; cv=none; d=zohomail.com; s=zohoarc; b=cq9xfqk9aE+kMyqdNDt2FTJz2aowfQIO9jqdbfsyBRLrhfj3rVOW3dajwdh2YzM4xerOjh59FJVTDisXuh0DNLQVWTvvzdZ9jRsW+muv0/4+kvzzPkDOqTiGmNCI41oU8QcD5gkG7jmpjXNKHhz07c387CmFkeO4n0Itk76MHYU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1620993009; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=T0Wbu6qoiD9bpA7ptuoV/Y2lkGPGEkTfUfR3V0sHXsw=; b=JyQoryJKEjDkG3Hqqo6WKY7N3tC53cgdl9sBvdN5zARdKRBvzFgj+hi/RoCvkgUeQBzcSdpG6NIgM3jr3VLeUG/SvKrXPz/qJ2I6v5wPJvNrsc8fY8ic5SvUrChn+Q93Hqg1lvUiXhG0ZciSG3EDQnXujaZF9lK2uxWGsz+ahBY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1620993009420441.4895702793566; Fri, 14 May 2021 04:50:09 -0700 (PDT) Received: from localhost ([::1]:39662 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhWKW-0004Vk-Dy for importer@patchew.org; Fri, 14 May 2021 07:50:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50866) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhWJR-0002QM-8W for qemu-devel@nongnu.org; Fri, 14 May 2021 07:49:01 -0400 Received: from mail-lf1-x12a.google.com ([2a00:1450:4864:20::12a]:43904) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lhWJM-0004pt-3k for qemu-devel@nongnu.org; Fri, 14 May 2021 07:49:01 -0400 Received: by mail-lf1-x12a.google.com with SMTP id x2so42680674lff.10 for ; Fri, 14 May 2021 04:48:55 -0700 (PDT) Received: from navi.cosmonova.net.ua ([95.67.24.131]) by smtp.gmail.com with ESMTPSA id t13sm583421lfc.57.2021.05.14.04.48.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 04:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T0Wbu6qoiD9bpA7ptuoV/Y2lkGPGEkTfUfR3V0sHXsw=; b=cvBr0cHF1fHjg4K2jZ7EnJboABI5dd0Wbl+LIbzgMk7TNNvJFGKXtRvMPJwklxSk6i hBgrUiBUMtFXUZseJCZaV3BfuDEZpTTFBOzCL8g8beDK9ZAVYToYFXVeEUJB/R3SoOxo PWUIw8s6iOu2+StMu1cORJB6daf5d0IIXI8EKMwxAeEOZUxM6GVqEkkoloq3otQ+KHt+ VwxrkyJDId/nfF80GVG4yD23OYMRW1tvNmd+TWdCNaCacaKWw7JVSkXu9XwCFtte/c4m kr9a3mWAYMIXHPalYqIB3IRw1+DMBWHOznoEINhiFC1VQ45t/dCzcDJGRx+hZks5Wb9b 3nng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T0Wbu6qoiD9bpA7ptuoV/Y2lkGPGEkTfUfR3V0sHXsw=; b=mCh6YCZPkf2OZTJWKsi2AxJptBiuzgP71s5yi44yXA+ryPXNb7oNs9tydSlmRYsVR2 5hF+Glt48SO/IeqIn93qjHd8IZNxTA1dgpJrubqSx0mNf+HgsaFgXmBhgQYcGqc6FRm2 SyQ33C1kPIa7ypBSSpVJ2TmvQntn0edpdj7waf+C9sSc5G3yfLpFrBZH2mCa2XFccEUi iKgX+o9JOkiHewMdN11APaECr0t7qumDHnR5OS5vIrUu2/SqfDsKPh/uuYZruGRyTPF7 QnjNwTdzqQSXsDteFpC7qH3m/iethb3qCO2tGFkA/72GhIgxgoxzoo9oLHStim2nCHJU cYMw== X-Gm-Message-State: AOAM531rRQY0uVBnQhpGkds3zwMF525tKvsXDMx8RozJOjLpEWx29Wgq Vu06NXEDdZB8+tIxN3VL1D+wpg== X-Google-Smtp-Source: ABdhPJwc0MzMB2ehfiYG0W/kJtb4g+Qxu/OGsOu1kI8zvW0FCOk4xAw+wW4rzes8YkHj8jZBrSl6EA== X-Received: by 2002:ac2:5a5c:: with SMTP id r28mr32361716lfn.126.1620992934595; Fri, 14 May 2021 04:48:54 -0700 (PDT) From: Andrew Melnychenko To: jasowang@redhat.com, mst@redhat.com Subject: [PATCH v8 6/7] docs: Added eBPF documentation. Date: Fri, 14 May 2021 14:48:34 +0300 Message-Id: <20210514114835.267316-7-andrew@daynix.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210514114835.267316-1-andrew@daynix.com> References: <20210514114835.267316-1-andrew@daynix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: none client-ip=2a00:1450:4864:20::12a; envelope-from=andrew@daynix.com; helo=mail-lf1-x12a.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yan@daynix.com, yuri.benditovich@daynix.com, berrange@redhat.com, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Signed-off-by: Yuri Benditovich Signed-off-by: Andrew Melnychenko --- docs/devel/ebpf_rss.rst | 125 ++++++++++++++++++++++++++++++++++++++++ docs/devel/index.rst | 1 + 2 files changed, 126 insertions(+) create mode 100644 docs/devel/ebpf_rss.rst diff --git a/docs/devel/ebpf_rss.rst b/docs/devel/ebpf_rss.rst new file mode 100644 index 0000000000..e00962577a --- /dev/null +++ b/docs/devel/ebpf_rss.rst @@ -0,0 +1,125 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +eBPF RSS virtio-net support +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D + +RSS(Receive Side Scaling) is used to distribute network packets to guest v= irtqueues +by calculating packet hash. Usually every queue is processed then by a spe= cific guest CPU core. + +For now there are 2 RSS implementations in qemu: +- 'in-qemu' RSS (functions if qemu receives network packets, i.e. vhost=3D= off) +- eBPF RSS (can function with also with vhost=3Don) + +eBPF support (CONFIG_EBPF) is enabled by 'configure' script. +To enable eBPF RSS support use './configure --enable-bpf'. + +If steering BPF is not set for kernel's TUN module, the TUN uses automatic= selection +of rx virtqueue based on lookup table built according to calculated symmet= ric hash +of transmitted packets. +If steering BPF is set for TUN the BPF code calculates the hash of packet = header and +returns the virtqueue number to place the packet to. + +Simplified decision formula: + +.. code:: C + + queue_index =3D indirection_table[hash()%] + + +Not for all packets, the hash can/should be calculated. + +Note: currently, eBPF RSS does not support hash reporting. + +eBPF RSS turned on by different combinations of vhost-net, vitrio-net and = tap configurations: + +- eBPF is used: + + tap,vhost=3Doff & virtio-net-pci,rss=3Don,hash=3Doff + +- eBPF is used: + + tap,vhost=3Don & virtio-net-pci,rss=3Don,hash=3Doff + +- 'in-qemu' RSS is used: + + tap,vhost=3Doff & virtio-net-pci,rss=3Don,hash=3Don + +- eBPF is used, hash population feature is not reported to the guest: + + tap,vhost=3Don & virtio-net-pci,rss=3Don,hash=3Don + +If CONFIG_EBPF is not set then only 'in-qemu' RSS is supported. +Also 'in-qemu' RSS, as a fallback, is used if the eBPF program failed to l= oad or set to TUN. + +RSS eBPF program +---------------- + +RSS program located in ebpf/rss.bpf.skeleton.h generated by bpftool. +So the program is part of the qemu binary. +Initially, the eBPF program was compiled by clang and source code located = at tools/ebpf/rss.bpf.c. +Prerequisites to recompile the eBPF program (regenerate ebpf/rss.bpf.skele= ton.h): + + llvm, clang, kernel source tree, bpftool + Adjust Makefile.ebpf to reflect the location of the kernel source = tree + + $ cd tools/ebpf + $ make -f Makefile.ebpf + +Current eBPF RSS implementation uses 'bounded loops' with 'backward jump i= nstructions' which present in the last kernels. +Overall eBPF RSS works on kernels 5.8+. + +eBPF RSS implementation +----------------------- + +eBPF RSS loading functionality located in ebpf/ebpf_rss.c and ebpf/ebpf_rs= s.h. + +The `struct EBPFRSSContext` structure that holds 4 file descriptors: + +- ctx - pointer of the libbpf context. +- program_fd - file descriptor of the eBPF RSS program. +- map_configuration - file descriptor of the 'configuration' map. This map= contains one element of 'struct EBPFRSSConfig'. This configuration determi= nes eBPF program behavior. +- map_toeplitz_key - file descriptor of the 'Toeplitz key' map. One elemen= t of the 40byte key prepared for the hashing algorithm. +- map_indirections_table - 128 elements of queue indexes. + +`struct EBPFRSSConfig` fields: + +- redirect - "boolean" value, should the hash be calculated, on false - `= default_queue` would be used as the final decision. +- populate_hash - for now, not used. eBPF RSS doesn't support hash reporti= ng. +- hash_types - binary mask of different hash types. See `VIRTIO_NET_RSS_HA= SH_TYPE_*` defines. If for packet hash should not be calculated - `default_= queue` would be used. +- indirections_len - length of the indirections table, maximum 128. +- default_queue - the queue index that used for packet that shouldn't be h= ashed. For some packets, the hash can't be calculated(g.e ARP). + +Functions: + +- `ebpf_rss_init()` - sets ctx to NULL, which indicates that EBPFRSSContex= t is not loaded. +- `ebpf_rss_load()` - creates 3 maps and loads eBPF program from the rss.b= pf.skeleton.h. Returns 'true' on success. After that, program_fd can be use= d to set steering for TAP. +- `ebpf_rss_set_all()` - sets values for eBPF maps. `indirections_table` l= ength is in EBPFRSSConfig. `toeplitz_key` is VIRTIO_NET_RSS_MAX_KEY_SIZE ak= a 40 bytes array. +- `ebpf_rss_unload()` - close all file descriptors and set ctx to NULL. + +Simplified eBPF RSS workflow: + +.. code:: C + + struct EBPFRSSConfig config; + config.redirect =3D 1; + config.hash_types =3D VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | VIRTIO_NET_RSS_= HASH_TYPE_TCPv4; + config.indirections_len =3D VIRTIO_NET_RSS_MAX_TABLE_LEN; + config.default_queue =3D 0; + + uint16_t table[VIRTIO_NET_RSS_MAX_TABLE_LEN] =3D {...}; + uint8_t key[VIRTIO_NET_RSS_MAX_KEY_SIZE] =3D {...}; + + struct EBPFRSSContext ctx; + ebpf_rss_init(&ctx); + ebpf_rss_load(&ctx); + ebpf_rss_set_all(&ctx, &config, table, key); + if (net_client->info->set_steering_ebpf !=3D NULL) { + net_client->info->set_steering_ebpf(net_client, ctx->program_fd); + } + ... + ebpf_unload(&ctx); + + +NetClientState SetSteeringEBPF() +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For now, `set_steering_ebpf()` method supported by Linux TAP NetClientStat= e. The method requires an eBPF program file descriptor as an argument. diff --git a/docs/devel/index.rst b/docs/devel/index.rst index 6cf7e2d233..33179ba309 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -42,3 +42,4 @@ Contents: qom block-coroutine-wrapper multi-process + ebpf_rss --=20 2.31.1