From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561388122; cv=none; d=zoho.com; s=zohoarc; b=O0JV3eA8PbHM2XPB30iT5qgPIj0Z1dyE9wShQrm4/kxQCVy32v4rWBO8XN0ss4hSB9oRE0JIPF2p5xiUpxIrskcYxwXj2S+mdb2z0dBZ2aEeyw4RVTzjqMOwSOZWxE/QnsM9nvKeXWIRr5btxz40fcRKjyfSiMoMZRx0qU90uC4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561388122; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=qEyEN0NKr3khP3JxCHV/2z3wozzAH9ksfNCVIhkTzfA=; b=Tec8PDe7BGpszlExLYQO33OiVAAKTZsOKY+qnG4748hOPPE0zuBmz+flaMD9WF28hBEijrtNNl4mpMCNtDLImf8IrgbRRh47dYsaJEkE6wBKQ15mtFLNeI0nkOs08upT6aIWqHRAkBZpmR1gjlF8d5ugkdJOq5PI6YK8fsB0LfA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561388122465948.9453610225091; Mon, 24 Jun 2019 07:55:22 -0700 (PDT) Received: from localhost ([::1]:51980 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQNL-0000JV-SQ for importer@patchew.org; Mon, 24 Jun 2019 10:55:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60140) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGK-0002ob-Sh for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGJ-0006Gj-NU for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:27719) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGG-0005ff-Ca; Mon, 24 Jun 2019 10:48:00 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 579153082211; Mon, 24 Jun 2019 14:47:46 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E6ACF5D9C5; Mon, 24 Jun 2019 14:47:45 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:33 +0200 Message-Id: <20190624144740.5338-2-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Mon, 24 Jun 2019 14:47:46 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 1/8] nvme: do not advertise support for unsupported arbitration mechanism X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Klaus Birkelund Jensen The device mistakenly reports that the Weighted Round Robin with Urgent Priority Class arbitration mechanism is supported. It is not. Signed-off-by: Klaus Birkelund Jensen Message-id: 20190606092530.14206-1-klaus@birkelund.eu Acked-by: Maxim Levitsky Signed-off-by: Max Reitz --- hw/block/nvme.c | 1 - 1 file changed, 1 deletion(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 107a719b95..36d6a8bb3a 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -1384,7 +1384,6 @@ static void nvme_realize(PCIDevice *pci_dev, Error **= errp) n->bar.cap =3D 0; NVME_CAP_SET_MQES(n->bar.cap, 0x7ff); NVME_CAP_SET_CQR(n->bar.cap, 1); - NVME_CAP_SET_AMS(n->bar.cap, 1); NVME_CAP_SET_TO(n->bar.cap, 0xf); NVME_CAP_SET_CSS(n->bar.cap, 1); NVME_CAP_SET_MPSMAX(n->bar.cap, 4); --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561387880; cv=none; d=zoho.com; s=zohoarc; b=K4Ki61qdFIYB5gBONpB3A+aVI7j9PO+nqXq69xDO7AkMtM6eTr+jugRYHuU5U414IxZo9hhzBOvZZQ/IZH2btDX0a8Pbbtds3JxG1+djpTFL9umdPPHdMIuhNj9hdwkwiG3+tDzE6yg8ewoiQ9BrI8YlyQmwcZZ0sylaJwcAajA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561387880; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=DZJqEDbmVQ0r5aTE9GI75gYRqWtXaRSqY0nK4rYbaW0=; b=Rt0iScly/a/F82orD9avnN71D3HHMwfyvb5kBfl1RDoLUF7DtqCv/IGA1fSDj3utu6PTogcIxLkIrqqEe700L78Vw7gkQYVttTl3SXKSFObYOgdf47ljhWhTwVZgDSgtlJvzvd9i2Y1hLfPU17hbdyUaP+dxZAlo6FE0kFHlXxY= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561387880182811.4947749736207; Mon, 24 Jun 2019 07:51:20 -0700 (PDT) Received: from localhost ([::1]:51948 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQJL-0004yQ-0V for importer@patchew.org; Mon, 24 Jun 2019 10:51:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60044) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGI-0002je-9X for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGH-0006BE-Al for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:02 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44474) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGC-0005mX-0t; Mon, 24 Jun 2019 10:47:57 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AEAB730832DA; Mon, 24 Jun 2019 14:47:48 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 474F46085B; Mon, 24 Jun 2019 14:47:48 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:34 +0200 Message-Id: <20190624144740.5338-3-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 24 Jun 2019 14:47:48 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 2/8] blockdev: enable non-root nodes for transaction drive-backup source X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy We forget to enable it for transaction .prepare, while it is already enabled in do_drive_backup since commit a2d665c1bc362 "blockdev: loosen restrictions on drive-backup source node" Signed-off-by: Vladimir Sementsov-Ogievskiy Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com Reviewed-by: John Snow Signed-off-by: Max Reitz --- blockdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index 5d6a13dea9..4d141e9a1f 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1774,7 +1774,7 @@ static void drive_backup_prepare(BlkActionState *comm= on, Error **errp) assert(common->action->type =3D=3D TRANSACTION_ACTION_KIND_DRIVE_BACKU= P); backup =3D common->action->u.drive_backup.data; =20 - bs =3D qmp_get_root_bs(backup->device, errp); + bs =3D bdrv_lookup_bs(backup->device, backup->device, errp); if (!bs) { return; } --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561388320; cv=none; d=zoho.com; s=zohoarc; b=bULTlBlSEUtQxKiNQMihjZ2jt+ijV16PP4BfCJ+T4Hl6xoKipqL4QgIW4fXTAFfvDGy5Mfi0Vf/KsfowNGQh86Gn+kSeyTWODbrmGtr4Xq4NDZ+ZifYL5DfmqrxV2/I3TgtvyDHhqiVNAYTN2PtcJmGN7tJ5o6Z+ghQhNbC32WQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561388320; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=wKyVu2r+HVjO+DxqHw3YZi4kQcrckFxyjlvdF0wdlTA=; b=RwNaypVArthEc9TvtBjfcf0vPMF6MXdLnZut4MZ5el8i+SDWH5wCqBJYnlS0a6hXszp/smu43uzvkM8XiOwpH8zKnR8kwTFlr69mpilTsTM6WM1H++w246uqBKXxqInq4dNLdeZHPFczSc90dKZvHKRQJWbh8iaXC5LV65b8vLI= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561388320696526.86848879155; Mon, 24 Jun 2019 07:58:40 -0700 (PDT) Received: from localhost ([::1]:52028 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQQY-0003xf-O8 for importer@patchew.org; Mon, 24 Jun 2019 10:58:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60129) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGK-0002o6-KD for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGI-0006FL-Vk for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35281) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGG-0005pF-3D; Mon, 24 Jun 2019 10:48:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0E9D2C04B93D; Mon, 24 Jun 2019 14:47:51 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 998AD6090E; Mon, 24 Jun 2019 14:47:50 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:35 +0200 Message-Id: <20190624144740.5338-4-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 24 Jun 2019 14:47:51 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 3/8] iotest 134: test cluster-misaligned encrypted write X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Anton Nefedov COW (even empty/zero) areas require encryption too Signed-off-by: Anton Nefedov Reviewed-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Alberto Garcia Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com Signed-off-by: Max Reitz --- tests/qemu-iotests/134 | 9 +++++++++ tests/qemu-iotests/134.out | 10 ++++++++++ 2 files changed, 19 insertions(+) diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134 index 141a2eaa7e..5f0fb86211 100755 --- a/tests/qemu-iotests/134 +++ b/tests/qemu-iotests/134 @@ -56,6 +56,15 @@ echo echo "=3D=3D reading whole image =3D=3D" $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filte= r_qemu_io | _filter_testdir =20 +echo +echo "=3D=3D rewriting cluster part =3D=3D" +$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC = | _filter_qemu_io | _filter_testdir + +echo +echo "=3D=3D verify pattern =3D=3D" +$QEMU_IO --object $SECRET -c "read -P 0 0 512" --image-opts $IMGSPEC | _f= ilter_qemu_io | _filter_testdir +$QEMU_IO --object $SECRET -c "read -P 0xb 512 512" --image-opts $IMGSPEC = | _filter_qemu_io | _filter_testdir + echo echo "=3D=3D rewriting whole image =3D=3D" $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC = | _filter_qemu_io | _filter_testdir diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out index 972be49d91..09d46f6b17 100644 --- a/tests/qemu-iotests/134.out +++ b/tests/qemu-iotests/134.out @@ -5,6 +5,16 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D134217= 728 encryption=3Don encrypt. read 134217728/134217728 bytes at offset 0 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) =20 +=3D=3D rewriting cluster part =3D=3D +wrote 512/512 bytes at offset 512 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=3D=3D verify pattern =3D=3D +read 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 512/512 bytes at offset 512 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + =3D=3D rewriting whole image =3D=3D wrote 134217728/134217728 bytes at offset 0 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561387879; cv=none; d=zoho.com; s=zohoarc; b=koPjgCigCZI5+fXNE0p5UvXA1Y4XOI6M/KpEFijKad2E+3qfxzD9PwKtZ/PDWPbkl0I1/b6FnUP+5VDfxHC2Ys05XyrS83tV5So9ZAC4HUF4IHRBRwkjk6s5bScgUWFPwRMgR06ABcrL14mCoVSBMvKw7J5Sb+OkjTtazX1gLik= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561387879; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=B3nThCr0Bk/JulHvu6FryeMKxkgMmZ4utL1Cb4kvYIc=; b=hd5zC6r1AL0FJU5R6VannNLfTwwIAMhhozbJ2+AZqPVZZHbh+Yx8N2VmnKAEsgRve6ODLh6Ku+lOWyIkrANOzZTyQXda8HQ4/tTVcVZshL0/KByhII4r2SEbNQAyrbBtnbdYt2YZ2noFkELNYfb/h5BXOe8Ef8/PlpwEZvJiP24= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561387879269962.796906857862; Mon, 24 Jun 2019 07:51:19 -0700 (PDT) Received: from localhost ([::1]:51950 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQJN-00051i-2W for importer@patchew.org; Mon, 24 Jun 2019 10:51:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60112) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGK-0002n1-1y for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGI-0006EP-ME for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46902) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGE-0005tJ-55; Mon, 24 Jun 2019 10:48:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 61F93307D868; Mon, 24 Jun 2019 14:47:53 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EF121608D0; Mon, 24 Jun 2019 14:47:52 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:36 +0200 Message-Id: <20190624144740.5338-5-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.48]); Mon, 24 Jun 2019 14:47:53 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 4/8] vmdk: Fix comment regarding max l1_size coverage X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent") extended the l1_size check from VMDK4 to VMDK3 but did not update the default coverage in the moved comment. The previous vmdk4 calculation: (512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) =3D 16PB The added vmdk3 calculation: (512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) =3D 1PB Adding the calculation of vmdk3 to the comment. In any case, VMware does not offer virtual disks more than 2TB for vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is not implemented yet in qemu. Reviewed-by: Karl Heubaum Reviewed-by: Eyal Moscovici Reviewed-by: Liran Alon Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com Reviewed-by: yuchenlin Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/vmdk.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/block/vmdk.c b/block/vmdk.c index 51067c774f..0f2e453bf5 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -426,10 +426,15 @@ static int vmdk_add_extent(BlockDriverState *bs, return -EFBIG; } if (l1_size > 512 * 1024 * 1024) { - /* Although with big capacity and small l1_entry_sectors, we can g= et a + /* + * Although with big capacity and small l1_entry_sectors, we can g= et a * big l1_size, we don't want unbounded value to allocate the tabl= e. - * Limit it to 512M, which is 16PB for default cluster and L2 table - * size */ + * Limit it to 512M, which is: + * 16PB - for default "Hosted Sparse Extent" (VMDK4) + * cluster size: 64KB, L2 table size: 512 entries + * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSpa= rse) + * cluster size: 512B, L2 table size: 4096 entries + */ error_setg(errp, "L1 size too big"); return -EFBIG; } --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561387880; cv=none; d=zoho.com; s=zohoarc; b=OLF2pTAswy+g+D9SQpR73T10dgAH0J0dGfWpu2yMCBD4W7VpBTvxhAHac/bSWVoqOh423roq9U6m7JwcRNLZKA5g/WwdygeMRJZhGN14ou5wzDWCWoliDo6jeT4jCpu/hHjsHp2IGkhQZOF1qW12eMnb2D4YHleLDIh8m8BD8jo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561387880; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=jDfjdAw564I9LWxQDI/EOVWLRctu48hygnoykLIHht8=; b=Kxq4YUjbCjvVUSi/+3VAsa4fyX+Zuoz7MX8vpUZ5fUeJRc2hJdruxlqFOpIvSmo69gUBKVnjghWVfbukZPByUrb1Y4S+EZqF+Hzw1dOE9WkTrUQzenJURCF8VoWw5Z3SALynXBY2b7DEaBiav0EGjFoCZ/l9nzo+ob5EMr64nR4= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561387880556113.50004023161569; Mon, 24 Jun 2019 07:51:20 -0700 (PDT) Received: from localhost ([::1]:51952 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQJR-000581-WC for importer@patchew.org; Mon, 24 Jun 2019 10:51:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60124) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGK-0002nl-C5 for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGJ-0006Fa-3q for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49490) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGG-0005xW-Ab; Mon, 24 Jun 2019 10:48:00 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B19CF3162908; Mon, 24 Jun 2019 14:47:55 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 49E5D19C6A; Mon, 24 Jun 2019 14:47:55 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:37 +0200 Message-Id: <20190624144740.5338-6-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Mon, 24 Jun 2019 14:47:55 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 5/8] vmdk: Reduce the max bound for L1 table size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman 512M of L1 entries is a very loose bound, only 32M are required to store the maximal supported VMDK file size of 2TB. Fixed qemu-iotest 59# - now failure occures before on impossible L1 table size. Reviewed-by: Karl Heubaum Reviewed-by: Eyal Moscovici Reviewed-by: Liran Alon Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/vmdk.c | 13 +++++++------ tests/qemu-iotests/059.out | 2 +- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/block/vmdk.c b/block/vmdk.c index 0f2e453bf5..931eb2759c 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -425,15 +425,16 @@ static int vmdk_add_extent(BlockDriverState *bs, error_setg(errp, "Invalid granularity, image may be corrupt"); return -EFBIG; } - if (l1_size > 512 * 1024 * 1024) { + if (l1_size > 32 * 1024 * 1024) { /* * Although with big capacity and small l1_entry_sectors, we can g= et a * big l1_size, we don't want unbounded value to allocate the tabl= e. - * Limit it to 512M, which is: - * 16PB - for default "Hosted Sparse Extent" (VMDK4) - * cluster size: 64KB, L2 table size: 512 entries - * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSpa= rse) - * cluster size: 512B, L2 table size: 4096 entries + * Limit it to 32M, which is enough to store: + * 8TB - for both VMDK3 & VMDK4 with + * minimal cluster size: 512B + * minimal L2 table size: 512 entries + * 8 TB is still more than the maximal value supported = for + * VMDK3 & VMDK4 which is 2TB. */ error_setg(errp, "L1 size too big"); return -EFBIG; diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out index f51394ae8e..4fab42a28c 100644 --- a/tests/qemu-iotests/059.out +++ b/tests/qemu-iotests/059.out @@ -2358,5 +2358,5 @@ Offset Length Mapped to File 0x140000000 0x10000 0x50000 TEST_DIR/t-s003.vmdk =20 =3D=3D=3D Testing afl image with a very large capacity =3D=3D=3D -qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large +qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big *** done --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561388244; cv=none; d=zoho.com; s=zohoarc; b=RcxXdI/PCTNFTu0HcvS1+SPeizvd7dvWZ98zgxKfBFP5AbWQX5Rd6wW7pTmEk+q+GZ1OfL4zoX6nr+ju/xqHzPwvMFqc8i1vegvQRMNn0gZKrw08MUS4FC8TQNCUJiO3CbXXe/nm2RoGqndoTvKSKrQ13oVQmvTlh8CO6f531HE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561388244; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=zqfZQKAhCaMuMA2yTP8s35Qnm4BSwkcZ/h/BF2zF3XY=; b=ST8fRunlIY9GHKXORMLwUYdJbbgxRB6rTVl0UvBXQIquasA7mdAnFEvO/9xkkEIB3/L5vM2sdYDW6JO3r7pyGu55qiFPV5Fwk5MPSXbmXnHCkbaLD5Fo3tu/V65xq6W6/slJr02ulVqgBqiuWI2Ri3C3pAyyOc1yPogKjHmRtao= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561388244495387.8447472519264; Mon, 24 Jun 2019 07:57:24 -0700 (PDT) Received: from localhost ([::1]:51990 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQPL-0001uT-Du for importer@patchew.org; Mon, 24 Jun 2019 10:57:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60226) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGa-000323-81 for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGN-0006QW-SO for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:15 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60838) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGG-00064B-Cr; Mon, 24 Jun 2019 10:48:00 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 944BF301EA93; Mon, 24 Jun 2019 14:47:58 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 99BBF600D1; Mon, 24 Jun 2019 14:47:57 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:38 +0200 Message-Id: <20190624144740.5338-7-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Mon, 24 Jun 2019 14:47:58 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 6/8] vmdk: Add read-only support for seSparse snapshots X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in QEMU). This format was lacking in the following: * Grain directory (L1) and grain table (L2) entries were 32-bit, allowing access to only 2TB (slightly less) of data. * The grain size (default) was 512 bytes - leading to data fragmentation and many grain tables. * For space reclamation purposes, it was necessary to find all the grains which are not pointed to by any grain table - so a reverse mapping of "offset of grain in vmdk" to "grain table" must be constructed - which takes large amounts of CPU/RAM. The format specification can be found in VMware's documentation: https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf In ESXi 6.5, to support snapshot files larger than 2TB, a new format was introduced: SESparse (Space Efficient). This format fixes the above issues: * All entries are now 64-bit. * The grain size (default) is 4KB. * Grain directory and grain tables are now located at the beginning of the file. + seSparse format reserves space for all grain tables. + Grain tables can be addressed using an index. + Grains are located in the end of the file and can also be addressed with an index. - seSparse vmdks of large disks (64TB) have huge preallocated headers - mainly due to L2 tables, even for empty snapshots. * The header contains a reverse mapping ("backmap") of "offset of grain in vmdk" to "grain table" and a bitmap ("free bitmap") which specifies for each grain - whether it is allocated or not. Using these data structures we can implement space reclamation efficiently. * Due to the fact that the header now maintains two mappings: * The regular one (grain directory & grain tables) * A reverse one (backmap and free bitmap) These data structures can lose consistency upon crash and result in a corrupted VMDK. Therefore, a journal is also added to the VMDK and is replayed when the VMware reopens the file after a crash. Since ESXi 6.7 - SESparse is the only snapshot format available. Unfortunately, VMware does not provide documentation regarding the new seSparse format. This commit is based on black-box research of the seSparse format. Various in-guest block operations and their effect on the snapshot file were tested. The only VMware provided source of information (regarding the underlying implementation) was a log file on the ESXi: /var/log/hostd.log Whenever an seSparse snapshot is created - the log is being populated with seSparse records. Relevant log records are of the form: [...] Const Header: [...] constMagic =3D 0xcafebabe [...] version =3D 2.1 [...] capacity =3D 204800 [...] grainSize =3D 8 [...] grainTableSize =3D 64 [...] flags =3D 0 [...] Extents: [...] Header : <1 : 1> [...] JournalHdr : <2 : 2> [...] Journal : <2048 : 2048> [...] GrainDirectory : <4096 : 2048> [...] GrainTables : <6144 : 2048> [...] FreeBitmap : <8192 : 2048> [...] BackMap : <10240 : 2048> [...] Grain : <12288 : 204800> [...] Volatile Header: [...] volatileMagic =3D 0xcafecafe [...] FreeGTNumber =3D 0 [...] nextTxnSeqNumber =3D 0 [...] replayJournal =3D 0 The sizes that are seen in the log file are in sectors. Extents are of the following format: This commit is a strict implementation which enforces: * magics * version number 2.1 * grain size of 8 sectors (4KB) * grain table size of 64 sectors * zero flags * extent locations Additionally, this commit proivdes only a subset of the functionality offered by seSparse's format: * Read-only * No journal replay * No space reclamation * No unmap support Hence, journal header, journal, free bitmap and backmap extents are unused, only the "classic" (L1 -> L2 -> data) grain access is implemented. However there are several differences in the grain access itself. Grain directory (L1): * Grain directory entries are indexes (not offsets) to grain tables. * Valid grain directory entries have their highest nibble set to 0x1. * Since grain tables are always located in the beginning of the file - the index can fit into 32 bits - so we can use its low part if it's valid. Grain table (L2): * Grain table entries are indexes (not offsets) to grains. * If the highest nibble of the entry is: 0x0: The grain in not allocated. The rest of the bytes are 0. 0x1: The grain is unmapped - guest sees a zero grain. The rest of the bits point to the previously mapped grain, see 0x3 case. 0x2: The grain is zero. 0x3: The grain is allocated - to get the index calculate: ((entry & 0x0fff000000000000) >> 48) | ((entry & 0x0000ffffffffffff) << 12) * The difference between 0x1 and 0x2 is that 0x1 is an unallocated grain which results from the guest using sg_unmap to unmap the grain - but the grain itself still exists in the grain extent - a space reclamation procedure should delete it. Unmapping a zero grain has no effect (0x2 will not change to 0x1) but unmapping an unallocated grain will (0x0 to 0x1) - naturally. In order to implement seSparse some fields had to be changed to support both 32-bit and 64-bit entry sizes. Reviewed-by: Karl Heubaum Reviewed-by: Eyal Moscovici Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com Signed-off-by: Max Reitz --- block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 342 insertions(+), 16 deletions(-) diff --git a/block/vmdk.c b/block/vmdk.c index 931eb2759c..bd36ece125 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -91,6 +91,44 @@ typedef struct { uint16_t compressAlgorithm; } QEMU_PACKED VMDK4Header; =20 +typedef struct VMDKSESparseConstHeader { + uint64_t magic; + uint64_t version; + uint64_t capacity; + uint64_t grain_size; + uint64_t grain_table_size; + uint64_t flags; + uint64_t reserved1; + uint64_t reserved2; + uint64_t reserved3; + uint64_t reserved4; + uint64_t volatile_header_offset; + uint64_t volatile_header_size; + uint64_t journal_header_offset; + uint64_t journal_header_size; + uint64_t journal_offset; + uint64_t journal_size; + uint64_t grain_dir_offset; + uint64_t grain_dir_size; + uint64_t grain_tables_offset; + uint64_t grain_tables_size; + uint64_t free_bitmap_offset; + uint64_t free_bitmap_size; + uint64_t backmap_offset; + uint64_t backmap_size; + uint64_t grains_offset; + uint64_t grains_size; + uint8_t pad[304]; +} QEMU_PACKED VMDKSESparseConstHeader; + +typedef struct VMDKSESparseVolatileHeader { + uint64_t magic; + uint64_t free_gt_number; + uint64_t next_txn_seq_number; + uint64_t replay_journal; + uint8_t pad[480]; +} QEMU_PACKED VMDKSESparseVolatileHeader; + #define L2_CACHE_SIZE 16 =20 typedef struct VmdkExtent { @@ -99,19 +137,23 @@ typedef struct VmdkExtent { bool compressed; bool has_marker; bool has_zero_grain; + bool sesparse; + uint64_t sesparse_l2_tables_offset; + uint64_t sesparse_clusters_offset; + int32_t entry_size; int version; int64_t sectors; int64_t end_sector; int64_t flat_start_offset; int64_t l1_table_offset; int64_t l1_backup_table_offset; - uint32_t *l1_table; + void *l1_table; uint32_t *l1_backup_table; unsigned int l1_size; uint32_t l1_entry_sectors; =20 unsigned int l2_size; - uint32_t *l2_cache; + void *l2_cache; uint32_t l2_cache_offsets[L2_CACHE_SIZE]; uint32_t l2_cache_counts[L2_CACHE_SIZE]; =20 @@ -435,6 +477,11 @@ static int vmdk_add_extent(BlockDriverState *bs, * minimal L2 table size: 512 entries * 8 TB is still more than the maximal value supported = for * VMDK3 & VMDK4 which is 2TB. + * 64TB - for "ESXi seSparse Extent" + * minimal cluster size: 512B (default is 4KB) + * L2 table size: 4096 entries (const). + * 64TB is more than the maximal value supported for + * seSparse VMDKs (which is slightly less than 64TB) */ error_setg(errp, "L1 size too big"); return -EFBIG; @@ -460,6 +507,7 @@ static int vmdk_add_extent(BlockDriverState *bs, extent->l2_size =3D l2_size; extent->cluster_sectors =3D flat ? sectors : cluster_sectors; extent->next_cluster_sector =3D ROUND_UP(nb_sectors, cluster_sectors); + extent->entry_size =3D sizeof(uint32_t); =20 if (s->num_extents > 1) { extent->end_sector =3D (*(extent - 1)).end_sector + extent->sector= s; @@ -481,7 +529,7 @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkE= xtent *extent, int i; =20 /* read the L1 table */ - l1_size =3D extent->l1_size * sizeof(uint32_t); + l1_size =3D extent->l1_size * extent->entry_size; extent->l1_table =3D g_try_malloc(l1_size); if (l1_size && extent->l1_table =3D=3D NULL) { return -ENOMEM; @@ -499,10 +547,16 @@ static int vmdk_init_tables(BlockDriverState *bs, Vmd= kExtent *extent, goto fail_l1; } for (i =3D 0; i < extent->l1_size; i++) { - le32_to_cpus(&extent->l1_table[i]); + if (extent->entry_size =3D=3D sizeof(uint64_t)) { + le64_to_cpus((uint64_t *)extent->l1_table + i); + } else { + assert(extent->entry_size =3D=3D sizeof(uint32_t)); + le32_to_cpus((uint32_t *)extent->l1_table + i); + } } =20 if (extent->l1_backup_table_offset) { + assert(!extent->sesparse); extent->l1_backup_table =3D g_try_malloc(l1_size); if (l1_size && extent->l1_backup_table =3D=3D NULL) { ret =3D -ENOMEM; @@ -525,7 +579,7 @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkE= xtent *extent, } =20 extent->l2_cache =3D - g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE); + g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE); return 0; fail_l1b: g_free(extent->l1_backup_table); @@ -571,6 +625,205 @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs, return ret; } =20 +#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe) +#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe) + +/* Strict checks - format not officially documented */ +static int check_se_sparse_const_header(VMDKSESparseConstHeader *header, + Error **errp) +{ + header->magic =3D le64_to_cpu(header->magic); + header->version =3D le64_to_cpu(header->version); + header->grain_size =3D le64_to_cpu(header->grain_size); + header->grain_table_size =3D le64_to_cpu(header->grain_table_size); + header->flags =3D le64_to_cpu(header->flags); + header->reserved1 =3D le64_to_cpu(header->reserved1); + header->reserved2 =3D le64_to_cpu(header->reserved2); + header->reserved3 =3D le64_to_cpu(header->reserved3); + header->reserved4 =3D le64_to_cpu(header->reserved4); + + header->volatile_header_offset =3D + le64_to_cpu(header->volatile_header_offset); + header->volatile_header_size =3D le64_to_cpu(header->volatile_header_s= ize); + + header->journal_header_offset =3D le64_to_cpu(header->journal_header_o= ffset); + header->journal_header_size =3D le64_to_cpu(header->journal_header_siz= e); + + header->journal_offset =3D le64_to_cpu(header->journal_offset); + header->journal_size =3D le64_to_cpu(header->journal_size); + + header->grain_dir_offset =3D le64_to_cpu(header->grain_dir_offset); + header->grain_dir_size =3D le64_to_cpu(header->grain_dir_size); + + header->grain_tables_offset =3D le64_to_cpu(header->grain_tables_offse= t); + header->grain_tables_size =3D le64_to_cpu(header->grain_tables_size); + + header->free_bitmap_offset =3D le64_to_cpu(header->free_bitmap_offset); + header->free_bitmap_size =3D le64_to_cpu(header->free_bitmap_size); + + header->backmap_offset =3D le64_to_cpu(header->backmap_offset); + header->backmap_size =3D le64_to_cpu(header->backmap_size); + + header->grains_offset =3D le64_to_cpu(header->grains_offset); + header->grains_size =3D le64_to_cpu(header->grains_size); + + if (header->magic !=3D SESPARSE_CONST_HEADER_MAGIC) { + error_setg(errp, "Bad const header magic: 0x%016" PRIx64, + header->magic); + return -EINVAL; + } + + if (header->version !=3D 0x0000000200000001) { + error_setg(errp, "Unsupported version: 0x%016" PRIx64, + header->version); + return -ENOTSUP; + } + + if (header->grain_size !=3D 8) { + error_setg(errp, "Unsupported grain size: %" PRIu64, + header->grain_size); + return -ENOTSUP; + } + + if (header->grain_table_size !=3D 64) { + error_setg(errp, "Unsupported grain table size: %" PRIu64, + header->grain_table_size); + return -ENOTSUP; + } + + if (header->flags !=3D 0) { + error_setg(errp, "Unsupported flags: 0x%016" PRIx64, + header->flags); + return -ENOTSUP; + } + + if (header->reserved1 !=3D 0 || header->reserved2 !=3D 0 || + header->reserved3 !=3D 0 || header->reserved4 !=3D 0) { + error_setg(errp, "Unsupported reserved bits:" + " 0x%016" PRIx64 " 0x%016" PRIx64 + " 0x%016" PRIx64 " 0x%016" PRIx64, + header->reserved1, header->reserved2, + header->reserved3, header->reserved4); + return -ENOTSUP; + } + + /* check that padding is 0 */ + if (!buffer_is_zero(header->pad, sizeof(header->pad))) { + error_setg(errp, "Unsupported non-zero const header padding"); + return -ENOTSUP; + } + + return 0; +} + +static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *hea= der, + Error **errp) +{ + header->magic =3D le64_to_cpu(header->magic); + header->free_gt_number =3D le64_to_cpu(header->free_gt_number); + header->next_txn_seq_number =3D le64_to_cpu(header->next_txn_seq_numbe= r); + header->replay_journal =3D le64_to_cpu(header->replay_journal); + + if (header->magic !=3D SESPARSE_VOLATILE_HEADER_MAGIC) { + error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64, + header->magic); + return -EINVAL; + } + + if (header->replay_journal) { + error_setg(errp, "Image is dirty, Replaying journal not supported"= ); + return -ENOTSUP; + } + + /* check that padding is 0 */ + if (!buffer_is_zero(header->pad, sizeof(header->pad))) { + error_setg(errp, "Unsupported non-zero volatile header padding"); + return -ENOTSUP; + } + + return 0; +} + +static int vmdk_open_se_sparse(BlockDriverState *bs, + BdrvChild *file, + int flags, Error **errp) +{ + int ret; + VMDKSESparseConstHeader const_header; + VMDKSESparseVolatileHeader volatile_header; + VmdkExtent *extent; + + ret =3D bdrv_apply_auto_read_only(bs, + "No write support for seSparse images available", errp); + if (ret < 0) { + return ret; + } + + assert(sizeof(const_header) =3D=3D SECTOR_SIZE); + + ret =3D bdrv_pread(file, 0, &const_header, sizeof(const_header)); + if (ret < 0) { + bdrv_refresh_filename(file->bs); + error_setg_errno(errp, -ret, + "Could not read const header from file '%s'", + file->bs->filename); + return ret; + } + + /* check const header */ + ret =3D check_se_sparse_const_header(&const_header, errp); + if (ret < 0) { + return ret; + } + + assert(sizeof(volatile_header) =3D=3D SECTOR_SIZE); + + ret =3D bdrv_pread(file, + const_header.volatile_header_offset * SECTOR_SIZE, + &volatile_header, sizeof(volatile_header)); + if (ret < 0) { + bdrv_refresh_filename(file->bs); + error_setg_errno(errp, -ret, + "Could not read volatile header from file '%s'", + file->bs->filename); + return ret; + } + + /* check volatile header */ + ret =3D check_se_sparse_volatile_header(&volatile_header, errp); + if (ret < 0) { + return ret; + } + + ret =3D vmdk_add_extent(bs, file, false, + const_header.capacity, + const_header.grain_dir_offset * SECTOR_SIZE, + 0, + const_header.grain_dir_size * + SECTOR_SIZE / sizeof(uint64_t), + const_header.grain_table_size * + SECTOR_SIZE / sizeof(uint64_t), + const_header.grain_size, + &extent, + errp); + if (ret < 0) { + return ret; + } + + extent->sesparse =3D true; + extent->sesparse_l2_tables_offset =3D const_header.grain_tables_offset; + extent->sesparse_clusters_offset =3D const_header.grains_offset; + extent->entry_size =3D sizeof(uint64_t); + + ret =3D vmdk_init_tables(bs, extent, errp); + if (ret) { + /* free extent allocated by vmdk_add_extent */ + vmdk_free_last_extent(bs); + } + + return ret; +} + static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf, QDict *options, Error **errp); =20 @@ -848,6 +1101,7 @@ static int vmdk_parse_extents(const char *desc, BlockD= riverState *bs, * RW [size in sectors] SPARSE "file-name.vmdk" * RW [size in sectors] VMFS "file-name.vmdk" * RW [size in sectors] VMFSSPARSE "file-name.vmdk" + * RW [size in sectors] SESPARSE "file-name.vmdk" */ flat_offset =3D -1; matches =3D sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" = SCNd64, @@ -870,7 +1124,8 @@ static int vmdk_parse_extents(const char *desc, BlockD= riverState *bs, =20 if (sectors <=3D 0 || (strcmp(type, "FLAT") && strcmp(type, "SPARSE") && - strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) || + strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") && + strcmp(type, "SESPARSE")) || (strcmp(access, "RW"))) { continue; } @@ -923,6 +1178,13 @@ static int vmdk_parse_extents(const char *desc, Block= DriverState *bs, return ret; } extent =3D &s->extents[s->num_extents - 1]; + } else if (!strcmp(type, "SESPARSE")) { + ret =3D vmdk_open_se_sparse(bs, extent_file, bs->open_flags, e= rrp); + if (ret) { + bdrv_unref_child(bs, extent_file); + return ret; + } + extent =3D &s->extents[s->num_extents - 1]; } else { error_setg(errp, "Unsupported extent type '%s'", type); bdrv_unref_child(bs, extent_file); @@ -957,6 +1219,7 @@ static int vmdk_open_desc_file(BlockDriverState *bs, i= nt flags, char *buf, if (strcmp(ct, "monolithicFlat") && strcmp(ct, "vmfs") && strcmp(ct, "vmfsSparse") && + strcmp(ct, "seSparse") && strcmp(ct, "twoGbMaxExtentSparse") && strcmp(ct, "twoGbMaxExtentFlat")) { error_setg(errp, "Unsupported image type '%s'", ct); @@ -1207,10 +1470,12 @@ static int get_cluster_offset(BlockDriverState *bs, { unsigned int l1_index, l2_offset, l2_index; int min_index, i, j; - uint32_t min_count, *l2_table; + uint32_t min_count; + void *l2_table; bool zeroed =3D false; int64_t ret; int64_t cluster_sector; + unsigned int l2_size_bytes =3D extent->l2_size * extent->entry_size; =20 if (m_data) { m_data->valid =3D 0; @@ -1225,7 +1490,36 @@ static int get_cluster_offset(BlockDriverState *bs, if (l1_index >=3D extent->l1_size) { return VMDK_ERROR; } - l2_offset =3D extent->l1_table[l1_index]; + if (extent->sesparse) { + uint64_t l2_offset_u64; + + assert(extent->entry_size =3D=3D sizeof(uint64_t)); + + l2_offset_u64 =3D ((uint64_t *)extent->l1_table)[l1_index]; + if (l2_offset_u64 =3D=3D 0) { + l2_offset =3D 0; + } else if ((l2_offset_u64 & 0xffffffff00000000) !=3D 0x10000000000= 00000) { + /* + * Top most nibble is 0x1 if grain table is allocated. + * strict check - top most 4 bytes must be 0x10000000 since max + * supported size is 64TB for disk - so no more than 64TB / 16= MB + * grain directories which is smaller than uint32, + * where 16MB is the only supported default grain table covera= ge. + */ + return VMDK_ERROR; + } else { + l2_offset_u64 =3D l2_offset_u64 & 0x00000000ffffffff; + l2_offset_u64 =3D extent->sesparse_l2_tables_offset + + l2_offset_u64 * l2_size_bytes / SECTOR_SIZE; + if (l2_offset_u64 > 0x00000000ffffffff) { + return VMDK_ERROR; + } + l2_offset =3D (unsigned int)(l2_offset_u64); + } + } else { + assert(extent->entry_size =3D=3D sizeof(uint32_t)); + l2_offset =3D ((uint32_t *)extent->l1_table)[l1_index]; + } if (!l2_offset) { return VMDK_UNALLOC; } @@ -1237,7 +1531,7 @@ static int get_cluster_offset(BlockDriverState *bs, extent->l2_cache_counts[j] >>=3D 1; } } - l2_table =3D extent->l2_cache + (i * extent->l2_size); + l2_table =3D (char *)extent->l2_cache + (i * l2_size_bytes); goto found; } } @@ -1250,13 +1544,13 @@ static int get_cluster_offset(BlockDriverState *bs, min_index =3D i; } } - l2_table =3D extent->l2_cache + (min_index * extent->l2_size); + l2_table =3D (char *)extent->l2_cache + (min_index * l2_size_bytes); BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD); if (bdrv_pread(extent->file, (int64_t)l2_offset * 512, l2_table, - extent->l2_size * sizeof(uint32_t) - ) !=3D extent->l2_size * sizeof(uint32_t)) { + l2_size_bytes + ) !=3D l2_size_bytes) { return VMDK_ERROR; } =20 @@ -1264,16 +1558,45 @@ static int get_cluster_offset(BlockDriverState *bs, extent->l2_cache_counts[min_index] =3D 1; found: l2_index =3D ((offset >> 9) / extent->cluster_sectors) % extent->l2_si= ze; - cluster_sector =3D le32_to_cpu(l2_table[l2_index]); =20 - if (extent->has_zero_grain && cluster_sector =3D=3D VMDK_GTE_ZEROED) { - zeroed =3D true; + if (extent->sesparse) { + cluster_sector =3D le64_to_cpu(((uint64_t *)l2_table)[l2_index]); + switch (cluster_sector & 0xf000000000000000) { + case 0x0000000000000000: + /* unallocated grain */ + if (cluster_sector !=3D 0) { + return VMDK_ERROR; + } + break; + case 0x1000000000000000: + /* scsi-unmapped grain - fallthrough */ + case 0x2000000000000000: + /* zero grain */ + zeroed =3D true; + break; + case 0x3000000000000000: + /* allocated grain */ + cluster_sector =3D (((cluster_sector & 0x0fff000000000000) >> = 48) | + ((cluster_sector & 0x0000ffffffffffff) << 12= )); + cluster_sector =3D extent->sesparse_clusters_offset + + cluster_sector * extent->cluster_sectors; + break; + default: + return VMDK_ERROR; + } + } else { + cluster_sector =3D le32_to_cpu(((uint32_t *)l2_table)[l2_index]); + + if (extent->has_zero_grain && cluster_sector =3D=3D VMDK_GTE_ZEROE= D) { + zeroed =3D true; + } } =20 if (!cluster_sector || zeroed) { if (!allocate) { return zeroed ? VMDK_ZEROED : VMDK_UNALLOC; } + assert(!extent->sesparse); =20 if (extent->next_cluster_sector >=3D VMDK_EXTENT_MAX_SECTORS) { return VMDK_ERROR; @@ -1297,7 +1620,7 @@ static int get_cluster_offset(BlockDriverState *bs, m_data->l1_index =3D l1_index; m_data->l2_index =3D l2_index; m_data->l2_offset =3D l2_offset; - m_data->l2_cache_entry =3D &l2_table[l2_index]; + m_data->l2_cache_entry =3D ((uint32_t *)l2_table) + l2_index; } } *cluster_offset =3D cluster_sector << BDRV_SECTOR_BITS; @@ -1623,6 +1946,9 @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_= t offset, if (!extent) { return -EIO; } + if (extent->sesparse) { + return -ENOTSUP; + } offset_in_cluster =3D vmdk_find_offset_in_cluster(extent, offset); n_bytes =3D MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE - offset_in_cluster); --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561388495; cv=none; d=zoho.com; s=zohoarc; b=TGnKRREgISjCWsFRIVCv+DZRUSSkr72vItbTGEUTWq7h80nMG9R4YXw9kMFc4lv1ZzCeKN8+gr7MlpqAsFfKQSgOhYWO1sQJmMn0+qFcQNKHR/oLzIw77uYCIQRyBx7bKEuMquO98pwkaLtDc2lC9tZDXDWNcI00mOkaL7N9SM4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561388495; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=T43vwLKxwFUHlzf88+EcPdZ9PGVp3lMBnz+oiPPt5as=; b=hZmd0c4E5w3nS/4jruiaR4N6ag/tKlMJHLRmNnulgYQTueRs4WNkqTBkJXxTrgoQ37gBjspSbpCHaJ15jpUqsrb/MBZnPEGnz8nM+IOJwOydzoRYysBsJAdUgHYOXkGhOrRf00DpARzmzFzIpZYWEj62Uo3kNsYQQabgaur2pxs= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561388495150545.8486942779878; Mon, 24 Jun 2019 08:01:35 -0700 (PDT) Received: from localhost ([::1]:52070 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQTK-0006r0-Tf for importer@patchew.org; Mon, 24 Jun 2019 11:01:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60252) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGg-00033N-2W for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGa-0006tG-8h for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:25 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38570) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGI-0006C6-7A; Mon, 24 Jun 2019 10:48:02 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7BD873001A77; Mon, 24 Jun 2019 14:48:01 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 81A0219C79; Mon, 24 Jun 2019 14:48:00 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:39 +0200 Message-Id: <20190624144740.5338-8-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Mon, 24 Jun 2019 14:48:01 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 7/8] ssh: switch from libssh2 to libssh X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Pino Toscano Rewrite the implementation of the ssh block driver to use libssh instead of libssh2. The libssh library has various advantages over libssh2: - easier API for authentication (for example for using ssh-agent) - easier API for known_hosts handling - supports newer types of keys in known_hosts Use APIs/features available in libssh 0.8 conditionally, to support older versions (which are not recommended though). Adjust the iotest 207 according to the different error message, and to find the default key type for localhost (to properly compare the fingerprint with). Contributed-by: Max Reitz Adjust the various Docker/Travis scripts to use libssh when available instead of libssh2. The mingw/mxe testing is dropped for now, as there are no packages for it. Signed-off-by: Pino Toscano Tested-by: Philippe Mathieu-Daud=C3=A9 Acked-by: Alex Benn=C3=A9e Message-id: 20190620200840.17655-1-ptoscano@redhat.com Reviewed-by: Philippe Mathieu-Daud=C3=A9 Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com Signed-off-by: Max Reitz --- configure | 65 +- block/Makefile.objs | 6 +- block/ssh.c | 652 ++++++++++-------- .travis.yml | 4 +- block/trace-events | 14 +- docs/qemu-block-drivers.texi | 2 +- .../dockerfiles/debian-win32-cross.docker | 1 - .../dockerfiles/debian-win64-cross.docker | 1 - tests/docker/dockerfiles/fedora.docker | 4 +- tests/docker/dockerfiles/ubuntu.docker | 2 +- tests/docker/dockerfiles/ubuntu1804.docker | 2 +- tests/qemu-iotests/207 | 54 +- tests/qemu-iotests/207.out | 2 +- 13 files changed, 449 insertions(+), 360 deletions(-) diff --git a/configure b/configure index b091b82cb3..5c7914570e 100755 --- a/configure +++ b/configure @@ -472,7 +472,7 @@ auth_pam=3D"" vte=3D"" virglrenderer=3D"" tpm=3D"" -libssh2=3D"" +libssh=3D"" live_block_migration=3D"yes" numa=3D"" tcmalloc=3D"no" @@ -1439,9 +1439,9 @@ for opt do ;; --enable-tpm) tpm=3D"yes" ;; - --disable-libssh2) libssh2=3D"no" + --disable-libssh) libssh=3D"no" ;; - --enable-libssh2) libssh2=3D"yes" + --enable-libssh) libssh=3D"yes" ;; --disable-live-block-migration) live_block_migration=3D"no" ;; @@ -1810,7 +1810,7 @@ disabled with --disable-FEATURE, default is enabled i= f available: coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backend tpm TPM support - libssh2 ssh block device support + libssh ssh block device support numa libnuma support libxml2 for Parallels image format tcmalloc tcmalloc support @@ -3914,43 +3914,34 @@ EOF fi =20 ########################################## -# libssh2 probe -min_libssh2_version=3D1.2.8 -if test "$libssh2" !=3D "no" ; then - if $pkg_config --atleast-version=3D$min_libssh2_version libssh2; then - libssh2_cflags=3D$($pkg_config libssh2 --cflags) - libssh2_libs=3D$($pkg_config libssh2 --libs) - libssh2=3Dyes +# libssh probe +if test "$libssh" !=3D "no" ; then + if $pkg_config --exists libssh; then + libssh_cflags=3D$($pkg_config libssh --cflags) + libssh_libs=3D$($pkg_config libssh --libs) + libssh=3Dyes else - if test "$libssh2" =3D "yes" ; then - error_exit "libssh2 >=3D $min_libssh2_version required for --enable-= libssh2" + if test "$libssh" =3D "yes" ; then + error_exit "libssh required for --enable-libssh" fi - libssh2=3Dno + libssh=3Dno fi fi =20 ########################################## -# libssh2_sftp_fsync probe +# Check for libssh 0.8 +# This is done like this instead of using the LIBSSH_VERSION_* and +# SSH_VERSION_* macros because some distributions in the past shipped +# snapshots of the future 0.8 from Git, and those snapshots did not +# have updated version numbers (still referring to 0.7.0). =20 -if test "$libssh2" =3D "yes"; then +if test "$libssh" =3D "yes"; then cat > $TMPC < -#include -#include -int main(void) { - LIBSSH2_SESSION *session; - LIBSSH2_SFTP *sftp; - LIBSSH2_SFTP_HANDLE *sftp_handle; - session =3D libssh2_session_init (); - sftp =3D libssh2_sftp_init (session); - sftp_handle =3D libssh2_sftp_open (sftp, "/", 0, 0); - libssh2_sftp_fsync (sftp_handle); - return 0; -} +#include +int main(void) { return ssh_get_server_publickey(NULL, NULL); } EOF - # libssh2_cflags/libssh2_libs defined in previous test. - if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then - QEMU_CFLAGS=3D"-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS" + if compile_prog "$libssh_cflags" "$libssh_libs"; then + libssh_cflags=3D"-DHAVE_LIBSSH_0_8 $libssh_cflags" fi fi =20 @@ -6451,7 +6442,7 @@ echo "GlusterFS support $glusterfs" echo "gcov $gcov_tool" echo "gcov enabled $gcov" echo "TPM support $tpm" -echo "libssh2 support $libssh2" +echo "libssh support $libssh" echo "QOM debugging $qom_cast_debug" echo "Live block migration $live_block_migration" echo "lzo support $lzo" @@ -7144,10 +7135,10 @@ if test "$glusterfs_iocb_has_stat" =3D "yes" ; then echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=3Dy" >> $config_host_mak fi =20 -if test "$libssh2" =3D "yes" ; then - echo "CONFIG_LIBSSH2=3Dm" >> $config_host_mak - echo "LIBSSH2_CFLAGS=3D$libssh2_cflags" >> $config_host_mak - echo "LIBSSH2_LIBS=3D$libssh2_libs" >> $config_host_mak +if test "$libssh" =3D "yes" ; then + echo "CONFIG_LIBSSH=3Dm" >> $config_host_mak + echo "LIBSSH_CFLAGS=3D$libssh_cflags" >> $config_host_mak + echo "LIBSSH_LIBS=3D$libssh_libs" >> $config_host_mak fi =20 if test "$live_block_migration" =3D "yes" ; then diff --git a/block/Makefile.objs b/block/Makefile.objs index dbd1522722..35f3bca4d9 100644 --- a/block/Makefile.objs +++ b/block/Makefile.objs @@ -31,7 +31,7 @@ block-obj-$(CONFIG_CURL) +=3D curl.o block-obj-$(CONFIG_RBD) +=3D rbd.o block-obj-$(CONFIG_GLUSTERFS) +=3D gluster.o block-obj-$(CONFIG_VXHS) +=3D vxhs.o -block-obj-$(CONFIG_LIBSSH2) +=3D ssh.o +block-obj-$(CONFIG_LIBSSH) +=3D ssh.o block-obj-y +=3D accounting.o dirty-bitmap.o block-obj-y +=3D write-threshold.o block-obj-y +=3D backup.o @@ -52,8 +52,8 @@ rbd.o-libs :=3D $(RBD_LIBS) gluster.o-cflags :=3D $(GLUSTERFS_CFLAGS) gluster.o-libs :=3D $(GLUSTERFS_LIBS) vxhs.o-libs :=3D $(VXHS_LIBS) -ssh.o-cflags :=3D $(LIBSSH2_CFLAGS) -ssh.o-libs :=3D $(LIBSSH2_LIBS) +ssh.o-cflags :=3D $(LIBSSH_CFLAGS) +ssh.o-libs :=3D $(LIBSSH_LIBS) block-obj-dmg-bz2-$(CONFIG_BZIP2) +=3D dmg-bz2.o block-obj-$(if $(CONFIG_DMG),m,n) +=3D $(block-obj-dmg-bz2-y) dmg-bz2.o-libs :=3D $(BZIP2_LIBS) diff --git a/block/ssh.c b/block/ssh.c index 6da7b9cbfe..501933b855 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -24,8 +24,8 @@ =20 #include "qemu/osdep.h" =20 -#include -#include +#include +#include =20 #include "block/block_int.h" #include "block/qdict.h" @@ -46,13 +46,11 @@ #include "trace.h" =20 /* - * TRACE_LIBSSH2=3D enables tracing in libssh2 itself. Note - * that this requires that libssh2 was specially compiled with the - * `./configure --enable-debug' option, so most likely you will have - * to compile it yourself. The meaning of is described - * here: http://www.libssh2.org/libssh2_trace.html + * TRACE_LIBSSH=3D enables tracing in libssh itself. + * The meaning of is described here: + * http://api.libssh.org/master/group__libssh__log.html */ -#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */ +#define TRACE_LIBSSH 0 /* see: SSH_LOG_* */ =20 typedef struct BDRVSSHState { /* Coroutine. */ @@ -60,18 +58,15 @@ typedef struct BDRVSSHState { =20 /* SSH connection. */ int sock; /* socket */ - LIBSSH2_SESSION *session; /* ssh session */ - LIBSSH2_SFTP *sftp; /* sftp session */ - LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */ + ssh_session session; /* ssh session */ + sftp_session sftp; /* sftp session */ + sftp_file sftp_handle; /* sftp remote file handle */ =20 - /* See ssh_seek() function below. */ - int64_t offset; - bool offset_op_read; - - /* File attributes at open. We try to keep the .filesize field + /* + * File attributes at open. We try to keep the .size field * updated if it changes (eg by writing at the end of the file). */ - LIBSSH2_SFTP_ATTRIBUTES attrs; + sftp_attributes attrs; =20 InetSocketAddress *inet; =20 @@ -91,7 +86,6 @@ static void ssh_state_init(BDRVSSHState *s) { memset(s, 0, sizeof *s); s->sock =3D -1; - s->offset =3D -1; qemu_co_mutex_init(&s->lock); } =20 @@ -99,20 +93,18 @@ static void ssh_state_free(BDRVSSHState *s) { g_free(s->user); =20 + if (s->attrs) { + sftp_attributes_free(s->attrs); + } if (s->sftp_handle) { - libssh2_sftp_close(s->sftp_handle); + sftp_close(s->sftp_handle); } if (s->sftp) { - libssh2_sftp_shutdown(s->sftp); + sftp_free(s->sftp); } if (s->session) { - libssh2_session_disconnect(s->session, - "from qemu ssh client: " - "user closed the connection"); - libssh2_session_free(s->session); - } - if (s->sock >=3D 0) { - close(s->sock); + ssh_disconnect(s->session); + ssh_free(s->session); /* This frees s->sock */ } } =20 @@ -127,13 +119,13 @@ session_error_setg(Error **errp, BDRVSSHState *s, con= st char *fs, ...) va_end(args); =20 if (s->session) { - char *ssh_err; + const char *ssh_err; int ssh_err_code; =20 - /* This is not an errno. See . */ - ssh_err_code =3D libssh2_session_last_error(s->session, - &ssh_err, NULL, 0); - error_setg(errp, "%s: %s (libssh2 error code: %d)", + /* This is not an errno. See . */ + ssh_err =3D ssh_get_error(s->session); + ssh_err_code =3D ssh_get_error_code(s->session); + error_setg(errp, "%s: %s (libssh error code: %d)", msg, ssh_err, ssh_err_code); } else { error_setg(errp, "%s", msg); @@ -152,18 +144,18 @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const = char *fs, ...) va_end(args); =20 if (s->sftp) { - char *ssh_err; + const char *ssh_err; int ssh_err_code; - unsigned long sftp_err_code; + int sftp_err_code; =20 - /* This is not an errno. See . */ - ssh_err_code =3D libssh2_session_last_error(s->session, - &ssh_err, NULL, 0); - /* See . */ - sftp_err_code =3D libssh2_sftp_last_error((s)->sftp); + /* This is not an errno. See . */ + ssh_err =3D ssh_get_error(s->session); + ssh_err_code =3D ssh_get_error_code(s->session); + /* See . */ + sftp_err_code =3D sftp_get_error(s->sftp); =20 error_setg(errp, - "%s: %s (libssh2 error code: %d, sftp error code: %lu)", + "%s: %s (libssh error code: %d, sftp error code: %d)", msg, ssh_err, ssh_err_code, sftp_err_code); } else { error_setg(errp, "%s", msg); @@ -173,15 +165,15 @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const = char *fs, ...) =20 static void sftp_error_trace(BDRVSSHState *s, const char *op) { - char *ssh_err; + const char *ssh_err; int ssh_err_code; - unsigned long sftp_err_code; + int sftp_err_code; =20 - /* This is not an errno. See . */ - ssh_err_code =3D libssh2_session_last_error(s->session, - &ssh_err, NULL, 0); - /* See . */ - sftp_err_code =3D libssh2_sftp_last_error((s)->sftp); + /* This is not an errno. See . */ + ssh_err =3D ssh_get_error(s->session); + ssh_err_code =3D ssh_get_error_code(s->session); + /* See . */ + sftp_err_code =3D sftp_get_error(s->sftp); =20 trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code); } @@ -282,82 +274,120 @@ static void ssh_parse_filename(const char *filename,= QDict *options, parse_uri(filename, options, errp); } =20 -static int check_host_key_knownhosts(BDRVSSHState *s, - const char *host, int port, Error **e= rrp) +static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp) { - const char *home; - char *knh_file =3D NULL; - LIBSSH2_KNOWNHOSTS *knh =3D NULL; - struct libssh2_knownhost *found; - int ret, r; - const char *hostkey; - size_t len; - int type; - - hostkey =3D libssh2_session_hostkey(s->session, &len, &type); - if (!hostkey) { + int ret; +#ifdef HAVE_LIBSSH_0_8 + enum ssh_known_hosts_e state; + int r; + ssh_key pubkey; + enum ssh_keytypes_e pubkey_type; + unsigned char *server_hash =3D NULL; + size_t server_hash_len; + char *fingerprint =3D NULL; + + state =3D ssh_session_is_known_server(s->session); + trace_ssh_server_status(state); + + switch (state) { + case SSH_KNOWN_HOSTS_OK: + /* OK */ + trace_ssh_check_host_key_knownhosts(); + break; + case SSH_KNOWN_HOSTS_CHANGED: ret =3D -EINVAL; - session_error_setg(errp, s, "failed to read remote host key"); + r =3D ssh_get_server_publickey(s->session, &pubkey); + if (r =3D=3D 0) { + r =3D ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256, + &server_hash, &server_hash_len); + pubkey_type =3D ssh_key_type(pubkey); + ssh_key_free(pubkey); + } + if (r =3D=3D 0) { + fingerprint =3D ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SH= A256, + server_hash, + server_hash_len); + ssh_clean_pubkey_hash(&server_hash); + } + if (fingerprint) { + error_setg(errp, + "host key (%s key with fingerprint %s) does not mat= ch " + "the one in known_hosts; this may be a possible att= ack", + ssh_key_type_to_char(pubkey_type), fingerprint); + ssh_string_free_char(fingerprint); + } else { + error_setg(errp, + "host key does not match the one in known_hosts; th= is " + "may be a possible attack"); + } goto out; - } - - knh =3D libssh2_knownhost_init(s->session); - if (!knh) { + case SSH_KNOWN_HOSTS_OTHER: ret =3D -EINVAL; - session_error_setg(errp, s, - "failed to initialize known hosts support"); + error_setg(errp, + "host key for this server not found, another type exist= s"); + goto out; + case SSH_KNOWN_HOSTS_UNKNOWN: + ret =3D -EINVAL; + error_setg(errp, "no host key was found in known_hosts"); + goto out; + case SSH_KNOWN_HOSTS_NOT_FOUND: + ret =3D -ENOENT; + error_setg(errp, "known_hosts file not found"); + goto out; + case SSH_KNOWN_HOSTS_ERROR: + ret =3D -EINVAL; + error_setg(errp, "error while checking the host"); + goto out; + default: + ret =3D -EINVAL; + error_setg(errp, "error while checking for known server (%d)", sta= te); goto out; } +#else /* !HAVE_LIBSSH_0_8 */ + int state; =20 - home =3D getenv("HOME"); - if (home) { - knh_file =3D g_strdup_printf("%s/.ssh/known_hosts", home); - } else { - knh_file =3D g_strdup_printf("/root/.ssh/known_hosts"); - } - - /* Read all known hosts from OpenSSH-style known_hosts file. */ - libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENS= SH); + state =3D ssh_is_server_known(s->session); + trace_ssh_server_status(state); =20 - r =3D libssh2_knownhost_checkp(knh, host, port, hostkey, len, - LIBSSH2_KNOWNHOST_TYPE_PLAIN| - LIBSSH2_KNOWNHOST_KEYENC_RAW, - &found); - switch (r) { - case LIBSSH2_KNOWNHOST_CHECK_MATCH: + switch (state) { + case SSH_SERVER_KNOWN_OK: /* OK */ - trace_ssh_check_host_key_knownhosts(found->key); + trace_ssh_check_host_key_knownhosts(); break; - case LIBSSH2_KNOWNHOST_CHECK_MISMATCH: + case SSH_SERVER_KNOWN_CHANGED: ret =3D -EINVAL; - session_error_setg(errp, s, - "host key does not match the one in known_hosts" - " (found key %s)", found->key); + error_setg(errp, + "host key does not match the one in known_hosts; this " + "may be a possible attack"); goto out; - case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND: + case SSH_SERVER_FOUND_OTHER: ret =3D -EINVAL; - session_error_setg(errp, s, "no host key was found in known_hosts"= ); + error_setg(errp, + "host key for this server not found, another type exist= s"); + goto out; + case SSH_SERVER_FILE_NOT_FOUND: + ret =3D -ENOENT; + error_setg(errp, "known_hosts file not found"); goto out; - case LIBSSH2_KNOWNHOST_CHECK_FAILURE: + case SSH_SERVER_NOT_KNOWN: ret =3D -EINVAL; - session_error_setg(errp, s, - "failure matching the host key with known_hosts"); + error_setg(errp, "no host key was found in known_hosts"); + goto out; + case SSH_SERVER_ERROR: + ret =3D -EINVAL; + error_setg(errp, "server error"); goto out; default: ret =3D -EINVAL; - session_error_setg(errp, s, "unknown error matching the host key" - " with known_hosts (%d)", r); + error_setg(errp, "error while checking for known server (%d)", sta= te); goto out; } +#endif /* !HAVE_LIBSSH_0_8 */ =20 /* known_hosts checking successful. */ ret =3D 0; =20 out: - if (knh !=3D NULL) { - libssh2_knownhost_free(knh); - } - g_free(knh_file); return ret; } =20 @@ -401,18 +431,34 @@ static int compare_fingerprint(const unsigned char *f= ingerprint, size_t len, =20 static int check_host_key_hash(BDRVSSHState *s, const char *hash, - int hash_type, size_t fingerprint_len, Error **errp) + enum ssh_publickey_hash_type type, Error **errp) { - const char *fingerprint; - - fingerprint =3D libssh2_hostkey_hash(s->session, hash_type); - if (!fingerprint) { + int r; + ssh_key pubkey; + unsigned char *server_hash; + size_t server_hash_len; + +#ifdef HAVE_LIBSSH_0_8 + r =3D ssh_get_server_publickey(s->session, &pubkey); +#else + r =3D ssh_get_publickey(s->session, &pubkey); +#endif + if (r !=3D SSH_OK) { session_error_setg(errp, s, "failed to read remote host key"); return -EINVAL; } =20 - if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len, - hash) !=3D 0) { + r =3D ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_= len); + ssh_key_free(pubkey); + if (r !=3D 0) { + session_error_setg(errp, s, + "failed reading the hash of the server SSH key"= ); + return -EINVAL; + } + + r =3D compare_fingerprint(server_hash, server_hash_len, hash); + ssh_clean_pubkey_hash(&server_hash); + if (r !=3D 0) { error_setg(errp, "remote host key does not match host_key_check '%= s'", hash); return -EPERM; @@ -421,8 +467,7 @@ check_host_key_hash(BDRVSSHState *s, const char *hash, return 0; } =20 -static int check_host_key(BDRVSSHState *s, const char *host, int port, - SshHostKeyCheck *hkc, Error **errp) +static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **e= rrp) { SshHostKeyCheckMode mode; =20 @@ -438,15 +483,15 @@ static int check_host_key(BDRVSSHState *s, const char= *host, int port, case SSH_HOST_KEY_CHECK_MODE_HASH: if (hkc->u.hash.type =3D=3D SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) { return check_host_key_hash(s, hkc->u.hash.hash, - LIBSSH2_HOSTKEY_HASH_MD5, 16, errp); + SSH_PUBLICKEY_HASH_MD5, errp); } else if (hkc->u.hash.type =3D=3D SSH_HOST_KEY_CHECK_HASH_TYPE_SH= A1) { return check_host_key_hash(s, hkc->u.hash.hash, - LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp= ); + SSH_PUBLICKEY_HASH_SHA1, errp); } g_assert_not_reached(); break; case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS: - return check_host_key_knownhosts(s, host, port, errp); + return check_host_key_knownhosts(s, errp); default: g_assert_not_reached(); } @@ -454,60 +499,43 @@ static int check_host_key(BDRVSSHState *s, const char= *host, int port, return -EINVAL; } =20 -static int authenticate(BDRVSSHState *s, const char *user, Error **errp) +static int authenticate(BDRVSSHState *s, Error **errp) { int r, ret; - const char *userauthlist; - LIBSSH2_AGENT *agent =3D NULL; - struct libssh2_agent_publickey *identity; - struct libssh2_agent_publickey *prev_identity =3D NULL; + int method; =20 - userauthlist =3D libssh2_userauth_list(s->session, user, strlen(user)); - if (strstr(userauthlist, "publickey") =3D=3D NULL) { + /* Try to authenticate with the "none" method. */ + r =3D ssh_userauth_none(s->session, NULL); + if (r =3D=3D SSH_AUTH_ERROR) { ret =3D -EPERM; - error_setg(errp, - "remote server does not support \"publickey\" authenticati= on"); + session_error_setg(errp, s, "failed to authenticate using none " + "authentication"); goto out; - } - - /* Connect to ssh-agent and try each identity in turn. */ - agent =3D libssh2_agent_init(s->session); - if (!agent) { - ret =3D -EINVAL; - session_error_setg(errp, s, "failed to initialize ssh-agent suppor= t"); - goto out; - } - if (libssh2_agent_connect(agent)) { - ret =3D -ECONNREFUSED; - session_error_setg(errp, s, "failed to connect to ssh-agent"); - goto out; - } - if (libssh2_agent_list_identities(agent)) { - ret =3D -EINVAL; - session_error_setg(errp, s, - "failed requesting identities from ssh-agent"); + } else if (r =3D=3D SSH_AUTH_SUCCESS) { + /* Authenticated! */ + ret =3D 0; goto out; } =20 - for(;;) { - r =3D libssh2_agent_get_identity(agent, &identity, prev_identity); - if (r =3D=3D 1) { /* end of list */ - break; - } - if (r < 0) { + method =3D ssh_userauth_list(s->session, NULL); + trace_ssh_auth_methods(method); + + /* + * Try to authenticate with publickey, using the ssh-agent + * if available. + */ + if (method & SSH_AUTH_METHOD_PUBLICKEY) { + r =3D ssh_userauth_publickey_auto(s->session, NULL, NULL); + if (r =3D=3D SSH_AUTH_ERROR) { ret =3D -EINVAL; - session_error_setg(errp, s, - "failed to obtain identity from ssh-agent"); + session_error_setg(errp, s, "failed to authenticate using " + "publickey authentication"); goto out; - } - r =3D libssh2_agent_userauth(agent, user, identity); - if (r =3D=3D 0) { + } else if (r =3D=3D SSH_AUTH_SUCCESS) { /* Authenticated! */ ret =3D 0; goto out; } - /* Failed to authenticate with this identity, try the next one. */ - prev_identity =3D identity; } =20 ret =3D -EPERM; @@ -515,13 +543,6 @@ static int authenticate(BDRVSSHState *s, const char *u= ser, Error **errp) "and the identities held by your ssh-agent"); =20 out: - if (agent !=3D NULL) { - /* Note: libssh2 implementation implicitly calls - * libssh2_agent_disconnect if necessary. - */ - libssh2_agent_free(agent); - } - return ret; } =20 @@ -640,7 +661,8 @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOpti= onsSsh *opts, int ssh_flags, int creat_mode, Error **errp) { int r, ret; - long port =3D 0; + unsigned int port =3D 0; + int new_sock =3D -1; =20 if (opts->has_user) { s->user =3D g_strdup(opts->user); @@ -657,71 +679,147 @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevO= ptionsSsh *opts, s->inet =3D opts->server; opts->server =3D NULL; =20 - if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) { + if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) { error_setg(errp, "Use only numeric port value"); ret =3D -EINVAL; goto err; } =20 /* Open the socket and connect. */ - s->sock =3D inet_connect_saddr(s->inet, errp); - if (s->sock < 0) { + new_sock =3D inet_connect_saddr(s->inet, errp); + if (new_sock < 0) { ret =3D -EIO; goto err; } =20 + /* + * Try to disable the Nagle algorithm on TCP sockets to reduce latency, + * but do not fail if it cannot be disabled. + */ + r =3D socket_set_nodelay(new_sock); + if (r < 0) { + warn_report("can't set TCP_NODELAY for the ssh server %s: %s", + s->inet->host, strerror(errno)); + } + /* Create SSH session. */ - s->session =3D libssh2_session_init(); + s->session =3D ssh_new(); if (!s->session) { ret =3D -EINVAL; - session_error_setg(errp, s, "failed to initialize libssh2 session"= ); + session_error_setg(errp, s, "failed to initialize libssh session"); goto err; } =20 -#if TRACE_LIBSSH2 !=3D 0 - libssh2_trace(s->session, TRACE_LIBSSH2); -#endif + /* + * Make sure we are in blocking mode during the connection and + * authentication phases. + */ + ssh_set_blocking(s->session, 1); =20 - r =3D libssh2_session_handshake(s->session, s->sock); - if (r !=3D 0) { + r =3D ssh_options_set(s->session, SSH_OPTIONS_USER, s->user); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, + "failed to set the user in the libssh session"); + goto err; + } + + r =3D ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, + "failed to set the host in the libssh session"); + goto err; + } + + if (port > 0) { + r =3D ssh_options_set(s->session, SSH_OPTIONS_PORT, &port); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, + "failed to set the port in the libssh sessi= on"); + goto err; + } + } + + r =3D ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none"); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, + "failed to disable the compression in the libss= h " + "session"); + goto err; + } + + /* Read ~/.ssh/config. */ + r =3D ssh_options_parse_config(s->session, NULL); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, "failed to parse ~/.ssh/config"); + goto err; + } + + r =3D ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock); + if (r < 0) { + ret =3D -EINVAL; + session_error_setg(errp, s, + "failed to set the socket in the libssh session= "); + goto err; + } + /* libssh took ownership of the socket. */ + s->sock =3D new_sock; + new_sock =3D -1; + + /* Connect. */ + r =3D ssh_connect(s->session); + if (r !=3D SSH_OK) { ret =3D -EINVAL; session_error_setg(errp, s, "failed to establish SSH session"); goto err; } =20 /* Check the remote host's key against known_hosts. */ - ret =3D check_host_key(s, s->inet->host, port, opts->host_key_check, e= rrp); + ret =3D check_host_key(s, opts->host_key_check, errp); if (ret < 0) { goto err; } =20 /* Authenticate. */ - ret =3D authenticate(s, s->user, errp); + ret =3D authenticate(s, errp); if (ret < 0) { goto err; } =20 /* Start SFTP. */ - s->sftp =3D libssh2_sftp_init(s->session); + s->sftp =3D sftp_new(s->session); if (!s->sftp) { - session_error_setg(errp, s, "failed to initialize sftp handle"); + session_error_setg(errp, s, "failed to create sftp handle"); + ret =3D -EINVAL; + goto err; + } + + r =3D sftp_init(s->sftp); + if (r < 0) { + sftp_error_setg(errp, s, "failed to initialize sftp handle"); ret =3D -EINVAL; goto err; } =20 /* Open the remote file. */ trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode); - s->sftp_handle =3D libssh2_sftp_open(s->sftp, opts->path, ssh_flags, - creat_mode); + s->sftp_handle =3D sftp_open(s->sftp, opts->path, ssh_flags, creat_mod= e); if (!s->sftp_handle) { - session_error_setg(errp, s, "failed to open remote file '%s'", - opts->path); + sftp_error_setg(errp, s, "failed to open remote file '%s'", + opts->path); ret =3D -EINVAL; goto err; } =20 - r =3D libssh2_sftp_fstat(s->sftp_handle, &s->attrs); - if (r < 0) { + /* Make sure the SFTP file is handled in blocking mode. */ + sftp_file_set_blocking(s->sftp_handle); + + s->attrs =3D sftp_fstat(s->sftp_handle); + if (!s->attrs) { sftp_error_setg(errp, s, "failed to read file attributes"); return -EINVAL; } @@ -729,21 +827,27 @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOp= tionsSsh *opts, return 0; =20 err: + if (s->attrs) { + sftp_attributes_free(s->attrs); + } + s->attrs =3D NULL; if (s->sftp_handle) { - libssh2_sftp_close(s->sftp_handle); + sftp_close(s->sftp_handle); } s->sftp_handle =3D NULL; if (s->sftp) { - libssh2_sftp_shutdown(s->sftp); + sftp_free(s->sftp); } s->sftp =3D NULL; if (s->session) { - libssh2_session_disconnect(s->session, - "from qemu ssh client: " - "error opening connection"); - libssh2_session_free(s->session); + ssh_disconnect(s->session); + ssh_free(s->session); } s->session =3D NULL; + s->sock =3D -1; + if (new_sock >=3D 0) { + close(new_sock); + } =20 return ret; } @@ -758,9 +862,11 @@ static int ssh_file_open(BlockDriverState *bs, QDict *= options, int bdrv_flags, =20 ssh_state_init(s); =20 - ssh_flags =3D LIBSSH2_FXF_READ; + ssh_flags =3D 0; if (bdrv_flags & BDRV_O_RDWR) { - ssh_flags |=3D LIBSSH2_FXF_WRITE; + ssh_flags |=3D O_RDWR; + } else { + ssh_flags |=3D O_RDONLY; } =20 opts =3D ssh_parse_options(options, errp); @@ -775,18 +881,13 @@ static int ssh_file_open(BlockDriverState *bs, QDict = *options, int bdrv_flags, } =20 /* Go non-blocking. */ - libssh2_session_set_blocking(s->session, 0); + ssh_set_blocking(s->session, 0); =20 qapi_free_BlockdevOptionsSsh(opts); =20 return 0; =20 err: - if (s->sock >=3D 0) { - close(s->sock); - } - s->sock =3D -1; - qapi_free_BlockdevOptionsSsh(opts); =20 return ret; @@ -797,25 +898,25 @@ static int ssh_grow_file(BDRVSSHState *s, int64_t off= set, Error **errp) { ssize_t ret; char c[1] =3D { '\0' }; - int was_blocking =3D libssh2_session_get_blocking(s->session); + int was_blocking =3D ssh_is_blocking(s->session); =20 /* offset must be strictly greater than the current size so we do * not overwrite anything */ - assert(offset > 0 && offset > s->attrs.filesize); + assert(offset > 0 && offset > s->attrs->size); =20 - libssh2_session_set_blocking(s->session, 1); + ssh_set_blocking(s->session, 1); =20 - libssh2_sftp_seek64(s->sftp_handle, offset - 1); - ret =3D libssh2_sftp_write(s->sftp_handle, c, 1); + sftp_seek64(s->sftp_handle, offset - 1); + ret =3D sftp_write(s->sftp_handle, c, 1); =20 - libssh2_session_set_blocking(s->session, was_blocking); + ssh_set_blocking(s->session, was_blocking); =20 if (ret < 0) { sftp_error_setg(errp, s, "Failed to grow file"); return -EIO; } =20 - s->attrs.filesize =3D offset; + s->attrs->size =3D offset; return 0; } =20 @@ -843,8 +944,7 @@ static int ssh_co_create(BlockdevCreateOptions *options= , Error **errp) ssh_state_init(&s); =20 ret =3D connect_to_ssh(&s, opts->location, - LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE| - LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC, + O_RDWR | O_CREAT | O_TRUNC, 0644, errp); if (ret < 0) { goto fail; @@ -913,10 +1013,8 @@ static int ssh_has_zero_init(BlockDriverState *bs) /* Assume false, unless we can positively prove it's true. */ int has_zero_init =3D 0; =20 - if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) { - if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) { - has_zero_init =3D 1; - } + if (s->attrs->type =3D=3D SSH_FILEXFER_TYPE_REGULAR) { + has_zero_init =3D 1; } =20 return has_zero_init; @@ -953,12 +1051,12 @@ static coroutine_fn void co_yield(BDRVSSHState *s, B= lockDriverState *bs) .co =3D qemu_coroutine_self() }; =20 - r =3D libssh2_session_block_directions(s->session); + r =3D ssh_get_poll_flags(s->session); =20 - if (r & LIBSSH2_SESSION_BLOCK_INBOUND) { + if (r & SSH_READ_PENDING) { rd_handler =3D restart_coroutine; } - if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) { + if (r & SSH_WRITE_PENDING) { wr_handler =3D restart_coroutine; } =20 @@ -970,33 +1068,6 @@ static coroutine_fn void co_yield(BDRVSSHState *s, Bl= ockDriverState *bs) trace_ssh_co_yield_back(s->sock); } =20 -/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position - * in the remote file. Notice that it just updates a field in the - * sftp_handle structure, so there is no network traffic and it cannot - * fail. - * - * However, `libssh2_sftp_seek64' does have a catastrophic effect on - * performance since it causes the handle to throw away all in-flight - * reads and buffered readahead data. Therefore this function tries - * to be intelligent about when to call the underlying libssh2 function. - */ -#define SSH_SEEK_WRITE 0 -#define SSH_SEEK_READ 1 -#define SSH_SEEK_FORCE 2 - -static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags) -{ - bool op_read =3D (flags & SSH_SEEK_READ) !=3D 0; - bool force =3D (flags & SSH_SEEK_FORCE) !=3D 0; - - if (force || op_read !=3D s->offset_op_read || offset !=3D s->offset) { - trace_ssh_seek(offset); - libssh2_sftp_seek64(s->sftp_handle, offset); - s->offset =3D offset; - s->offset_op_read =3D op_read; - } -} - static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs, int64_t offset, size_t size, QEMUIOVector *qiov) @@ -1008,7 +1079,8 @@ static coroutine_fn int ssh_read(BDRVSSHState *s, Blo= ckDriverState *bs, =20 trace_ssh_read(offset, size); =20 - ssh_seek(s, offset, SSH_SEEK_READ); + trace_ssh_seek(offset); + sftp_seek64(s->sftp_handle, offset); =20 /* This keeps track of the current iovec element ('i'), where we * will write to next ('buf'), and the end of the current iovec @@ -1018,35 +1090,35 @@ static coroutine_fn int ssh_read(BDRVSSHState *s, B= lockDriverState *bs, buf =3D i->iov_base; end_of_vec =3D i->iov_base + i->iov_len; =20 - /* libssh2 has a hard-coded limit of 2000 bytes per request, - * although it will also do readahead behind our backs. Therefore - * we may have to do repeated reads here until we have read 'size' - * bytes. - */ for (got =3D 0; got < size; ) { + size_t request_read_size; again: - trace_ssh_read_buf(buf, end_of_vec - buf); - r =3D libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf); - trace_ssh_read_return(r); + /* + * The size of SFTP packets is limited to 32K bytes, so limit + * the amount of data requested to 16K, as libssh currently + * does not handle multiple requests on its own. + */ + request_read_size =3D MIN(end_of_vec - buf, 16384); + trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size); + r =3D sftp_read(s->sftp_handle, buf, request_read_size); + trace_ssh_read_return(r, sftp_get_error(s->sftp)); =20 - if (r =3D=3D LIBSSH2_ERROR_EAGAIN || r =3D=3D LIBSSH2_ERROR_TIMEOU= T) { + if (r =3D=3D SSH_AGAIN) { co_yield(s, bs); goto again; } - if (r < 0) { - sftp_error_trace(s, "read"); - s->offset =3D -1; - return -EIO; - } - if (r =3D=3D 0) { + if (r =3D=3D SSH_EOF || (r =3D=3D 0 && sftp_get_error(s->sftp) =3D= =3D SSH_FX_EOF)) { /* EOF: Short read so pad the buffer with zeroes and return it= . */ qemu_iovec_memset(qiov, got, 0, size - got); return 0; } + if (r <=3D 0) { + sftp_error_trace(s, "read"); + return -EIO; + } =20 got +=3D r; buf +=3D r; - s->offset +=3D r; if (buf >=3D end_of_vec && got < size) { i++; buf =3D i->iov_base; @@ -1083,7 +1155,8 @@ static int ssh_write(BDRVSSHState *s, BlockDriverStat= e *bs, =20 trace_ssh_write(offset, size); =20 - ssh_seek(s, offset, SSH_SEEK_WRITE); + trace_ssh_seek(offset); + sftp_seek64(s->sftp_handle, offset); =20 /* This keeps track of the current iovec element ('i'), where we * will read from next ('buf'), and the end of the current iovec @@ -1094,46 +1167,37 @@ static int ssh_write(BDRVSSHState *s, BlockDriverSt= ate *bs, end_of_vec =3D i->iov_base + i->iov_len; =20 for (written =3D 0; written < size; ) { + size_t request_write_size; again: - trace_ssh_write_buf(buf, end_of_vec - buf); - r =3D libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf); - trace_ssh_write_return(r); + /* + * Avoid too large data packets, as libssh currently does not + * handle multiple requests on its own. + */ + request_write_size =3D MIN(end_of_vec - buf, 131072); + trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size); + r =3D sftp_write(s->sftp_handle, buf, request_write_size); + trace_ssh_write_return(r, sftp_get_error(s->sftp)); =20 - if (r =3D=3D LIBSSH2_ERROR_EAGAIN || r =3D=3D LIBSSH2_ERROR_TIMEOU= T) { + if (r =3D=3D SSH_AGAIN) { co_yield(s, bs); goto again; } if (r < 0) { sftp_error_trace(s, "write"); - s->offset =3D -1; return -EIO; } - /* The libssh2 API is very unclear about this. A comment in - * the code says "nothing was acked, and no EAGAIN was - * received!" which apparently means that no data got sent - * out, and the underlying channel didn't return any EAGAIN - * indication. I think this is a bug in either libssh2 or - * OpenSSH (server-side). In any case, forcing a seek (to - * discard libssh2 internal buffers), and then trying again - * works for me. - */ - if (r =3D=3D 0) { - ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE); - co_yield(s, bs); - goto again; - } =20 written +=3D r; buf +=3D r; - s->offset +=3D r; if (buf >=3D end_of_vec && written < size) { i++; buf =3D i->iov_base; end_of_vec =3D i->iov_base + i->iov_len; } =20 - if (offset + written > s->attrs.filesize) - s->attrs.filesize =3D offset + written; + if (offset + written > s->attrs->size) { + s->attrs->size =3D offset + written; + } } =20 return 0; @@ -1168,24 +1232,24 @@ static void unsafe_flush_warning(BDRVSSHState *s, c= onst char *what) } } =20 -#ifdef HAS_LIBSSH2_SFTP_FSYNC +#ifdef HAVE_LIBSSH_0_8 =20 static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs) { int r; =20 trace_ssh_flush(); + + if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) { + unsafe_flush_warning(s, "OpenSSH >=3D 6.3"); + return 0; + } again: - r =3D libssh2_sftp_fsync(s->sftp_handle); - if (r =3D=3D LIBSSH2_ERROR_EAGAIN || r =3D=3D LIBSSH2_ERROR_TIMEOUT) { + r =3D sftp_fsync(s->sftp_handle); + if (r =3D=3D SSH_AGAIN) { co_yield(s, bs); goto again; } - if (r =3D=3D LIBSSH2_ERROR_SFTP_PROTOCOL && - libssh2_sftp_last_error(s->sftp) =3D=3D LIBSSH2_FX_OP_UNSUPPORTED)= { - unsafe_flush_warning(s, "OpenSSH >=3D 6.3"); - return 0; - } if (r < 0) { sftp_error_trace(s, "fsync"); return -EIO; @@ -1206,25 +1270,25 @@ static coroutine_fn int ssh_co_flush(BlockDriverSta= te *bs) return ret; } =20 -#else /* !HAS_LIBSSH2_SFTP_FSYNC */ +#else /* !HAVE_LIBSSH_0_8 */ =20 static coroutine_fn int ssh_co_flush(BlockDriverState *bs) { BDRVSSHState *s =3D bs->opaque; =20 - unsafe_flush_warning(s, "libssh2 >=3D 1.4.4"); + unsafe_flush_warning(s, "libssh >=3D 0.8.0"); return 0; } =20 -#endif /* !HAS_LIBSSH2_SFTP_FSYNC */ +#endif /* !HAVE_LIBSSH_0_8 */ =20 static int64_t ssh_getlength(BlockDriverState *bs) { BDRVSSHState *s =3D bs->opaque; int64_t length; =20 - /* Note we cannot make a libssh2 call here. */ - length =3D (int64_t) s->attrs.filesize; + /* Note we cannot make a libssh call here. */ + length =3D (int64_t) s->attrs->size; trace_ssh_getlength(length); =20 return length; @@ -1241,12 +1305,12 @@ static int coroutine_fn ssh_co_truncate(BlockDriver= State *bs, int64_t offset, return -ENOTSUP; } =20 - if (offset < s->attrs.filesize) { + if (offset < s->attrs->size) { error_setg(errp, "ssh driver does not support shrinking files"); return -ENOTSUP; } =20 - if (offset =3D=3D s->attrs.filesize) { + if (offset =3D=3D s->attrs->size) { return 0; } =20 @@ -1341,12 +1405,16 @@ static void bdrv_ssh_init(void) { int r; =20 - r =3D libssh2_init(0); + r =3D ssh_init(); if (r !=3D 0) { - fprintf(stderr, "libssh2 initialization failed, %d\n", r); + fprintf(stderr, "libssh initialization failed, %d\n", r); exit(EXIT_FAILURE); } =20 +#if TRACE_LIBSSH !=3D 0 + ssh_set_log_level(TRACE_LIBSSH); +#endif + bdrv_register(&bdrv_ssh); } =20 diff --git a/.travis.yml b/.travis.yml index aeb9b211cd..279658b116 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,7 +31,7 @@ addons: - libseccomp-dev - libspice-protocol-dev - libspice-server-dev - - libssh2-1-dev + - libssh-dev - liburcu-dev - libusb-1.0-0-dev - libvte-2.91-dev @@ -270,7 +270,7 @@ matrix: - libseccomp-dev - libspice-protocol-dev - libspice-server-dev - - libssh2-1-dev + - libssh-dev - liburcu-dev - libusb-1.0-0-dev - libvte-2.91-dev diff --git a/block/trace-events b/block/trace-events index 9ccea755da..d724df0117 100644 --- a/block/trace-events +++ b/block/trace-events @@ -171,19 +171,21 @@ nbd_client_connect_success(const char *export_name) "= export '%s'" # ssh.c ssh_restart_coroutine(void *co) "co=3D%p" ssh_flush(void) "fsync" -ssh_check_host_key_knownhosts(const char *key) "host key OK: %s" +ssh_check_host_key_knownhosts(void) "host key OK" ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags= =3D0x%x creat_mode=3D0%o" ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=3D%d r= d_handler=3D%p wr_handler=3D%p" ssh_co_yield_back(int sock) "s->sock=3D%d - back" ssh_getlength(int64_t length) "length=3D%" PRIi64 ssh_co_create_opts(uint64_t size) "total_size=3D%" PRIu64 ssh_read(int64_t offset, size_t size) "offset=3D%" PRIi64 " size=3D%zu" -ssh_read_buf(void *buf, size_t size) "sftp_read buf=3D%p size=3D%zu" -ssh_read_return(ssize_t ret) "sftp_read returned %zd" +ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=3D= %p size=3D%zu (actual size=3D%zu)" +ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp e= rror=3D%d)" ssh_write(int64_t offset, size_t size) "offset=3D%" PRIi64 " size=3D%zu" -ssh_write_buf(void *buf, size_t size) "sftp_write buf=3D%p size=3D%zu" -ssh_write_return(ssize_t ret) "sftp_write returned %zd" +ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf= =3D%p size=3D%zu (actual size=3D%zu)" +ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp= error=3D%d)" ssh_seek(int64_t offset) "seeking to offset=3D%" PRIi64 +ssh_auth_methods(int methods) "auth methods=3D0x%x" +ssh_server_status(int status) "server status=3D%d" =20 # curl.c curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld" @@ -216,4 +218,4 @@ sheepdog_snapshot_create(const char *sn_name, const cha= r *id) "%s %s" sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t v= di) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32 =20 # ssh.c -sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned= long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error cod= e: %lu)" +sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp= _err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)" diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi index da06a9bc83..91ab0eceae 100644 --- a/docs/qemu-block-drivers.texi +++ b/docs/qemu-block-drivers.texi @@ -782,7 +782,7 @@ print a warning when @code{fsync} is not supported: =20 warning: ssh server @code{ssh.example.com:22} does not support fsync =20 -With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is +With sufficiently new versions of libssh and OpenSSH, @code{fsync} is supported. =20 @node disk_images_nvme diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/doc= ker/dockerfiles/debian-win32-cross.docker index dd021f2df0..0a4970c068 100644 --- a/tests/docker/dockerfiles/debian-win32-cross.docker +++ b/tests/docker/dockerfiles/debian-win32-cross.docker @@ -15,7 +15,6 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ mxe-$TARGET-w64-mingw32.shared-curl \ mxe-$TARGET-w64-mingw32.shared-glib \ mxe-$TARGET-w64-mingw32.shared-libgcrypt \ - mxe-$TARGET-w64-mingw32.shared-libssh2 \ mxe-$TARGET-w64-mingw32.shared-libusb1 \ mxe-$TARGET-w64-mingw32.shared-lzo \ mxe-$TARGET-w64-mingw32.shared-nettle \ diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/doc= ker/dockerfiles/debian-win64-cross.docker index 4542bcc821..b27985b1b1 100644 --- a/tests/docker/dockerfiles/debian-win64-cross.docker +++ b/tests/docker/dockerfiles/debian-win64-cross.docker @@ -15,7 +15,6 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ mxe-$TARGET-w64-mingw32.shared-curl \ mxe-$TARGET-w64-mingw32.shared-glib \ mxe-$TARGET-w64-mingw32.shared-libgcrypt \ - mxe-$TARGET-w64-mingw32.shared-libssh2 \ mxe-$TARGET-w64-mingw32.shared-libusb1 \ mxe-$TARGET-w64-mingw32.shared-lzo \ mxe-$TARGET-w64-mingw32.shared-nettle \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfi= les/fedora.docker index 12c460597e..619d1b5656 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -35,7 +35,7 @@ ENV PACKAGES \ libpng-devel \ librbd-devel \ libseccomp-devel \ - libssh2-devel \ + libssh-devel \ libubsan \ libusbx-devel \ libxml2-devel \ @@ -50,7 +50,6 @@ ENV PACKAGES \ mingw32-gtk3 \ mingw32-libjpeg-turbo \ mingw32-libpng \ - mingw32-libssh2 \ mingw32-libtasn1 \ mingw32-nettle \ mingw32-pixman \ @@ -64,7 +63,6 @@ ENV PACKAGES \ mingw64-gtk3 \ mingw64-libjpeg-turbo \ mingw64-libpng \ - mingw64-libssh2 \ mingw64-libtasn1 \ mingw64-nettle \ mingw64-pixman \ diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfi= les/ubuntu.docker index 8d256961f0..d3b72209c8 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -53,7 +53,7 @@ ENV PACKAGES flex bison \ libsnappy-dev \ libspice-protocol-dev \ libspice-server-dev \ - libssh2-1-dev \ + libssh-dev \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \ diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dock= erfiles/ubuntu1804.docker index 2e2900150b..9d80b11500 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -40,7 +40,7 @@ ENV PACKAGES flex bison \ libsnappy-dev \ libspice-protocol-dev \ libspice-server-dev \ - libssh2-1-dev \ + libssh-dev \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \ diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207 index b3816136f7..ec8c1d06f0 100755 --- a/tests/qemu-iotests/207 +++ b/tests/qemu-iotests/207 @@ -110,12 +110,49 @@ with iotests.FilePath('t.img') as disk_path, \ =20 iotests.img_info_log(remote_path) =20 - md5_key =3D subprocess.check_output( - 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + - 'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1', - shell=3DTrue).rstrip().decode('ascii') + keys =3D subprocess.check_output( + 'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + + 'cut -d" " -f3', + shell=3DTrue).rstrip().decode('ascii').split('\n') + + # Mappings of base64 representations to digests + md5_keys =3D {} + sha1_keys =3D {} + + for key in keys: + md5_keys[key] =3D subprocess.check_output( + 'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key, + shell=3DTrue).rstrip().decode('ascii') + + sha1_keys[key] =3D subprocess.check_output( + 'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key, + shell=3DTrue).rstrip().decode('ascii') =20 vm.launch() + + # Find correct key first + matching_key =3D None + for key in keys: + result =3D vm.qmp('blockdev-add', + driver=3D'ssh', node_name=3D'node0', path=3Ddisk_p= ath, + server=3D{ + 'host': '127.0.0.1', + 'port': '22', + }, host_key_check=3D{ + 'mode': 'hash', + 'type': 'md5', + 'hash': md5_keys[key], + }) + + if 'error' not in result: + vm.qmp('blockdev-del', node_name=3D'node0') + matching_key =3D key + break + + if matching_key is None: + vm.shutdown() + iotests.notrun('Did not find a key that fits 127.0.0.1') + blockdev_create(vm, { 'driver': 'ssh', 'location': { 'path': disk_path, @@ -140,7 +177,7 @@ with iotests.FilePath('t.img') as disk_path, \ 'host-key-check': { 'mode': 'hash', 'type': 'md5', - 'hash': md5_key, + 'hash': md5_keys[matching_key], } }, 'size': 8388608 }) @@ -148,11 +185,6 @@ with iotests.FilePath('t.img') as disk_path, \ =20 iotests.img_info_log(remote_path) =20 - sha1_key =3D subprocess.check_output( - 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + - 'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1', - shell=3DTrue).rstrip().decode('ascii') - vm.launch() blockdev_create(vm, { 'driver': 'ssh', 'location': { @@ -178,7 +210,7 @@ with iotests.FilePath('t.img') as disk_path, \ 'host-key-check': { 'mode': 'hash', 'type': 'sha1', - 'hash': sha1_key, + 'hash': sha1_keys[matching_key], } }, 'size': 4194304 }) diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out index ec9823793a..1239d9d648 100644 --- a/tests/qemu-iotests/207.out +++ b/tests/qemu-iotests/207.out @@ -68,7 +68,7 @@ virtual size: 4 MiB (4194304 bytes) =20 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": = {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": = "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "2= 2"}}, "size": 4194304}}} {"return": {}} -Job failed: failed to open remote file '/this/is/not/an/existing/path': Fa= iled opening remote file (libssh2 error code: -31) +Job failed: failed to open remote file '/this/is/not/an/existing/path': SF= TP server: No such file (libssh error code: 1, sftp error code: 2) {"execute": "job-dismiss", "arguments": {"id": "job0"}} {"return": {}} =20 --=20 2.21.0 From nobody Wed May 15 21:35:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561388315; cv=none; d=zoho.com; s=zohoarc; b=IqgNHVzneKnPxUTyCul3CyfcjbUj/9jM4HCcs4+zYXQMitpxCT//tH8OO5ebizSRG8GFdmHweGzsaRHahjYQpWstQSfQ5eoWHmm8trrJz7KY77CGJo4xFmpLpAR1NmgeUVjfq5tnJ4nuQrWRuq2O9w/fjG1GCi7eXDgbeE3DVPs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561388315; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=/ycUdqLPh6IjKAMjP4KuRywCSc7d6UgKnokEVQR2qb0=; b=JV1Pya0WiDN+1esveDJSooZT92nnOC/vAd7xw5l7UMl1H0ZW9Rr5g1u8nt75zC2BwebRhT2u1kvs4y4M6vI1lWfMjsYTxruXvuAOyhdsN5C6JLihIJjRSlLdxJaLVExA/SkpDUjWxWbMH0CNJZyvqs+mjbC8cE7/BKoLCVO6cds= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561388315609490.77992324374577; Mon, 24 Jun 2019 07:58:35 -0700 (PDT) Received: from localhost ([::1]:52026 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQQU-0003wF-NR for importer@patchew.org; Mon, 24 Jun 2019 10:58:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60192) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfQGO-0002wy-1Z for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfQGN-0006PD-2T for qemu-devel@nongnu.org; Mon, 24 Jun 2019 10:48:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55094) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfQGK-0006Hh-Kf; Mon, 24 Jun 2019 10:48:04 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CF8FF90906; Mon, 24 Jun 2019 14:48:03 +0000 (UTC) Received: from localhost (ovpn-204-152.brq.redhat.com [10.40.204.152]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 680A919C79; Mon, 24 Jun 2019 14:48:03 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Mon, 24 Jun 2019 16:47:40 +0200 Message-Id: <20190624144740.5338-9-mreitz@redhat.com> In-Reply-To: <20190624144740.5338-1-mreitz@redhat.com> References: <20190624144740.5338-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 24 Jun 2019 14:48:03 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL v2 8/8] iotests: Fix 205 for concurrent runs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Tests should place their files into the test directory. This includes Unix sockets. 205 currently fails to do so, which prevents it from being run concurrently. Signed-off-by: Max Reitz Message-id: 20190618210238.9524-1-mreitz@redhat.com Reviewed-by: Eric Blake Signed-off-by: Max Reitz --- tests/qemu-iotests/205 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205 index 69f2c1d392..b8a86c446e 100755 --- a/tests/qemu-iotests/205 +++ b/tests/qemu-iotests/205 @@ -24,7 +24,7 @@ import iotests import time from iotests import qemu_img_create, qemu_io, filter_qemu_io, QemuIoIntera= ctive =20 -nbd_sock =3D 'nbd_sock' +nbd_sock =3D os.path.join(iotests.test_dir, 'nbd_sock') nbd_uri =3D 'nbd+unix:///exp?socket=3D' + nbd_sock disk =3D os.path.join(iotests.test_dir, 'disk') =20 --=20 2.21.0