From nobody Sun Mar 22 15:40:02 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1773962124; cv=none; d=zohomail.com; s=zohoarc; b=VglldzMID8hmMp0v0V8ykkg6CYAGk1IDjQCtPkp0Us+9KbBUUXbZFH8fM/cUM/SCbAFoTvTbU0X1iC7BF9t5v4Qoo7RxWh8RzOrtTFmnyTtrHCMxb8lGIgkdLTYWplje3NVKACwg5Ugi8IoyPBV6ZLBmPRWP7QneGs45pu3Rh3s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1773962124; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=bsHlTUEhhNOjys8kon0snbRzLZZAGqbckreegTmuncE=; b=HqwTz4/zM2nZebHEG9KiGwkknhLdz8Ptp2dW47nUzlHpSlAiSViIzvD3j4c9m1yWVLAMwmROQigz1F+uuun7+kkiPr5UcvYGNo3nZKg6zPmurcH0O3pA7FaKUETxlXPGaw98+jZZblCUlCKpBCJD/h6DrNcHd4ni/ZywRLjlu5g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17739621249430.39743463402930956; Thu, 19 Mar 2026 16:15:24 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1w3MYX-0001Pt-TR; Thu, 19 Mar 2026 19:13:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w3MYW-0001PZ-Ol for qemu-devel@nongnu.org; Thu, 19 Mar 2026 19:13:32 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w3MYV-0001bx-4P for qemu-devel@nongnu.org; Thu, 19 Mar 2026 19:13:32 -0400 Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-235-oDgWHqbxPSGTzaefkCfwYA-1; Thu, 19 Mar 2026 19:13:28 -0400 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-5091ee9f1d8so120914631cf.0 for ; Thu, 19 Mar 2026 16:13:28 -0700 (PDT) Received: from x1.local ([142.189.10.167]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-50b36e5bee3sm6717161cf.21.2026.03.19.16.13.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Mar 2026 16:13:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1773962010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bsHlTUEhhNOjys8kon0snbRzLZZAGqbckreegTmuncE=; b=iR/Vjf4wkM1Phij366R6fM/q4WpUEyYSc11EQJ4tJW7bCLKFALtwkTgLkOrhYP74wR58Ke UmVuiB/acuIWObe3XAM0D7geQmxoz5mTkiKqLsPEeimISNbkKDWDVXKGe5wMdrk+gQxCUW 7hVR7Wrwdiw4EvFsfgmGJgDHdPpqnkE= X-MC-Unique: oDgWHqbxPSGTzaefkCfwYA-1 X-Mimecast-MFC-AGG-ID: oDgWHqbxPSGTzaefkCfwYA_1773962008 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1773962008; x=1774566808; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bsHlTUEhhNOjys8kon0snbRzLZZAGqbckreegTmuncE=; b=tujCAcUQwsBfPjkXU9iPcsuPKfPIsJxF8YEwYJhnHvYEp0zxkZQrG09nb127FcLy36 D2U/R7K8y623lsSj9DtH5QzyEABSa6yq5xMx75zY5BuMXhPm/lnsVdAnqHnUulFGXYXw awsjYrhEVDKWZcEdi+z/spUMsdf+mP4PzqMIVuNnjtQwdXHsTzlnVwy3OuHXDBl/Gs9V NwaUZ8LqPQOPuRkIMzpq4UWVGI1j063g4hP1j+jSgFIhfAF45Ap7RsRPkXZgoDEqbhLa 9N5LMjNAQCjX6ZLuPGrX0VF+CwNWzwzZF8uj5si8SE3yJ1iVOzDYSx2gmwi7PeSnsPJj T/8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773962008; x=1774566808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bsHlTUEhhNOjys8kon0snbRzLZZAGqbckreegTmuncE=; b=QmwjI2X9VMI4KvapMaMUnGY6yWfKucxe2bxsXeZ7JaO72ONFZIviym3tr49JhR4Mt2 YZsyME2BJf9ADbiolZHGC3Pf852FE9ckFvxYpEk89SvENGBZfeq6Cn5hhRdWD8WOs2Ra IQ+O+3MrQMgMm6lDoXS85MAdCjrQqck9XT4eGBsQa5RYQV1Xefcf8ydKIXTfB1cZSabV 7XBELB4omrMwTF9C3poD1duQqBPIlbcIhBJ6hKvlzp86nvrSMW5lkOt3UV8lFMoDSkaA TcgQ6ZGBZtdWTYtzgXYasjF4bWoBmK1zNU+gFAYgB7pD2SxWJo5cCNFRCR3Vch9KhkX7 si1w== X-Gm-Message-State: AOJu0YwhT6MLp8HCFAyEi5svTKDaNSa/clHSqI221qJ68KVSPL2QPC7S t6FCqvGROOkwKrpB5Qbkh1PXEM8MKbukB7YSk9quNLEtanKkxXfDBVzi1P/jPnf19CNQInVbs80 9j8j7NBiyMvmspUseM6nJ/JcvVJwSrhBHcWUoUYcx/VL75ue4VYgxbsQaS4KmuUnS9yLC/IRXB7 Ae0w4QHo/ohBg/dv0Lwpg+DM7LsH53ztxkbzI/Wg== X-Gm-Gg: ATEYQzwmxVypaLObHiySIf0UUEUTiZH4BK8OtejvTGEMkYRcgtwsgtkRB1AgN5r1TnH Zam6vC+EHcb+kHZfHSqX2rYtqdfcDk15NljVJFvJoMCnAEku8/Tv2FKUpZfsUMXGc4M0SrCzRRv k8rg5Qu2vGWH8WoyEIKGHmwoU7hqTipJj1Pl7hYvQ9muMx1Oc9Ha56GvFPkM/DVE/dOoup7zsxo ew5VvtxEWUc9bHn47rYGuMadDlwUdeMIT9u/tbe5rZQdGhqPDL/tLzTr8zPJW7HqAolLChxUcCh pEa1gXae/lmiJ19VviphGTWxfTM4PAAJn4LT/uPWSc3xnnBAHQO5uNhY8gn8xQZhdwTBnbJZ7bI sjlG3ko8gNy5v6A== X-Received: by 2002:a05:622a:1181:b0:509:205:230f with SMTP id d75a77b69052e-50b3702fcbfmr15596941cf.4.1773962007876; Thu, 19 Mar 2026 16:13:27 -0700 (PDT) X-Received: by 2002:a05:622a:1181:b0:509:205:230f with SMTP id d75a77b69052e-50b3702fcbfmr15596521cf.4.1773962007255; Thu, 19 Mar 2026 16:13:27 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Juraj Marcin , Kirti Wankhede , "Maciej S . Szmigiero" , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Joao Martins , Alex Williamson , Yishai Hadas , Fabiano Rosas , Pranav Tyagi , peterx@redhat.com, Zhiyi Guo , Markus Armbruster , Avihai Horon , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Yong Huang Subject: [PATCH RFC 09/12] migration: Make iteration counter out of RAM Date: Thu, 19 Mar 2026 19:12:59 -0400 Message-ID: <20260319231302.123135-10-peterx@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260319231302.123135-1-peterx@redhat.com> References: <20260319231302.123135-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -3 X-Spam_score: -0.4 X-Spam_bar: / X-Spam_report: (-0.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.819, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.903, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1773962125384158500 Content-Type: text/plain; charset="utf-8" It used to hide in RAM dirty sync path. Now with more modules being able to slow sync on dirty information, keeping it there may not be good anymore because it's not RAM's own concept for iterations: all modules should follow. More importantly, mgmt may try to query dirty info (to make policy decisions like adjusting downtime) by listening to iteration count changes via QMP events. So we must make sure the boost of iterations only happens _after_ the dirty sync operations with whatever form (RAM's dirty bitmap sync, or VFIO's different ioctls to fetch latest dirty info from kernel). Move this to core migration path to manage, together with the event generation, so that it can be well ordered with the sync operations for all modules. This brings a good side effect that we should have an old issue regarding to cpu_throttle_dirty_sync_timer_tick() which can randomly boost iteration counts (because it invokes sync ops). Now it won't, which is actually the right behavior. Said that, we have code (not only QEMU, but likely mgmt too) assuming the 1st iteration will always shows dirty count to 1. Make it initialized with 1 this time, because we'll miss the dirty sync for setup() on boosting this counter now. Cc: Yong Huang Signed-off-by: Peter Xu Reviewed-by: Hyman Huang Reviewed-by: Prasad Pandit --- migration/migration-stats.h | 3 ++- migration/migration.c | 29 ++++++++++++++++++++++++++--- migration/ram.c | 6 ------ 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/migration/migration-stats.h b/migration/migration-stats.h index 1153520f7a..326ddb0088 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -43,7 +43,8 @@ typedef struct { */ uint64_t dirty_pages_rate; /* - * Number of times we have synchronized guest bitmaps. + * Number of times we have synchronized guest bitmaps. This always + * starts from 1 for the 1st iteration. */ uint64_t dirty_sync_count; /* diff --git a/migration/migration.c b/migration/migration.c index 42facb16d1..ad8a824585 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1654,10 +1654,15 @@ int migrate_init(MigrationState *s, Error **errp) s->threshold_size =3D 0; s->switchover_acked =3D false; s->rdma_migration =3D false; + /* - * set mig_stats memory to zero for a new migration + * set mig_stats memory to zero for a new migration.. except the + * iteration counter, which we want to make sure it returns 1 for the + * first iteration. */ memset(&mig_stats, 0, sizeof(mig_stats)); + mig_stats.dirty_sync_count =3D 1; + migration_reset_vfio_bytes_transferred(); =20 s->postcopy_package_loaded =3D false; @@ -3230,10 +3235,28 @@ static bool migration_iteration_next_ready(Migratio= nState *s, static void migration_iteration_go_next(MigPendingData *pending) { /* - * Do a slow sync will achieve this. TODO: move RAM iteration code - * into the core layer. + * Do a slow sync first before boosting the iteration count. */ qemu_savevm_query_pending(pending, false); + + /* + * Boost dirty sync count to reflect we finished one iteration. + * + * NOTE: we need to make sure when this happens (together with the + * event sent below) all modules have slow-synced the pending data + * above. That means a write mem barrier, but qatomic_add() should be + * enough. + * + * It's because a mgmt could wait on the iteration event to query again + * on pending data for policy changes (e.g. downtime adjustments). The + * ordering will make sure the query will fetch the latest results from + * all the modules. + */ + qatomic_add(&mig_stats.dirty_sync_count, 1); + + if (migrate_events()) { + qapi_event_send_migration_pass(mig_stats.dirty_sync_count); + } } =20 /* diff --git a/migration/ram.c b/migration/ram.c index 89f761a471..29e9608715 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1136,8 +1136,6 @@ static void migration_bitmap_sync(RAMState *rs, bool = last_stage) RAMBlock *block; int64_t end_time; =20 - qatomic_add(&mig_stats.dirty_sync_count, 1); - if (!rs->time_last_bitmap_sync) { rs->time_last_bitmap_sync =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIM= E); } @@ -1172,10 +1170,6 @@ static void migration_bitmap_sync(RAMState *rs, bool= last_stage) rs->num_dirty_pages_period =3D 0; rs->bytes_xfer_prev =3D migration_transferred_bytes(); } - if (migrate_events()) { - uint64_t generation =3D qatomic_read(&mig_stats.dirty_sync_count); - qapi_event_send_migration_pass(generation); - } } =20 void migration_bitmap_sync_precopy(bool last_stage) --=20 2.50.1