The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: Merge tag 'q800-for-8.1-pull-request' of https://github.com/vivier/qemu-m68k into staging (2023-06-22 10:18:32 +0200) are available in the Git repository at: https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request for you to fetch changes up to 23e4307eadc1497bd0a11ca91041768f15963b68: migration/rdma: Split qemu_fopen_rdma() into input/output functions (2023-06-22 18:11:58 +0200) ---------------------------------------------------------------- Migration Pull request (20230621) take 2 In this pull request the only change is fixing 32 bits complitaion issue. Please apply. [take 1] - fix for multifd thread creation (fabiano) - dirtylimity (hyman) * migration-test will go on next PULL request, as it has failures. - Improve error description (tejus) - improve -incoming and set parameters before calling incoming (wei) - migration atomic counters reviewed patches (quintela) - migration-test refacttoring reviewed (quintela) ---------------------------------------------------------------- Fabiano Rosas (2): migration/multifd: Rename threadinfo.c functions migration/multifd: Protect accesses to migration_threads Hyman Huang(黄勇) (8): softmmu/dirtylimit: Add parameter check for hmp "set_vcpu_dirty_limit" qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters migration: Introduce dirty-limit capability migration: Refactor auto-converge capability logic migration: Put the detection logic before auto-converge checking migration: Implement dirty-limit convergence algo migration: Extend query-migrate to provide dirty page limit info Juan Quintela (16): migration-test: Be consistent for ppc migration-test: Make machine_opts regular with other options migration-test: Create arch_opts migration-test: machine_opts is really arch specific migration-test: Create kvm_opts migration-test: bootpath is the same for all tests and for all archs migration-test: Add bootfile_create/delete() functions migration-test: dirtylimit checks for x86_64 arch before migration-test: simplify shmem_opts handling qemu-file: Rename qemu_file_transferred_ fast -> noflush migration: Change qemu_file_transferred to noflush migration: Use qemu_file_transferred_noflush() for block migration. qemu_file: Make qemu_file_is_writable() static qemu-file: Simplify qemu_file_shutdown() qemu-file: Make qemu_file_get_error_obj() static migration/rdma: Split qemu_fopen_rdma() into input/output functions Tejus GK (2): migration: Update error description whenever migration fails migration: Refactor repeated call of yank_unregister_instance Wei Wang (2): migration: enforce multifd and postcopy preempt to be set before incoming qtest/migration-tests.c: use "-incoming defer" for postcopy tests qapi/migration.json | 74 +++++++++++++++++--- include/sysemu/dirtylimit.h | 2 + migration/options.h | 1 + migration/qemu-file.h | 14 ++-- migration/threadinfo.h | 7 +- migration/block.c | 4 +- migration/migration-hmp-cmds.c | 26 +++++++ migration/migration.c | 40 +++++++---- migration/multifd.c | 4 +- migration/options.c | 87 +++++++++++++++++++++++ migration/qemu-file.c | 24 ++----- migration/ram.c | 59 +++++++++++++--- migration/rdma.c | 39 +++++------ migration/savevm.c | 6 +- migration/threadinfo.c | 19 ++++- migration/vmstate.c | 4 +- softmmu/dirtylimit.c | 97 +++++++++++++++++++++++--- tests/qtest/migration-test.c | 123 ++++++++++++++++++--------------- migration/trace-events | 1 + 19 files changed, 472 insertions(+), 159 deletions(-) base-commit: 5f9dd6a8ce3961db4ce47411ed2097ad88bdf5fc prerequisite-patch-id: 99c8bffa9428838925e330eb2881bab476122579 prerequisite-patch-id: 77ba427fd916aeb395e95aa0e7190f84e98e96ab prerequisite-patch-id: 9983d46fa438d7075a37be883529e37ae41e4228 prerequisite-patch-id: 207f7529924b12dcb57f6557d6db6f79ceb2d682 prerequisite-patch-id: 5ad1799a13845dbf893a28a202b51a6b50d95d90 prerequisite-patch-id: c51959aacd6d65ee84fcd4f1b2aed3dd6f6af879 prerequisite-patch-id: da9dbb6799b2da002c0896574334920097e4c50a prerequisite-patch-id: c1110ffafbaf5465fb277a20db809372291f7846 prerequisite-patch-id: 8307c92bedd07446214b35b40206eb6793a7384d prerequisite-patch-id: 0a6106cd4a508d5e700a7ff6c25edfdd03c8ca3d prerequisite-patch-id: 83205051de22382e75bf4acdf69e59315801fa0d prerequisite-patch-id: 8c9b3cba89d555c071a410041e6da41806106a7e prerequisite-patch-id: 0ff62a33b9a242226ccc1f5424a516de803c9fe5 prerequisite-patch-id: 25b8ae1ebe09ace14457c454cfcb23077c37346c prerequisite-patch-id: 466ea91d5be41fe345dacd4d17bbbe5ce13118c2 prerequisite-patch-id: d1045858f9729ac62eccf2e83ebf95cfebae2cb5 prerequisite-patch-id: 0276ec02073bda5426de39e2f2e81eef080b4f54 prerequisite-patch-id: 7afb4450a163cc1a63ea23831c50214966969131 prerequisite-patch-id: 06c053ce4f41db9675bd1778ae8f6a483641fcef prerequisite-patch-id: 13ea05d54d741ed08b3bfefa1fc8bedb9c81c782 prerequisite-patch-id: 99c4e2b7101bc8c4b9515129a1bbe6f068053dbf prerequisite-patch-id: 1e393a196dc7a1ee75f3cc3cebbb591c5422102f prerequisite-patch-id: 2cf497b41f5024ede0a224b1f5b172226067a534 prerequisite-patch-id: 2a70276ed61d33fc4f3b52560753c05d1cd413be prerequisite-patch-id: 17ec40f4388b62ba8bf3ac1546c6913f5d1f6079 prerequisite-patch-id: dba969ce9d6cf69c1319661a7d81b1c1c719804d prerequisite-patch-id: 8d800cda87167314f07320bdb3df936c323e4a40 prerequisite-patch-id: 25d4aaf54ea66f30e426fa38bdd4e0f47303c513 prerequisite-patch-id: 082c9d8584c1daff1e827e44ee3047178e7004a7 prerequisite-patch-id: 0ef73900899425ae2f00751347afdce3739aa954 prerequisite-patch-id: e7db4730b791b71aaf417ee0f65fb6304566aaf8 prerequisite-patch-id: 62d7f28f8196039507ffe362f97723395d7bb704 prerequisite-patch-id: ea8de47bcb54e33bcc67e59e9ed752a4d1fad703 prerequisite-patch-id: 497893ef92e1ea56bd8605e6990a05cb4c7f9293 prerequisite-patch-id: 3dc869c80ee568449bbfa2a9bc427524d0e8970b prerequisite-patch-id: 52c14b6fb14ed4ccd685385a9fbc6297b762c0ef prerequisite-patch-id: 23de8371e9e3277c374a47f9bd10de209a22fdd5 prerequisite-patch-id: d21f036dd106af3375fb920bf0a557fe2b86d98e prerequisite-patch-id: ca717076e9de83d6ce370f7374c4729a9f586fae prerequisite-patch-id: a235b6ab3237155f2b39e8e10d47ddd250f6b6cc prerequisite-patch-id: 6db2aa3d8a5804c85dd171864f5140fadf5f72da prerequisite-patch-id: a799b883f4cb41c34ad074220293f372c6e0a9c7 prerequisite-patch-id: 5e012c426aef7b2f07513cec68e7efa1cf85fe52 prerequisite-patch-id: 4e614e7e3395dda7bae5f9fa21257c57cce45a39 prerequisite-patch-id: 67f8e68622c9698280ff5c5dc7469f36daf9a012 prerequisite-patch-id: d86078331449a21499e3f955e27bc87294969346 prerequisite-patch-id: 3f30d10e0ac7f53307f6b462eaf5b47151b73631 prerequisite-patch-id: a5d84769b776873697b1eb18b6b170a168e68133 prerequisite-patch-id: e5058b3e0ca9f80886bd2c1667349bffc6949da6 prerequisite-patch-id: 669d7e2ef30468034acfa33d4da0404caae526a9 prerequisite-patch-id: 3276a659cbe45efca6a6d3a4579af3cf7ceda992 prerequisite-patch-id: 624657b1f538edfb852b09889d6583970665cad9 prerequisite-patch-id: d7d70d557970f203c288f153361c6782245ba9f9 prerequisite-patch-id: fcbab0e48ac1385d2de1be7852b2adb74f04acfc prerequisite-patch-id: 88e76e8dc51b054c84c497344424f24e877e325b prerequisite-patch-id: dc4c193c8571ae5969a6c498edc7cec8ef438bc1 prerequisite-patch-id: 39c1ee32082f6ce1dd088d5a8340ca9b3e2434a4 prerequisite-patch-id: 28deb0caf0f5eac3fc0a866d2901667a56870f6d prerequisite-patch-id: 53b48b35c9b8217b2da5212d7ed48320d355a943 prerequisite-patch-id: 4469f3663714a65a06b7c472ce9c92ab7502a00f prerequisite-patch-id: a75f08b401d251fc3465b65f94a8db8d32d8900a prerequisite-patch-id: 4b7ac86af078a260f656914089da8f43cdaab8ff prerequisite-patch-id: 398f62abad95f55181d2cf8a383f69000de97af4 prerequisite-patch-id: e102cdc3834018fa88f9fe98a78afb46ff316124 prerequisite-patch-id: 1c30c139ef694b9fd8314bffb4b9b2a02c6c9233 prerequisite-patch-id: d96c304f1c63fa840440508f5b3be70a922b2210 prerequisite-patch-id: a8be25a6b6834ac542d40eaec47a60caf2fe545c prerequisite-patch-id: d29aa52f7a6c5e61cf793addbbc7d7ea5f383725 prerequisite-patch-id: c7a6d171e568f4aba6f94126abc6a9a64caefcc2 prerequisite-patch-id: dcbf2e0e9b3b3cd985423993bdd411fc0cdf4646 prerequisite-patch-id: d5f1a450785cb6e8d74e152b7026621cd4077d04 prerequisite-patch-id: 6179fd2a7c7d6866906365b564fcef05428318b3 prerequisite-patch-id: 3f0e5189f1046e8fd37b2f104a5ee6c306438bed prerequisite-patch-id: 9b8f5520663a347e1981c46d1d72f6b7efeef222 prerequisite-patch-id: 6171b4310abc56d03a21f95848beff3f04c04333 prerequisite-patch-id: 93336ba3fd1e3875b644ec6007ef22480df750a2 prerequisite-patch-id: 9a76b6408f38b96af9ad91269a0fac8d1893bdef prerequisite-patch-id: 74c166a9a1ccafef04f3894085a3b3ed5215df7a prerequisite-patch-id: 71833e48cfa5875247770ac8b94ab957e4eab4fe prerequisite-patch-id: 9a14e6755e7d7605267abaeab62d70e651d622e3 prerequisite-patch-id: e0b2ce3211263007de8e2a2b60402cfea4a2f1c2 prerequisite-patch-id: bacccbe583a69dd0f081aa74822e92c23f344a64 prerequisite-patch-id: bee3183ab6c76123bc8795dfd5dd6eb19d417832 prerequisite-patch-id: eee73769e563ea8ad64f82473c235a4fef18ce83 prerequisite-patch-id: 5c2c95e068dc614a7708fe830d773e03e8cde4ec prerequisite-patch-id: deae58b435dc10d80f1ea754613e9ed3e7bd4fe1 prerequisite-patch-id: 1e6e138aeb18a51b6496a265f84f5df1b98471b6 prerequisite-patch-id: 15865988ed0b88b1e28f0e16c50222f61db5ac35 prerequisite-patch-id: 7884031de035d910c1bddee4331a765c6a266243 prerequisite-patch-id: a2deb3751dec34448ab776b455a7b29bd6d8217f prerequisite-patch-id: d24167f841efc3b8fd8afe06ae02da4d4a594fdc prerequisite-patch-id: 4f23aea0e73aa0933a53db59ed716a19dcbd575c prerequisite-patch-id: f54b42bfeb4241cb96b2490e1261e2bc44e7cc07 prerequisite-patch-id: 7e3f3e34be720b25f3db15d2a878bce23b99353c prerequisite-patch-id: fc91b48c6593c862aa17b221663d2bbed9261f16 prerequisite-patch-id: f20dbadd1962f16e7f4fc4a2b8747443ada94f80 prerequisite-patch-id: 78de9ba010ec43b9ebbc998f4d3afb38fed6be8e prerequisite-patch-id: 013c99f721a9a0a4af86772cb8d1d40f6942f16f prerequisite-patch-id: 7ee4a5b056b343ba2d7491158965023af84f56ad prerequisite-patch-id: d489b42dc81bf9dd7a51381c06182ddffb2230e5 prerequisite-patch-id: fcf1400a8d8b417801baf1450c8c10efa06f4481 prerequisite-patch-id: e475e1e9cd7ce746410ef9839d02ba242cb72090 prerequisite-patch-id: 68d24ec2081c2224a1c133f2c07544177e693a53 prerequisite-patch-id: 6e70864f4065b79db49a206523bcc8b4afded934 prerequisite-patch-id: 0d259c5d3ebb259d36c5d059a82df3b0ffbf1429 prerequisite-patch-id: 392a9fa483070625cad85778e4742c4fc5a2ac5d prerequisite-patch-id: 62804c908c4abe71d644f18303780084a4c224e8 prerequisite-patch-id: 3f3f1eed1a6150df53aba846237329cf24bb9548 prerequisite-patch-id: 4d1387c6e00b9c62e638ce955cd5d9681754bfb5 prerequisite-patch-id: 3382e00119d0ecd997ae6f99855c74e3eed9237c prerequisite-patch-id: 95fcef043070fc902f1cad42ef47a94744624530 prerequisite-patch-id: 5b6737d17f3ef9a4ae5d33343125bc597ad2fa62 prerequisite-patch-id: acc0a1e809cb0639e16d4b05cccfd809f8e299b5 prerequisite-patch-id: aa5c2db287402596e20ef0f935aea67007484f58 prerequisite-patch-id: a0318b66bb3f1f096f7d77c3f03e2018f68430b4 prerequisite-patch-id: 4d709e0d8ac627462de259723ac4fcb8efcc5342 prerequisite-patch-id: 811deca81c0cbdc5a3e977dd282f46427e79dbe2 prerequisite-patch-id: 8f66923ad1790ee9e567b9b1d155c7836c3e7cb8 prerequisite-patch-id: c0f2a71d9835e66a216af2ff3b3d434eec44ab0d prerequisite-patch-id: 5e1a47e077d7bd22b1af9439e1cf1ae38830f763 prerequisite-patch-id: 04bd404d9e3268cbc399458be5bd69190852884a prerequisite-patch-id: 6d630be7173e6e8da9a14739d32701f76bdf32a3 prerequisite-patch-id: 463c4f52f60b2c8fbf165caedbbc1bfa3bd2a265 prerequisite-patch-id: 67403a8553d60ec8938906f28319176f2ef0ecc4 prerequisite-patch-id: 18443a910e260eed010a2d4cd8f255678476eb0a prerequisite-patch-id: cf8d1aab1c343ad9bb6de1b980bf7a8aa7479e79 prerequisite-patch-id: cb8ee956876a21478178d46383e760f2a29ec60a prerequisite-patch-id: 0384e88e9105ce6907f44a55ae0d22bb57b11701 prerequisite-patch-id: 010b9094c8a9b1b41dc49fc07d9704a7be6374c0 prerequisite-patch-id: 3e96fcf9486ad8decca891534d4d2b41f786087a prerequisite-patch-id: d77c491a6a5760ca828ecbb1445a77dcf9bebf0b prerequisite-patch-id: 5c2e9b005429bca543ac6dd39912731d9388eb10 prerequisite-patch-id: 741c7b38494d42817d2d71adb6f267268c9cb208 prerequisite-patch-id: b97fb8b1e5848e8561fc599ef7173b2b07782918 prerequisite-patch-id: cca04ee447a117242679dc946638dd876bbaf931 prerequisite-patch-id: 7e7784bd5ec5ecfdb8d7130975a915fb88e8c26b prerequisite-patch-id: 5a5137d28df7153718ef924d49a821fc1482b220 prerequisite-patch-id: 3d010e879709b5540948b11c2692a28fd5bdd1ea prerequisite-patch-id: 3d418aca0f2a4a5a3e8cbd3c9da2a5229dd26f61 prerequisite-patch-id: 423c72136d6c85888dfac083efb54b518323f0dd prerequisite-patch-id: 099453248d834931415e431e1bb470c1f99cdfa2 prerequisite-patch-id: 9fde172c51e107415194ecce2a3e218d2fab7f60 prerequisite-patch-id: 602a0f6b0121ef0296a3fe14872d59b8a0673b24 prerequisite-patch-id: ded33eb3b89b9da9e7a32ca5c6da3da2b65d3a83 prerequisite-patch-id: 56baaaf053e49e6306c076a80c5e7759a7c27805 prerequisite-patch-id: ec1ace1ef1e30d4c56143b3148b4f70c123ddd85 prerequisite-patch-id: da789135fed934d00e9020a90ec700a273fb18fa prerequisite-patch-id: ca5e644043851df5c7472b4149806ac98b6cce5c prerequisite-patch-id: d4f92797793802ce3021c4d1b45d2d0a94c51f07 prerequisite-patch-id: 992ab3fb9e82572494ff6f8af494ffbe8ba5cc6e prerequisite-patch-id: 8baf34ed9789ac6a26f1d84648869131cea522fa prerequisite-patch-id: 63f23c96115916e7707201d828fcab42ceeea385 prerequisite-patch-id: 7c166511597619b0f03043ce3a0a7237032bf7e0 prerequisite-patch-id: 35e2044bb5d5ea5d041ed01579943f6cffa6ee41 prerequisite-patch-id: 497b25db87c4c64173167f70d3a48154e44ca655 prerequisite-patch-id: d78941852e275e8b8ed03254a89324fef00fcf18 prerequisite-patch-id: 3cc8bd3f0886b6dca9197bfdba4bfa6ff360bf24 prerequisite-patch-id: 61890eaa88bcab4ac0aa2ef9a131d56ef8376f1d prerequisite-patch-id: 097b002021ddd12f7b5180ccaa744ecce925f04b prerequisite-patch-id: f599ecb06efafb8954fc9ad6d08a9819da860d28 prerequisite-patch-id: 484f86ad1f4fb5d8b2debfb1e7a8c7b6d0610f77 prerequisite-patch-id: 52db9f2823a1ad78ffb91f84911cd9963b295465 prerequisite-patch-id: d936469bba0e3720bf957995c0f842e56c815e85 prerequisite-patch-id: 9bfb8fba3a480dd5c8878fd307d96cbaaf521cff prerequisite-patch-id: 13798977298ceb264b789af0eae36c714b0d17c1 prerequisite-patch-id: fc68338d9f90388d062471e46e9b691b00302c69 prerequisite-patch-id: 25e5e3c9ec0a10aa564b3cf32fe793aaf1e705b7 prerequisite-patch-id: 9160fde4ddf465965328264e443638a3ab0d09d9 prerequisite-patch-id: 20d490f27f569e5784d5501d842b4be8a4931127 prerequisite-patch-id: 9b7866c753c962fa8107eeff4d25ff40a546f329 prerequisite-patch-id: 653af78a35e216c7dc891a2d46849bdde83a5dfd prerequisite-patch-id: f27f61687c1f423d4ad8f690a5114644364d26c5 prerequisite-patch-id: 705f5636837961012d4ad7cef93e3c06bf2828ad prerequisite-patch-id: cf18ec6cc3cfb012e68e34f5319e5ca5fc08b04f prerequisite-patch-id: 496d56c7025c252650cef84afef17c1cbe761011 prerequisite-patch-id: 849633dc3d51365926008341078b46e05349daf3 prerequisite-patch-id: bdd1c3731521ab2bec54b5757a1b1460ffa8a76b prerequisite-patch-id: 7a4af5cc7a8d0c73fac182936c3034b8b97a1d81 prerequisite-patch-id: b6640f64140c35ef6614a47ac213d815eaacc385 prerequisite-patch-id: 619f2ef543c43d0e537cc5ec16a88aff8d08f7ee prerequisite-patch-id: 5b7d7481f225dd506c53bc58d913b3dcf670ab3b prerequisite-patch-id: 76b2c41e04d00656cff256f72fb81617baf75004 prerequisite-patch-id: e1e44b6305e28dc040a7495749756d39f086b407 prerequisite-patch-id: 23b6508f200dc43698cc460285ec47fef2e54a45 prerequisite-patch-id: b225a20694d23a03f11eab8c86a0a4b7b640a302 prerequisite-patch-id: e630536287ec238942c1948f8b966e15384fd3ff prerequisite-patch-id: 327ccf1526eb31e0899de28c42c6bc687e274be5 prerequisite-patch-id: 6b4ab77555a60de86ad045538ea19460e0f65a27 prerequisite-patch-id: e9766937f032e867e3953ea53c47a5c385c68feb prerequisite-patch-id: b6f9c8b0ac1722d3809bc48d89ace7070e1c447a prerequisite-patch-id: e72efe42eec650c70433a02ce9c97ddeaef02d0d prerequisite-patch-id: 9f17b908169ba05390869f3e35b69df00bd46d41 prerequisite-patch-id: b9a82d490d17914a2f80b0430cf4973dd93ccd2f prerequisite-patch-id: 2563a1dcb7798936aa2c4a83dc1e1ddd585af241 prerequisite-patch-id: d4e0829e923f60dbf80b39493879280c8da2ff4e prerequisite-patch-id: e42a585260f999d56d222e7b2f44405f6cfbc406 prerequisite-patch-id: e163a8cb28f70824dcde6295354df58a9ece12c0 prerequisite-patch-id: fb1ecde59655a27c8d3fdff20a66cbbe3f00bd6d prerequisite-patch-id: 8cb6ee4b61ec57e6972ac1ee1253e85a7765deaa prerequisite-patch-id: 499ac0cc3c1b5a08c8609ecc7d210c35edf01370 prerequisite-patch-id: 8434161ce61b9b53baf7c2643709e08bede8891d prerequisite-patch-id: bf01608b5710ec0f4d041056988bf42e8785b7be prerequisite-patch-id: 31a846018ddc61b29dffd1c56e8ef29c5aeb164b prerequisite-patch-id: 1a0ab80ff0cde1bf49af90be105764d6504c42e2 prerequisite-patch-id: 4463e631ff73f48de1f761fa08fc2d3a0d02d2e3 prerequisite-patch-id: 1ef269c11829325108dc950ebb6ebbb0413f491d prerequisite-patch-id: 244d96e340cb984ccaeb47ba1fc32c093df85877 prerequisite-patch-id: 6490e0a9923c2341d6aa355576fc178e830a10ef prerequisite-patch-id: 73099e3b819e51e70cedf57b5de2da94161a789b prerequisite-patch-id: 49c793919ad92fe3162b6cc8efac6762037534e1 prerequisite-patch-id: 53796f63653ecdf3f2a66d9cb3181546441d4ac0 prerequisite-patch-id: aa6fbd949a032b6fce2b599309f633393bc0bb08 prerequisite-patch-id: eb33de06c18bd23ef820d12cd5e340eda4fc15dc prerequisite-patch-id: 9acaaae21935bd460bd45f89ac9e5b3c7be398fc prerequisite-patch-id: d00c2306f14a27317918b63539f65df88ee08222 prerequisite-patch-id: 87cb389190ca57ff8250be430e3470b39575271d prerequisite-patch-id: 2eaee2a99c51fd5a0df72b5253e8064d9facbcd0 prerequisite-patch-id: fa319f93ccd1f3f07955a69b793d0c3ca697fe56 prerequisite-patch-id: 8541186582e59ab7fc5b8242042d068b5ff344ec prerequisite-patch-id: 02406627f62825a368975d64a3da2c68fafeccea prerequisite-patch-id: 5ff0c64a0f40ad9c17a67f3ec4af3c24ca81ba7c prerequisite-patch-id: 7f2ccb5fa7573dd10b4dec4d9b3dda8c5416e77e prerequisite-patch-id: f32307884ffd5b79bb333fffbf7a1a1fb11e1077 prerequisite-patch-id: dc6d9a25a341dbd7ec34453b5692f0c4ee0e5ebd prerequisite-patch-id: fdc3da57663352fd762b6ea8b3fb39545cd86f69 prerequisite-patch-id: 93933d68c7698bf5c66aaff4b9e61b8a7cd49deb prerequisite-patch-id: 601286c6aae0c275b6ebc489197c21f6425d1dea prerequisite-patch-id: b6bd48dfa8f1d956be0a63ccc27e0f2648ea6188 prerequisite-patch-id: d7bc5f0deb3489e698b5d93d756bb41fe334229e prerequisite-patch-id: 3f3d8b03dc8e39e30049f94c234fb32333bea7ab prerequisite-patch-id: e7e5398f510b2490efec81bc227ab3a66fda6db0 prerequisite-patch-id: db92723389041e01bed4d3cab7d800bd0060d44d prerequisite-patch-id: 7ba892dca22bd88cb2b4f03e9c7ec3ab4828f365 prerequisite-patch-id: 56aac78c79b3a4329e7014d173c8f6edea4ebee1 prerequisite-patch-id: 10e17f897a2bb483d2cd0038598e5e2f08514e4a prerequisite-patch-id: dfb16a3bd2e302f02dd89fd4337446b454100ed4 prerequisite-patch-id: 145f9e823ba743c9bf1c3f9abcdd1d4648a82d53 prerequisite-patch-id: f7152653caf4f759f0537fc8f7898261e839261a prerequisite-patch-id: e727e63dd83f98fc829b6b74a11d7f936950aa14 prerequisite-patch-id: c5e20e303c45ddc096195a9d7c5c2cc2432f0290 prerequisite-patch-id: f448aa53083685570eb21b735641430a91610921 -- 2.40.1 From: Fabiano Rosas <farosas@suse.de> We're about to add more functions to this file so make it use the same coding style as the rest of the code. Signed-off-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Peter Xu <peterx@redhat.com> Message-Id: <20230607161306.31425-2-farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/threadinfo.h | 5 ++--- migration/migration.c | 4 ++-- migration/multifd.c | 4 ++-- migration/threadinfo.c | 4 ++-- 4 files changed, 8 insertions(+), 9 deletions(-) diff --git a/migration/threadinfo.h b/migration/threadinfo.h index XXXXXXX..XXXXXXX 100644 --- a/migration/threadinfo.h +++ b/migration/threadinfo.h @@ -XXX,XX +XXX,XX @@ struct MigrationThread { QLIST_ENTRY(MigrationThread) node; }; -MigrationThread *MigrationThreadAdd(const char *name, int thread_id); - -void MigrationThreadDel(MigrationThread *info); +MigrationThread *migration_threads_add(const char *name, int thread_id); +void migration_threads_remove(MigrationThread *info); diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ static void *migration_thread(void *opaque) MigThrError thr_error; bool urgent = false; - thread = MigrationThreadAdd("live_migration", qemu_get_thread_id()); + thread = migration_threads_add("live_migration", qemu_get_thread_id()); rcu_register_thread(); @@ -XXX,XX +XXX,XX @@ static void *migration_thread(void *opaque) migration_iteration_finish(s); object_unref(OBJECT(s)); rcu_unregister_thread(); - MigrationThreadDel(thread); + migration_threads_remove(thread); return NULL; } diff --git a/migration/multifd.c b/migration/multifd.c index XXXXXXX..XXXXXXX 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque) int ret = 0; bool use_zero_copy_send = migrate_zero_copy_send(); - thread = MigrationThreadAdd(p->name, qemu_get_thread_id()); + thread = migration_threads_add(p->name, qemu_get_thread_id()); trace_multifd_send_thread_start(p->id); rcu_register_thread(); @@ -XXX,XX +XXX,XX @@ out: qemu_mutex_unlock(&p->mutex); rcu_unregister_thread(); - MigrationThreadDel(thread); + migration_threads_remove(thread); trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages); return NULL; diff --git a/migration/threadinfo.c b/migration/threadinfo.c index XXXXXXX..XXXXXXX 100644 --- a/migration/threadinfo.c +++ b/migration/threadinfo.c @@ -XXX,XX +XXX,XX @@ static QLIST_HEAD(, MigrationThread) migration_threads; -MigrationThread *MigrationThreadAdd(const char *name, int thread_id) +MigrationThread *migration_threads_add(const char *name, int thread_id) { MigrationThread *thread = g_new0(MigrationThread, 1); thread->name = name; @@ -XXX,XX +XXX,XX @@ MigrationThread *MigrationThreadAdd(const char *name, int thread_id) return thread; } -void MigrationThreadDel(MigrationThread *thread) +void migration_threads_remove(MigrationThread *thread) { if (thread) { QLIST_REMOVE(thread, node); -- 2.40.1 From: Fabiano Rosas <farosas@suse.de> This doubly linked list is common for all the multifd and migration threads so we need to avoid concurrent access. Add a mutex to protect the data from concurrent access. This fixes a crash when removing two MigrationThread objects from the list at the same time during cleanup of multifd threads. Fixes: 671326201d ("migration: Introduce interface query-migrationthreads") Signed-off-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <20230607161306.31425-3-farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/threadinfo.h | 2 -- migration/threadinfo.c | 15 ++++++++++++++- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/migration/threadinfo.h b/migration/threadinfo.h index XXXXXXX..XXXXXXX 100644 --- a/migration/threadinfo.h +++ b/migration/threadinfo.h @@ -XXX,XX +XXX,XX @@ * See the COPYING file in the top-level directory. */ -#include "qemu/queue.h" -#include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/qapi-commands-migration.h" diff --git a/migration/threadinfo.c b/migration/threadinfo.c index XXXXXXX..XXXXXXX 100644 --- a/migration/threadinfo.c +++ b/migration/threadinfo.c @@ -XXX,XX +XXX,XX @@ * See the COPYING file in the top-level directory. */ +#include "qemu/osdep.h" +#include "qemu/queue.h" +#include "qemu/lockable.h" #include "threadinfo.h" +QemuMutex migration_threads_lock; static QLIST_HEAD(, MigrationThread) migration_threads; +static void __attribute__((constructor)) migration_threads_init(void) +{ + qemu_mutex_init(&migration_threads_lock); +} + MigrationThread *migration_threads_add(const char *name, int thread_id) { MigrationThread *thread = g_new0(MigrationThread, 1); thread->name = name; thread->thread_id = thread_id; - QLIST_INSERT_HEAD(&migration_threads, thread, node); + WITH_QEMU_LOCK_GUARD(&migration_threads_lock) { + QLIST_INSERT_HEAD(&migration_threads, thread, node); + } return thread; } void migration_threads_remove(MigrationThread *thread) { + QEMU_LOCK_GUARD(&migration_threads_lock); if (thread) { QLIST_REMOVE(thread, node); g_free(thread); @@ -XXX,XX +XXX,XX @@ MigrationThreadInfoList *qmp_query_migrationthreads(Error **errp) MigrationThreadInfoList **tail = &head; MigrationThread *thread = NULL; + QEMU_LOCK_GUARD(&migration_threads_lock); QLIST_FOREACH(thread, &migration_threads, node) { MigrationThreadInfo *info = g_new0(MigrationThreadInfo, 1); info->name = g_strdup(thread->name); -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> dirty_rate paraemter of hmp command "set_vcpu_dirty_limit" is invalid if less than 0, so add parameter check for it. Note that this patch also delete the unsolicited help message and clean up the code. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <168618975839.6361.17407633874747688653-1@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- softmmu/dirtylimit.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index XXXXXXX..XXXXXXX 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -XXX,XX +XXX,XX @@ void hmp_set_vcpu_dirty_limit(Monitor *mon, const QDict *qdict) int64_t cpu_index = qdict_get_try_int(qdict, "cpu_index", -1); Error *err = NULL; + if (dirty_rate < 0) { + error_setg(&err, "invalid dirty page limit %" PRId64, dirty_rate); + goto out; + } + qmp_set_vcpu_dirty_limit(!!(cpu_index != -1), cpu_index, dirty_rate, &err); - if (err) { - hmp_handle_error(mon, err); - return; - } - monitor_printf(mon, "[Please use 'info vcpu_dirty_limit' to query " - "dirty limit for virtual CPU]\n"); +out: + hmp_handle_error(mon, err); } static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index) -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is in the range of 1 to 1000ms and used to make dirtyrate calculation period configurable. Currently with the "x-vcpu-dirty-limit-period" varies, the total time of live migration changes, test results show the optimal value of "x-vcpu-dirty-limit-period" ranges from 500ms to 1000 ms. "x-vcpu-dirty-limit-period" should be made stable once it proves best value can not be determined with developer's experiments. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <168618975839.6361.17407633874747688653-2@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- qapi/migration.json | 34 +++++++++++++++++++++++++++------- migration/migration-hmp-cmds.c | 8 ++++++++ migration/options.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 7 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -XXX,XX +XXX,XX @@ 'multifd-channels', 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', - 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'multifd-zlib-level', 'multifd-zstd-level', + 'block-bitmap-mapping', + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } ## # @MigrateSetParameters: @@ -XXX,XX +XXX,XX @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # TODO: either fuse back into MigrationParameters, or make # MigrationParameters members mandatory @@ -XXX,XX +XXX,XX @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @migrate-set-parameters: @@ -XXX,XX +XXX,XX @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty limit during +# live migration. Should be in the range 1 to 1000ms, +# defaults to 1000ms. (Since 8.1) +# # Features: # -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -XXX,XX +XXX,XX @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } } } ## # @query-migrate-parameters: diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) } } } + + monitor_printf(mon, "%s: %" PRIu64 " ms\n", + MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD), + params->x_vcpu_dirty_limit_period); } qapi_free_MigrationParameters(params); @@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) error_setg(&err, "The block-bitmap-mapping parameter can only be set " "through QMP"); break; + case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD: + p->has_x_vcpu_dirty_limit_period = true; + visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); + break; default: assert(0); } diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ #define DEFINE_PROP_MIG_CAP(name, x) \ DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false) +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ + Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, store_global_state, true), @@ -XXX,XX +XXX,XX @@ Property migration_properties[] = { DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds), DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_hostname), DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), + DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, + parameters.x_vcpu_dirty_limit_period, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) s->parameters.block_bitmap_mapping); } + params->has_x_vcpu_dirty_limit_period = true; + params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; + return params; } @@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params) params->has_announce_max = true; params->has_announce_rounds = true; params->has_announce_step = true; + params->has_x_vcpu_dirty_limit_period = true; } /* @@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp) } #endif + if (params->has_x_vcpu_dirty_limit_period && + (params->x_vcpu_dirty_limit_period < 1 || + params->x_vcpu_dirty_limit_period > 1000)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "x-vcpu-dirty-limit-period", + "a value between 1 and 1000"); + return false; + } + return true; } @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->has_block_bitmap_mapping = true; dest->block_bitmap_mapping = params->block_bitmap_mapping; } + + if (params->has_x_vcpu_dirty_limit_period) { + dest->x_vcpu_dirty_limit_period = + params->x_vcpu_dirty_limit_period; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) QAPI_CLONE(BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } + + if (params->has_x_vcpu_dirty_limit_period) { + s->parameters.x_vcpu_dirty_limit_period = + params->x_vcpu_dirty_limit_period; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration. "vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters. This two parameters are used to help implement the dirty page rate limit algo of migration. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Acked-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <168618975839.6361.17407633874747688653-3@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- qapi/migration.json | 18 +++++++++++++++--- migration/migration-hmp-cmds.c | 8 ++++++++ migration/options.c | 21 +++++++++++++++++++++ 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -XXX,XX +XXX,XX @@ 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', 'block-bitmap-mapping', - { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] } ] } + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, + 'vcpu-dirty-limit'] } ## # @MigrateSetParameters: @@ -XXX,XX +XXX,XX @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -XXX,XX +XXX,XX @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @migrate-set-parameters: @@ -XXX,XX +XXX,XX @@ # live migration. Should be in the range 1 to 1000ms, # defaults to 1000ms. (Since 8.1) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period @@ -XXX,XX +XXX,XX @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } } } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } ## # @query-migrate-parameters: diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) monitor_printf(mon, "%s: %" PRIu64 " ms\n", MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD), params->x_vcpu_dirty_limit_period); + + monitor_printf(mon, "%s: %" PRIu64 " MB/s\n", + MigrationParameter_str(MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT), + params->vcpu_dirty_limit); } qapi_free_MigrationParameters(params); @@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_x_vcpu_dirty_limit_period = true; visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); break; + case MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT: + p->has_vcpu_dirty_limit = true; + visit_type_size(v, param, &p->vcpu_dirty_limit, &err); + break; default: assert(0); } diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false) #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, @@ -XXX,XX +XXX,XX @@ Property migration_properties[] = { DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, parameters.x_vcpu_dirty_limit_period, DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), + DEFINE_PROP_UINT64("vcpu-dirty-limit", MigrationState, + parameters.vcpu_dirty_limit, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_x_vcpu_dirty_limit_period = true; params->x_vcpu_dirty_limit_period = s->parameters.x_vcpu_dirty_limit_period; + params->has_vcpu_dirty_limit = true; + params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit; return params; } @@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params) params->has_announce_rounds = true; params->has_announce_step = true; params->has_x_vcpu_dirty_limit_period = true; + params->has_vcpu_dirty_limit = true; } /* @@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp) return false; } + if (params->has_vcpu_dirty_limit && + (params->vcpu_dirty_limit < 1)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "vcpu_dirty_limit", + "is invalid, it must greater then 1 MB/s"); + return false; + } + return true; } @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } + if (params->has_vcpu_dirty_limit) { + dest->vcpu_dirty_limit = params->vcpu_dirty_limit; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) s->parameters.x_vcpu_dirty_limit_period = params->x_vcpu_dirty_limit_period; } + if (params->has_vcpu_dirty_limit) { + s->parameters.vcpu_dirty_limit = params->vcpu_dirty_limit; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce migration dirty-limit capability, which can be turned on before live migration and limit dirty page rate durty live migration. Introduce migrate_dirty_limit function to help check if dirty-limit capability enabled during live migration. Meanwhile, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcoded. dirty-limit capability is kind of like auto-converge but using dirty limit instead of traditional cpu-throttle to throttle guest down. To enable this feature, turn on the dirty-limit capability before live migration using migrate-set-capabilities, and set the parameters "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably to speed up convergence. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Acked-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <168618975839.6361.17407633874747688653-4@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- qapi/migration.json | 12 +++++++++++- migration/options.h | 1 + migration/options.c | 23 +++++++++++++++++++++++ softmmu/dirtylimit.c | 18 ++++++++++++++---- 4 files changed, 49 insertions(+), 5 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # and should not affect the correctness of postcopy migration. # (since 7.1) # +# @dirty-limit: If enabled, migration will use the dirty-limit algo to +# throttle down guest instead of auto-converge algo. +# Throttle algo only works when vCPU's dirtyrate greater +# than 'vcpu-dirty-limit', read processes in guest os +# aren't penalized any more, so this algo can improve +# performance of vCPU during live migration. This is an +# optional performance feature and should not affect the +# correctness of the existing auto-converge algo. +# (since 8.1) +# # Features: # # @unstable: Members @x-colo and @x-ignore-shared are experimental. @@ -XXX,XX +XXX,XX @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt'] } + 'zero-copy-send', 'postcopy-preempt', 'dirty-limit'] } ## # @MigrationCapabilityStatus: diff --git a/migration/options.h b/migration/options.h index XXXXXXX..XXXXXXX 100644 --- a/migration/options.h +++ b/migration/options.h @@ -XXX,XX +XXX,XX @@ bool migrate_block(void); bool migrate_colo(void); bool migrate_compress(void); bool migrate_dirty_bitmaps(void); +bool migrate_dirty_limit(void); bool migrate_events(void); bool migrate_ignore_shared(void); bool migrate_late_block_activate(void); diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ #include "qemu-file.h" #include "ram.h" #include "options.h" +#include "sysemu/kvm.h" /* Maximum migrate downtime set to 2000 seconds */ #define MAX_MIGRATE_DOWNTIME_SECONDS 2000 @@ -XXX,XX +XXX,XX @@ Property migration_properties[] = { DEFINE_PROP_MIG_CAP("x-zero-copy-send", MIGRATION_CAPABILITY_ZERO_COPY_SEND), #endif + DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT), DEFINE_PROP_END_OF_LIST(), }; @@ -XXX,XX +XXX,XX @@ bool migrate_dirty_bitmaps(void) return s->capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS]; } +bool migrate_dirty_limit(void) +{ + MigrationState *s = migrate_get_current(); + + return s->capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT]; +} + bool migrate_events(void) { MigrationState *s = migrate_get_current(); @@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } } + if (new_caps[MIGRATION_CAPABILITY_DIRTY_LIMIT]) { + if (new_caps[MIGRATION_CAPABILITY_AUTO_CONVERGE]) { + error_setg(errp, "dirty-limit conflicts with auto-converge" + " either of then available currently"); + return false; + } + + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty-limit requires KVM with accelerator" + " property 'dirty-ring-size' set"); + return false; + } + } + return true; } diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index XXXXXXX..XXXXXXX 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -XXX,XX +XXX,XX @@ #include "hw/boards.h" #include "sysemu/kvm.h" #include "trace.h" +#include "migration/misc.h" +#include "migration/migration.h" +#include "migration/options.h" /* * Dirtylimit stop working if dirty page rate error @@ -XXX,XX +XXX,XX @@ static bool dirtylimit_quit; static void vcpu_dirty_rate_stat_collect(void) { + MigrationState *s = migrate_get_current(); VcpuStat stat; int i = 0; + int64_t period = DIRTYLIMIT_CALC_TIME_MS; + + if (migrate_dirty_limit() && + migration_is_active(s)) { + period = s->parameters.x_vcpu_dirty_limit_period; + } /* calculate vcpu dirtyrate */ - vcpu_calculate_dirtyrate(DIRTYLIMIT_CALC_TIME_MS, - &stat, - GLOBAL_DIRTY_LIMIT, - false); + vcpu_calculate_dirtyrate(period, + &stat, + GLOBAL_DIRTY_LIMIT, + false); for (i = 0; i < stat.nvcpu; i++) { vcpu_dirty_rate_stat->stat.rates[i].id = i; -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Check if block migration is running before throttling guest down in auto-converge way. Note that this modification is kind of like code clean, because block migration does not depend on auto-converge capability, so the order of checks can be adjusted. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Acked-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-Id: <168618975839.6361.17407633874747688653-5@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/ram.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ static void migration_trigger_throttle(RAMState *rs) /* During block migration the auto-converge logic incorrectly detects * that ram migration makes no progress. Avoid this by disabling the * throttling logic during the bulk phase of block migration. */ - if (migrate_auto_converge() && !blk_mig_bulk_active()) { + if (blk_mig_bulk_active()) { + return; + } + + if (migrate_auto_converge()) { /* The following detection logic can be refined later. For now: Check to see if the ratio between dirtied bytes and the approx. amount of bytes that just got transferred since the last time -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> This commit is prepared for the implementation of dirty-limit convergence algo. The detection logic of throttling condition can apply to both auto-converge and dirty-limit algo, putting it's position before the checking logic for auto-converge feature. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-ID: <168733225273.5845.15871826788879741674-6@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/ram.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ static void migration_trigger_throttle(RAMState *rs) return; } - if (migrate_auto_converge()) { - /* The following detection logic can be refined later. For now: - Check to see if the ratio between dirtied bytes and the approx. - amount of bytes that just got transferred since the last time - we were in this routine reaches the threshold. If that happens - twice, start or increase throttling. */ - - if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >= 2)) { + /* + * The following detection logic can be refined later. For now: + * Check to see if the ratio between dirtied bytes and the approx. + * amount of bytes that just got transferred since the last time + * we were in this routine reaches the threshold. If that happens + * twice, start or increase throttling. + */ + if ((bytes_dirty_period > bytes_dirty_threshold) && + (++rs->dirty_rate_high_cnt >= 2)) { + rs->dirty_rate_high_cnt = 0; + if (migrate_auto_converge()) { trace_migration_throttle(); - rs->dirty_rate_high_cnt = 0; mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); } -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent. Enable dirty page limit if dirty_rate_high_cnt greater than 2 when dirty-limit capability enabled, Disable dirty-limit if migration be canceled. Note that "set_vcpu_dirty_limit", "cancel_vcpu_dirty_limit" commands are not allowed during dirty-limit live migration. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-ID: <168733225273.5845.15871826788879741674-7@git.sr.ht> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/migration.c | 3 +++ migration/ram.c | 36 ++++++++++++++++++++++++++++++++++++ softmmu/dirtylimit.c | 29 +++++++++++++++++++++++++++++ migration/trace-events | 1 + 4 files changed, 69 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ void migration_cancel(const Error *error) if (error) { migrate_set_error(current_migration, error); } + if (migrate_dirty_limit()) { + qmp_cancel_vcpu_dirty_limit(false, -1, NULL); + } migrate_fd_cancel(current_migration); } diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ #include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h" @@ -XXX,XX +XXX,XX @@ #include "multifd.h" #include "sysemu/runstate.h" #include "options.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" #include "hw/boards.h" /* for machine_dump_guest_core() */ @@ -XXX,XX +XXX,XX @@ static void migration_update_rates(RAMState *rs, int64_t end_time) } } +/* + * Enable dirty-limit to throttle down the guest + */ +static void migration_dirty_limit_guest(void) +{ + /* + * dirty page rate quota for all vCPUs fetched from + * migration parameter 'vcpu_dirty_limit' + */ + static int64_t quota_dirtyrate; + MigrationState *s = migrate_get_current(); + + /* + * If dirty limit already enabled and migration parameter + * vcpu-dirty-limit untouched. + */ + if (dirtylimit_in_service() && + quota_dirtyrate == s->parameters.vcpu_dirty_limit) { + return; + } + + quota_dirtyrate = s->parameters.vcpu_dirty_limit; + + /* + * Set all vCPU a quota dirtyrate, note that the second + * parameter will be ignored if setting all vCPU for the vm + */ + qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL); + trace_migration_dirty_limit_guest(quota_dirtyrate); +} + static void migration_trigger_throttle(RAMState *rs) { uint64_t threshold = migrate_throttle_trigger_threshold(); @@ -XXX,XX +XXX,XX @@ static void migration_trigger_throttle(RAMState *rs) trace_migration_throttle(); mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); + } else if (migrate_dirty_limit()) { + migration_dirty_limit_guest(); } } } diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index XXXXXXX..XXXXXXX 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -XXX,XX +XXX,XX @@ static void dirtylimit_cleanup(void) dirtylimit_state_finalize(); } +/* + * dirty page rate limit is not allowed to set if migration + * is running with dirty-limit capability enabled. + */ +static bool dirtylimit_is_allowed(void) +{ + MigrationState *ms = migrate_get_current(); + + if (migration_is_running(ms->state) && + (!qemu_thread_is_self(&ms->thread)) && + migrate_dirty_limit() && + dirtylimit_in_service()) { + return false; + } + return true; +} + void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, int64_t cpu_index, Error **errp) @@ -XXX,XX +XXX,XX @@ void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, return; } + if (!dirtylimit_is_allowed()) { + error_setg(errp, "can't cancel dirty page rate limit while" + " migration is running"); + return; + } + if (!dirtylimit_in_service()) { return; } @@ -XXX,XX +XXX,XX @@ void qmp_set_vcpu_dirty_limit(bool has_cpu_index, return; } + if (!dirtylimit_is_allowed()) { + error_setg(errp, "can't set dirty page rate limit while" + " migration is running"); + return; + } + if (!dirty_rate) { qmp_cancel_vcpu_dirty_limit(has_cpu_index, cpu_index, errp); return; diff --git a/migration/trace-events b/migration/trace-events index XXXXXXX..XXXXXXX 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -XXX,XX +XXX,XX @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" migration_throttle(void) "" +migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=%d addr=0x%" PRIx64 " flags=0x%x" -- 2.40.1 From: Hyman Huang(黄勇) <yong.huang@smartx.com> Extend query-migrate to provide throttle time and estimated ring full time with dirty-limit capability enabled, through which we can observe if dirty limit take effect during live migration. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-ID: <168733225273.5845.15871826788879741674-8@git.sr.ht> Signed-off-by: Juan Quintela <quintela@redhat.com> --- qapi/migration.json | 16 +++++++++++++- include/sysemu/dirtylimit.h | 2 ++ migration/migration-hmp-cmds.c | 10 +++++++++ migration/migration.c | 10 +++++++++ softmmu/dirtylimit.c | 39 ++++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 1 deletion(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # blocked. Present and non-empty when migration is blocked. # (since 6.0) # +# @dirty-limit-throttle-time-per-round: Maximum throttle time (in microseconds) of virtual +# CPUs each dirty ring full round, which shows how +# MigrationCapability dirty-limit affects the guest +# during live migration. (since 8.1) +# +# @dirty-limit-ring-full-time: Estimated average dirty ring full time (in microseconds) +# each dirty ring full round, note that the value equals +# dirty ring memory size divided by average dirty page rate +# of virtual CPU, which can be used to observe the average +# memory load of virtual CPU indirectly. Note that zero +# means guest doesn't dirty memory (since 8.1) +# # Since: 0.14 ## { 'struct': 'MigrationInfo', @@ -XXX,XX +XXX,XX @@ '*postcopy-blocktime' : 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], '*compression': 'CompressionStats', - '*socket-address': ['SocketAddress'] } } + '*socket-address': ['SocketAddress'], + '*dirty-limit-throttle-time-per-round': 'uint64', + '*dirty-limit-ring-full-time': 'uint64'} } ## # @query-migrate: diff --git a/include/sysemu/dirtylimit.h b/include/sysemu/dirtylimit.h index XXXXXXX..XXXXXXX 100644 --- a/include/sysemu/dirtylimit.h +++ b/include/sysemu/dirtylimit.h @@ -XXX,XX +XXX,XX @@ void dirtylimit_set_vcpu(int cpu_index, void dirtylimit_set_all(uint64_t quota, bool enable); void dirtylimit_vcpu_execute(CPUState *cpu); +uint64_t dirtylimit_throttle_time_per_round(void); +uint64_t dirtylimit_ring_full_time(void); #endif diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->cpu_throttle_percentage); } + if (info->has_dirty_limit_throttle_time_per_round) { + monitor_printf(mon, "dirty-limit throttle time: %" PRIu64 " us\n", + info->dirty_limit_throttle_time_per_round); + } + + if (info->has_dirty_limit_ring_full_time) { + monitor_printf(mon, "dirty-limit ring full time: %" PRIu64 " us\n", + info->dirty_limit_ring_full_time); + } + if (info->has_postcopy_blocktime) { monitor_printf(mon, "postcopy blocktime: %u\n", info->postcopy_blocktime); diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ #include "yank_functions.h" #include "sysemu/qtest.h" #include "options.h" +#include "sysemu/dirtylimit.h" static NotifierList migration_state_notifiers = NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); @@ -XXX,XX +XXX,XX @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->dirty_pages_rate = stat64_get(&mig_stats.dirty_pages_rate); } + + if (migrate_dirty_limit() && dirtylimit_in_service()) { + info->has_dirty_limit_throttle_time_per_round = true; + info->dirty_limit_throttle_time_per_round = + dirtylimit_throttle_time_per_round(); + + info->has_dirty_limit_ring_full_time = true; + info->dirty_limit_ring_full_time = dirtylimit_ring_full_time(); + } } static void populate_disk_info(MigrationInfo *info) diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index XXXXXXX..XXXXXXX 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -XXX,XX +XXX,XX @@ out: hmp_handle_error(mon, err); } +/* Return the max throttle time of each virtual CPU */ +uint64_t dirtylimit_throttle_time_per_round(void) +{ + CPUState *cpu; + int64_t max = 0; + + CPU_FOREACH(cpu) { + if (cpu->throttle_us_per_full > max) { + max = cpu->throttle_us_per_full; + } + } + + return max; +} + +/* + * Estimate average dirty ring full time of each virtaul CPU. + * Return 0 if guest doesn't dirty memory. + */ +uint64_t dirtylimit_ring_full_time(void) +{ + CPUState *cpu; + uint64_t curr_rate = 0; + int nvcpus = 0; + + CPU_FOREACH(cpu) { + if (cpu->running) { + nvcpus++; + curr_rate += vcpu_dirty_rate_get(cpu->cpu_index); + } + } + + if (!curr_rate || !nvcpus) { + return 0; + } + + return dirtylimit_dirty_ring_full_time(curr_rate / nvcpus); +} + static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index) { DirtyLimitInfo *info = NULL; -- 2.40.1 It makes no sense that we don't have the same configuration on both sides. Reviewed-by: Laurent Vivier <lvivier@redhat.com> Message-ID: <20230608224943.3877-2-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "'nvramrc=hex .\" _\" begin %x %x " "do i c@ 1 + i c! 1000 +loop .\" B\" 0 " "until'", end_address, start_address); - arch_target = g_strdup(""); + arch_target = g_strdup("-nodefaults"); } else if (strcmp(arch, "aarch64") == 0) { init_bootfile(bootpath, aarch64_kernel, sizeof(aarch64_kernel)); machine_opts = "virt,gic-version=max"; -- 2.40.1 Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Message-ID: <20230608224943.3877-5-quintela@redhat.com> --- tests/qtest/migration-test.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, start_address = S390_TEST_MEM_START; end_address = S390_TEST_MEM_END; } else if (strcmp(arch, "ppc64") == 0) { - machine_opts = "vsmt=8"; + machine_opts = "-machine vsmt=8"; memory_size = "256M"; start_address = PPC_TEST_MEM_START; end_address = PPC_TEST_MEM_END; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, arch_target = g_strdup("-nodefaults"); } else if (strcmp(arch, "aarch64") == 0) { init_bootfile(bootpath, aarch64_kernel, sizeof(aarch64_kernel)); - machine_opts = "virt,gic-version=max"; + machine_opts = "-machine virt,gic-version=max"; memory_size = "150M"; arch_source = g_strdup_printf("-cpu max " "-kernel %s", @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, shmem_opts = g_strdup(""); } - cmd_source = g_strdup_printf("-accel kvm%s -accel tcg%s%s " + cmd_source = g_strdup_printf("-accel kvm%s -accel tcg %s " "-name source,debug-threads=on " "-m %s " "-serial file:%s/src_serial " "%s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", - machine_opts ? " -machine " : "", machine_opts ? machine_opts : "", memory_size, tmpfs, arch_source, shmem_opts, @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, &got_src_stop); } - cmd_target = g_strdup_printf("-accel kvm%s -accel tcg%s%s " + cmd_target = g_strdup_printf("-accel kvm%s -accel tcg %s " "-name target,debug-threads=on " "-m %s " "-serial file:%s/dest_serial " @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "%s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", - machine_opts ? " -machine " : "", machine_opts ? machine_opts : "", memory_size, tmpfs, uri, arch_target, shmem_opts, -- 2.40.1 This will contain the options needed for both source and target. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-6-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, { g_autofree gchar *arch_source = NULL; g_autofree gchar *arch_target = NULL; + /* options for source and target */ + g_autofree gchar *arch_opts = NULL; g_autofree gchar *cmd_source = NULL; g_autofree gchar *cmd_target = NULL; const gchar *ignore_stderr; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, assert(sizeof(x86_bootsect) == 512); init_bootfile(bootpath, x86_bootsect, sizeof(x86_bootsect)); memory_size = "150M"; - arch_source = g_strdup_printf("-drive file=%s,format=raw", bootpath); - arch_target = g_strdup(arch_source); + arch_opts = g_strdup_printf("-drive file=%s,format=raw", bootpath); start_address = X86_TEST_MEM_START; end_address = X86_TEST_MEM_END; } else if (g_str_equal(arch, "s390x")) { init_bootfile(bootpath, s390x_elf, sizeof(s390x_elf)); memory_size = "128M"; - arch_source = g_strdup_printf("-bios %s", bootpath); - arch_target = g_strdup(arch_source); + arch_opts = g_strdup_printf("-bios %s", bootpath); start_address = S390_TEST_MEM_START; end_address = S390_TEST_MEM_END; } else if (strcmp(arch, "ppc64") == 0) { @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, memory_size = "256M"; start_address = PPC_TEST_MEM_START; end_address = PPC_TEST_MEM_END; - arch_source = g_strdup_printf("-nodefaults " - "-prom-env 'use-nvramrc?=true' -prom-env " + arch_source = g_strdup_printf("-prom-env 'use-nvramrc?=true' -prom-env " "'nvramrc=hex .\" _\" begin %x %x " "do i c@ 1 + i c! 1000 +loop .\" B\" 0 " "until'", end_address, start_address); - arch_target = g_strdup("-nodefaults"); + arch_opts = g_strdup("-nodefaults"); } else if (strcmp(arch, "aarch64") == 0) { init_bootfile(bootpath, aarch64_kernel, sizeof(aarch64_kernel)); machine_opts = "-machine virt,gic-version=max"; memory_size = "150M"; - arch_source = g_strdup_printf("-cpu max " - "-kernel %s", - bootpath); - arch_target = g_strdup(arch_source); + arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath); start_address = ARM_TEST_MEM_START; end_address = ARM_TEST_MEM_END; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "-name source,debug-threads=on " "-m %s " "-serial file:%s/src_serial " - "%s %s %s %s", + "%s %s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", machine_opts ? machine_opts : "", memory_size, tmpfs, - arch_source, shmem_opts, + arch_opts ? arch_opts : "", + arch_source ? arch_source : "", + shmem_opts, args->opts_source ? args->opts_source : "", ignore_stderr); if (!args->only_target) { @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "-m %s " "-serial file:%s/dest_serial " "-incoming %s " - "%s %s %s %s", + "%s %s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", machine_opts ? machine_opts : "", memory_size, tmpfs, uri, - arch_target, shmem_opts, + arch_opts ? arch_opts : "", + arch_target ? arch_target : "", + shmem_opts, args->opts_target ? args->opts_target : "", ignore_stderr); *to = qtest_init(cmd_target); -- 2.40.1 And it needs to be in both source and target, so put it on arch_opts. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-7-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, g_autofree char *shmem_opts = NULL; g_autofree char *shmem_path = NULL; const char *arch = qtest_get_arch(); - const char *machine_opts = NULL; const char *memory_size; if (args->use_shmem) { @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, start_address = S390_TEST_MEM_START; end_address = S390_TEST_MEM_END; } else if (strcmp(arch, "ppc64") == 0) { - machine_opts = "-machine vsmt=8"; memory_size = "256M"; start_address = PPC_TEST_MEM_START; end_address = PPC_TEST_MEM_END; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "'nvramrc=hex .\" _\" begin %x %x " "do i c@ 1 + i c! 1000 +loop .\" B\" 0 " "until'", end_address, start_address); - arch_opts = g_strdup("-nodefaults"); + arch_opts = g_strdup("-nodefaults -machine vsmt=8"); } else if (strcmp(arch, "aarch64") == 0) { init_bootfile(bootpath, aarch64_kernel, sizeof(aarch64_kernel)); - machine_opts = "-machine virt,gic-version=max"; memory_size = "150M"; - arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath); + arch_opts = g_strdup_printf("-machine virt,gic-version=max -cpu max " + "-kernel %s", bootpath); start_address = ARM_TEST_MEM_START; end_address = ARM_TEST_MEM_END; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, shmem_opts = g_strdup(""); } - cmd_source = g_strdup_printf("-accel kvm%s -accel tcg %s " + cmd_source = g_strdup_printf("-accel kvm%s -accel tcg " "-name source,debug-threads=on " "-m %s " "-serial file:%s/src_serial " "%s %s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", - machine_opts ? machine_opts : "", memory_size, tmpfs, arch_opts ? arch_opts : "", arch_source ? arch_source : "", @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, &got_src_stop); } - cmd_target = g_strdup_printf("-accel kvm%s -accel tcg %s " + cmd_target = g_strdup_printf("-accel kvm%s -accel tcg " "-name target,debug-threads=on " "-m %s " "-serial file:%s/dest_serial " @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "%s %s %s %s %s", args->use_dirty_ring ? ",dirty-ring-size=4096" : "", - machine_opts ? machine_opts : "", memory_size, tmpfs, uri, arch_opts ? arch_opts : "", arch_target ? arch_target : "", -- 2.40.1 So arch_dirty_ring option becomes one option like the others. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-8-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, g_autofree char *bootpath = NULL; g_autofree char *shmem_opts = NULL; g_autofree char *shmem_path = NULL; + const char *kvm_opts = NULL; const char *arch = qtest_get_arch(); const char *memory_size; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, shmem_opts = g_strdup(""); } + if (args->use_dirty_ring) { + kvm_opts = ",dirty-ring-size=4096"; + } + cmd_source = g_strdup_printf("-accel kvm%s -accel tcg " "-name source,debug-threads=on " "-m %s " "-serial file:%s/src_serial " "%s %s %s %s %s", - args->use_dirty_ring ? - ",dirty-ring-size=4096" : "", + kvm_opts ? kvm_opts : "", memory_size, tmpfs, arch_opts ? arch_opts : "", arch_source ? arch_source : "", @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "-serial file:%s/dest_serial " "-incoming %s " "%s %s %s %s %s", - args->use_dirty_ring ? - ",dirty-ring-size=4096" : "", + kvm_opts ? kvm_opts : "", memory_size, tmpfs, uri, arch_opts ? arch_opts : "", arch_target ? arch_target : "", -- 2.40.1 So just make it a global variable. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-9-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static bool ufd_version_check(void) #endif static char *tmpfs; +static char *bootpath; /* The boot file modifies memory area in [start_address, end_address) * repeatedly. It outputs a 'B' at a fixed rate while it's still running. @@ -XXX,XX +XXX,XX @@ static char *tmpfs; #include "tests/migration/aarch64/a-b-kernel.h" #include "tests/migration/s390x/a-b-bios.h" -static void init_bootfile(const char *bootpath, void *content, size_t len) +static void init_bootfile(void *content, size_t len) { FILE *bootfile = fopen(bootpath, "wb"); @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, g_autofree gchar *cmd_source = NULL; g_autofree gchar *cmd_target = NULL; const gchar *ignore_stderr; - g_autofree char *bootpath = NULL; g_autofree char *shmem_opts = NULL; g_autofree char *shmem_path = NULL; const char *kvm_opts = NULL; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, got_src_stop = false; got_dst_resume = false; - bootpath = g_strdup_printf("%s/bootsect", tmpfs); if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { /* the assembled x86 boot sector should be exactly one sector large */ assert(sizeof(x86_bootsect) == 512); - init_bootfile(bootpath, x86_bootsect, sizeof(x86_bootsect)); + init_bootfile(x86_bootsect, sizeof(x86_bootsect)); memory_size = "150M"; arch_opts = g_strdup_printf("-drive file=%s,format=raw", bootpath); start_address = X86_TEST_MEM_START; end_address = X86_TEST_MEM_END; } else if (g_str_equal(arch, "s390x")) { - init_bootfile(bootpath, s390x_elf, sizeof(s390x_elf)); + init_bootfile(s390x_elf, sizeof(s390x_elf)); memory_size = "128M"; arch_opts = g_strdup_printf("-bios %s", bootpath); start_address = S390_TEST_MEM_START; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "until'", end_address, start_address); arch_opts = g_strdup("-nodefaults -machine vsmt=8"); } else if (strcmp(arch, "aarch64") == 0) { - init_bootfile(bootpath, aarch64_kernel, sizeof(aarch64_kernel)); + init_bootfile(aarch64_kernel, sizeof(aarch64_kernel)); memory_size = "150M"; arch_opts = g_strdup_printf("-machine virt,gic-version=max -cpu max " "-kernel %s", bootpath); @@ -XXX,XX +XXX,XX @@ static void test_migrate_end(QTestState *from, QTestState *to, bool test_dest) qtest_quit(to); - cleanup("bootsect"); cleanup("migsocket"); cleanup("src_serial"); cleanup("dest_serial"); @@ -XXX,XX +XXX,XX @@ static QTestState *dirtylimit_start_vm(void) QTestState *vm = NULL; g_autofree gchar *cmd = NULL; const char *arch = qtest_get_arch(); - g_autofree char *bootpath = NULL; assert((strcmp(arch, "x86_64") == 0)); - bootpath = g_strdup_printf("%s/bootsect", tmpfs); assert(sizeof(x86_bootsect) == 512); - init_bootfile(bootpath, x86_bootsect, sizeof(x86_bootsect)); + init_bootfile(x86_bootsect, sizeof(x86_bootsect)); cmd = g_strdup_printf("-accel kvm,dirty-ring-size=4096 " "-name dirtylimit-test,debug-threads=on " @@ -XXX,XX +XXX,XX @@ static QTestState *dirtylimit_start_vm(void) static void dirtylimit_stop_vm(QTestState *vm) { qtest_quit(vm); - cleanup("bootsect"); cleanup("vm_serial"); } @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_get_tmp_dir(), err->message); } g_assert(tmpfs); + bootpath = g_strdup_printf("%s/bootsect", tmpfs); module_call_init(MODULE_INIT_QOM); @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_assert_cmpint(ret, ==, 0); + cleanup("bootsect"); + g_free(bootpath); ret = rmdir(tmpfs); if (ret != 0) { g_test_message("unable to rmdir: path (%s): %s", -- 2.40.1 The bootsector code is read only from the guest (otherwise we are going to have problems with it being read from both source and destination). Create a single copy for all the tests. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-10-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 50 ++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 14 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static char *bootpath; #include "tests/migration/aarch64/a-b-kernel.h" #include "tests/migration/s390x/a-b-bios.h" -static void init_bootfile(void *content, size_t len) +static void bootfile_create(char *dir) { + const char *arch = qtest_get_arch(); + unsigned char *content; + size_t len; + + bootpath = g_strdup_printf("%s/bootsect", dir); + if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { + /* the assembled x86 boot sector should be exactly one sector large */ + g_assert(sizeof(x86_bootsect) == 512); + content = x86_bootsect; + len = sizeof(x86_bootsect); + } else if (g_str_equal(arch, "s390x")) { + content = s390x_elf; + len = sizeof(s390x_elf); + } else if (strcmp(arch, "ppc64") == 0) { + /* + * sane architectures can be programmed at the boot prompt + */ + return; + } else if (strcmp(arch, "aarch64") == 0) { + content = aarch64_kernel; + len = sizeof(aarch64_kernel); + g_assert(sizeof(aarch64_kernel) <= ARM_TEST_MAX_KERNEL_SIZE); + } else { + g_assert_not_reached(); + } + FILE *bootfile = fopen(bootpath, "wb"); g_assert_cmpint(fwrite(content, len, 1, bootfile), ==, 1); fclose(bootfile); } +static void bootfile_delete(void) +{ + unlink(bootpath); + g_free(bootpath); + bootpath = NULL; +} + /* * Wait for some output in the serial output file, * we get an 'A' followed by an endless string of 'B's @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, got_src_stop = false; got_dst_resume = false; if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { - /* the assembled x86 boot sector should be exactly one sector large */ - assert(sizeof(x86_bootsect) == 512); - init_bootfile(x86_bootsect, sizeof(x86_bootsect)); memory_size = "150M"; arch_opts = g_strdup_printf("-drive file=%s,format=raw", bootpath); start_address = X86_TEST_MEM_START; end_address = X86_TEST_MEM_END; } else if (g_str_equal(arch, "s390x")) { - init_bootfile(s390x_elf, sizeof(s390x_elf)); memory_size = "128M"; arch_opts = g_strdup_printf("-bios %s", bootpath); start_address = S390_TEST_MEM_START; @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "until'", end_address, start_address); arch_opts = g_strdup("-nodefaults -machine vsmt=8"); } else if (strcmp(arch, "aarch64") == 0) { - init_bootfile(aarch64_kernel, sizeof(aarch64_kernel)); memory_size = "150M"; arch_opts = g_strdup_printf("-machine virt,gic-version=max -cpu max " "-kernel %s", bootpath); start_address = ARM_TEST_MEM_START; end_address = ARM_TEST_MEM_END; - - g_assert(sizeof(aarch64_kernel) <= ARM_TEST_MAX_KERNEL_SIZE); } else { g_assert_not_reached(); } @@ -XXX,XX +XXX,XX @@ static QTestState *dirtylimit_start_vm(void) const char *arch = qtest_get_arch(); assert((strcmp(arch, "x86_64") == 0)); - assert(sizeof(x86_bootsect) == 512); - init_bootfile(x86_bootsect, sizeof(x86_bootsect)); - cmd = g_strdup_printf("-accel kvm,dirty-ring-size=4096 " "-name dirtylimit-test,debug-threads=on " "-m 150M -smp 1 " @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_get_tmp_dir(), err->message); } g_assert(tmpfs); - bootpath = g_strdup_printf("%s/bootsect", tmpfs); + bootfile_create(tmpfs); module_call_init(MODULE_INIT_QOM); @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_assert_cmpint(ret, ==, 0); - cleanup("bootsect"); - g_free(bootpath); + bootfile_delete(); ret = rmdir(tmpfs); if (ret != 0) { g_test_message("unable to rmdir: path (%s): %s", -- 2.40.1 So no need to assert we are in x86_64. Once there, refactor the function to remove useless variables. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-11-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int64_t get_limit_rate(QTestState *who) static QTestState *dirtylimit_start_vm(void) { QTestState *vm = NULL; - g_autofree gchar *cmd = NULL; - const char *arch = qtest_get_arch(); - - assert((strcmp(arch, "x86_64") == 0)); + g_autofree gchar * cmd = g_strdup_printf("-accel kvm,dirty-ring-size=4096 " "-name dirtylimit-test,debug-threads=on " "-m 150M -smp 1 " -- 2.40.1 Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230608224943.3877-4-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, "-object memory-backend-file,id=mem0,size=%s" ",mem-path=%s,share=on -numa node,memdev=mem0", memory_size, shmem_path); - } else { - shmem_path = NULL; - shmem_opts = g_strdup(""); } if (args->use_dirty_ring) { @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, memory_size, tmpfs, arch_opts ? arch_opts : "", arch_source ? arch_source : "", - shmem_opts, + shmem_opts ? shmem_opts : "", args->opts_source ? args->opts_source : "", ignore_stderr); if (!args->only_target) { @@ -XXX,XX +XXX,XX @@ static int test_migrate_start(QTestState **from, QTestState **to, memory_size, tmpfs, uri, arch_opts ? arch_opts : "", arch_target ? arch_target : "", - shmem_opts, + shmem_opts ? shmem_opts : "", args->opts_target ? args->opts_target : "", ignore_stderr); *to = qtest_init(cmd_target); -- 2.40.1 From: Tejus GK <tejus.gk@nutanix.com> There are places in migration.c where the migration is marked failed with MIGRATION_STATUS_FAILED, but the failure reason is never updated. Hence libvirt doesn't know why the migration failed when it queries for it. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Tejus GK <tejus.gk@nutanix.com> Message-ID: <20230621130940.178659-2-tejus.gk@nutanix.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/migration.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, if (!(has_resume && resume)) { yank_unregister_instance(MIGRATION_YANK_INSTANCE); } - error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri", + error_setg(&local_err, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol"); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED); @@ -XXX,XX +XXX,XX @@ migration_wait_main_channel(MigrationState *ms) * Switch from normal iteration to postcopy * Returns non-0 on error */ -static int postcopy_start(MigrationState *ms) +static int postcopy_start(MigrationState *ms, Error **errp) { int ret; QIOChannelBuffer *bioc; @@ -XXX,XX +XXX,XX @@ static int postcopy_start(MigrationState *ms) */ ret = qemu_file_get_error(ms->to_dst_file); if (ret) { - error_report("postcopy_start: Migration stream errored (pre package)"); + error_setg(errp, "postcopy_start: Migration stream errored (pre package)"); goto fail_closefb; } @@ -XXX,XX +XXX,XX @@ static int postcopy_start(MigrationState *ms) ret = qemu_file_get_error(ms->to_dst_file); if (ret) { - error_report("postcopy_start: Migration stream errored"); + error_setg(errp, "postcopy_start: Migration stream errored"); migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE, MIGRATION_STATUS_FAILED); } @@ -XXX,XX +XXX,XX @@ typedef enum { static MigIterateState migration_iteration_run(MigrationState *s) { uint64_t must_precopy, can_postcopy; + Error *local_err = NULL; bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE; qemu_savevm_state_pending_estimate(&must_precopy, &can_postcopy); @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) /* Still a significant amount to transfer */ if (!in_postcopy && must_precopy <= s->threshold_size && qatomic_read(&s->start_postcopy)) { - if (postcopy_start(s)) { - error_report("%s: postcopy failed to start", __func__); + if (postcopy_start(s, &local_err)) { + migrate_set_error(s, local_err); + error_report_err(local_err); } return MIG_ITERATE_SKIP; } @@ -XXX,XX +XXX,XX @@ void migrate_fd_connect(MigrationState *s, Error *error_in) */ if (migrate_postcopy_ram() || migrate_return_path()) { if (open_return_path_on_source(s, !resume)) { - error_report("Unable to open return-path for postcopy"); + error_setg(&local_err, "Unable to open return-path for postcopy"); migrate_set_state(&s->state, s->state, MIGRATION_STATUS_FAILED); + migrate_set_error(s, local_err); + error_report_err(local_err); migrate_fd_cleanup(s); return; } @@ -XXX,XX +XXX,XX @@ void migrate_fd_connect(MigrationState *s, Error *error_in) } if (multifd_save_setup(&local_err) != 0) { + migrate_set_error(s, local_err); error_report_err(local_err); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED); -- 2.40.1 From: Tejus GK <tejus.gk@nutanix.com> In the function qmp_migrate(), yank_unregister_instance() gets called twice which isn't required. Hence, refactoring it so that it gets called during the local_error cleanup. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Acked-by: Peter Xu <peterx@redhat.com> Signed-off-by: Tejus GK <tejus.gk@nutanix.com> Message-ID: <20230621130940.178659-3-tejus.gk@nutanix.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/migration.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, } else if (strstart(uri, "fd:", &p)) { fd_start_outgoing_migration(s, p, &local_err); } else { - if (!(has_resume && resume)) { - yank_unregister_instance(MIGRATION_YANK_INSTANCE); - } error_setg(&local_err, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol"); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED); block_cleanup_parameters(); - return; } if (local_err) { -- 2.40.1 From: Wei Wang <wei.w.wang@intel.com> qemu_start_incoming_migration needs to check the number of multifd channels or postcopy ram channels to configure the backlog parameter (i.e. the maximum length to which the queue of pending connections for sockfd may grow) of listen(). So enforce the usage of postcopy-preempt and multifd as below: - need to use "-incoming defer" on the destination; and - set_capability and set_parameter need to be done before migrate_incoming Otherwise, disable the use of the features and report error messages to remind users to adjust the commands. Signed-off-by: Wei Wang <wei.w.wang@intel.com> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-ID: <20230606101910.20456-2-wei.w.wang@intel.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Acked-by: Juan Quintela <quintela@redhat.com> --- migration/options.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot, MIGRATION_CAPABILITY_VALIDATE_UUID, MIGRATION_CAPABILITY_ZERO_COPY_SEND); +static bool migrate_incoming_started(void) +{ + return !!migration_incoming_get_current()->transport_data; +} + /** * @migration_caps_check - check capability compatibility * @@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) error_setg(errp, "Postcopy preempt not compatible with compress"); return false; } + + if (migrate_incoming_started()) { + error_setg(errp, + "Postcopy preempt must be set before incoming starts"); + return false; + } } if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) { @@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) error_setg(errp, "Multifd is not compatible with compress"); return false; } + if (migrate_incoming_started()) { + error_setg(errp, "Multifd must be set before incoming starts"); + return false; + } } if (new_caps[MIGRATION_CAPABILITY_DIRTY_LIMIT]) { -- 2.40.1 From: Wei Wang <wei.w.wang@intel.com> The Postcopy preempt capability is expected to be set before incoming starts, so change the postcopy tests to start with deferred incoming and call migrate-incoming after the cap has been set. Why the existing tests (without this patch) didn't fail? There could be two reasons: 1) "backlog" specifies the number of pending connections. As long as the server accepts the connections faster than the clients side connecting, connection will succeed. For the preempt test, it uses only 2 channels, so very likely to not have pending connections. 2) per my tests (on kernel 6.2), the number of pending connections allowed is actually "backlog + 1", which is 2 in this case. That said, the implementation of socket_start_incoming_migration_internal expects "migrate defer" to be used, and for safety, change the test to work with the expected usage. Signed-off-by: Wei Wang <wei.w.wang@intel.com> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-ID: <20230606101910.20456-3-wei.w.wang@intel.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- tests/qtest/migration-test.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr, QTestState **to_ptr, MigrateCommon *args) { - g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); + g_autofree char *uri = NULL; QTestState *from, *to; - if (test_migrate_start(&from, &to, uri, &args->start)) { + if (test_migrate_start(&from, &to, "defer", &args->start)) { return -1; } @@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr, migrate_ensure_non_converge(from); + qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming'," + " 'arguments': { 'uri': 'tcp:127.0.0.1:0' }}"); + /* Wait for the first serial output from the source */ wait_for_serial("src_serial"); + uri = migrate_get_socket_address(to, "socket-address"); migrate_qmp(from, uri, "{}"); wait_for_migration_pass(from); -- 2.40.1 Fast don't say much. Noflush indicates more clearly that it is like qemu_file_transferred but without the flush. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20230530183941.7223-2-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/qemu-file.h | 11 +++++------ migration/qemu-file.c | 2 +- migration/savevm.c | 4 ++-- migration/vmstate.c | 4 ++-- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/migration/qemu-file.h b/migration/qemu-file.h index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -XXX,XX +XXX,XX @@ int qemu_fclose(QEMUFile *f); uint64_t qemu_file_transferred(QEMUFile *f); /* - * qemu_file_transferred_fast: + * qemu_file_transferred_noflush: * - * As qemu_file_transferred except for writable - * files, where no flush is performed and the reported - * amount will include the size of any queued buffers, - * on top of the amount actually transferred. + * As qemu_file_transferred except for writable files, where no flush + * is performed and the reported amount will include the size of any + * queued buffers, on top of the amount actually transferred. * * Returns: the total bytes transferred and queued */ -uint64_t qemu_file_transferred_fast(QEMUFile *f); +uint64_t qemu_file_transferred_noflush(QEMUFile *f); /* * put_buffer without copying the buffer. diff --git a/migration/qemu-file.c b/migration/qemu-file.c index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -XXX,XX +XXX,XX @@ int coroutine_mixed_fn qemu_get_byte(QEMUFile *f) return result; } -uint64_t qemu_file_transferred_fast(QEMUFile *f) +uint64_t qemu_file_transferred_noflush(QEMUFile *f) { uint64_t ret = f->total_transferred; int i; diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ static int vmstate_load(QEMUFile *f, SaveStateEntry *se) static void vmstate_save_old_style(QEMUFile *f, SaveStateEntry *se, JSONWriter *vmdesc) { - uint64_t old_offset = qemu_file_transferred_fast(f); + uint64_t old_offset = qemu_file_transferred_noflush(f); se->ops->save_state(f, se->opaque); - uint64_t size = qemu_file_transferred_fast(f) - old_offset; + uint64_t size = qemu_file_transferred_noflush(f) - old_offset; if (vmdesc) { json_writer_int64(vmdesc, "size", size); diff --git a/migration/vmstate.c b/migration/vmstate.c index XXXXXXX..XXXXXXX 100644 --- a/migration/vmstate.c +++ b/migration/vmstate.c @@ -XXX,XX +XXX,XX @@ int vmstate_save_state_v(QEMUFile *f, const VMStateDescription *vmsd, void *curr_elem = first_elem + size * i; vmsd_desc_field_start(vmsd, vmdesc_loop, field, i, n_elems); - old_offset = qemu_file_transferred_fast(f); + old_offset = qemu_file_transferred_noflush(f); if (field->flags & VMS_ARRAY_OF_POINTER) { assert(curr_elem); curr_elem = *(void **)curr_elem; @@ -XXX,XX +XXX,XX @@ int vmstate_save_state_v(QEMUFile *f, const VMStateDescription *vmsd, return ret; } - written_bytes = qemu_file_transferred_fast(f) - old_offset; + written_bytes = qemu_file_transferred_noflush(f) - old_offset; vmsd_desc_field_end(vmsd, vmdesc_loop, field, written_bytes, i); /* Compressed arrays only care about the first element */ -- 2.40.1 We do a qemu_fclose() just after that, that also does a qemu_fflush(), so remove one qemu_fflush(). Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20230530183941.7223-3-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/savevm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ bool save_snapshot(const char *name, bool overwrite, const char *vmstate, goto the_end; } ret = qemu_savevm_state(f, errp); - vm_state_size = qemu_file_transferred(f); + vm_state_size = qemu_file_transferred_noflush(f); ret2 = qemu_fclose(f); if (ret < 0) { goto the_end; -- 2.40.1 We only care about the amount of bytes transferred. Flushing is done by the system somewhere else. Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com> Message-ID: <20230530183941.7223-4-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/migration/block.c b/migration/block.c index XXXXXXX..XXXXXXX 100644 --- a/migration/block.c +++ b/migration/block.c @@ -XXX,XX +XXX,XX @@ static int block_save_setup(QEMUFile *f, void *opaque) static int block_save_iterate(QEMUFile *f, void *opaque) { int ret; - uint64_t last_bytes = qemu_file_transferred(f); + uint64_t last_bytes = qemu_file_transferred_noflush(f); trace_migration_block_save("iterate", block_mig_state.submitted, block_mig_state.transferred); @@ -XXX,XX +XXX,XX @@ static int block_save_iterate(QEMUFile *f, void *opaque) } qemu_put_be64(f, BLK_MIG_FLAG_EOS); - uint64_t delta_bytes = qemu_file_transferred(f) - last_bytes; + uint64_t delta_bytes = qemu_file_transferred_noflush(f) - last_bytes; return (delta_bytes > 0); } -- 2.40.1 It is not used outside of qemu_file, and it shouldn't. Signed-off-by: Juan Quintela <quintela@redhat.com> Message-ID: <20230530183941.7223-19-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/qemu-file.h | 1 - migration/qemu-file.c | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/migration/qemu-file.h b/migration/qemu-file.h index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -XXX,XX +XXX,XX @@ uint64_t qemu_file_transferred_noflush(QEMUFile *f); void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size, bool may_free); bool qemu_file_mode_is_not_valid(const char *mode); -bool qemu_file_is_writable(QEMUFile *f); #include "migration/qemu-file-types.h" diff --git a/migration/qemu-file.c b/migration/qemu-file.c index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -XXX,XX +XXX,XX @@ void qemu_file_set_error(QEMUFile *f, int ret) qemu_file_set_error_obj(f, ret, NULL); } -bool qemu_file_is_writable(QEMUFile *f) +static bool qemu_file_is_writable(QEMUFile *f) { return f->is_writable; } -- 2.40.1 Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230530183941.7223-20-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/qemu-file.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -XXX,XX +XXX,XX @@ struct QEMUFile { */ int qemu_file_shutdown(QEMUFile *f) { - int ret = 0; - /* * We must set qemufile error before the real shutdown(), otherwise * there can be a race window where we thought IO all went though @@ -XXX,XX +XXX,XX @@ int qemu_file_shutdown(QEMUFile *f) } if (qio_channel_shutdown(f->ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL) < 0) { - ret = -EIO; + return -EIO; } - return ret; + return 0; } bool qemu_file_mode_is_not_valid(const char *mode) -- 2.40.1 It was not used outside of qemu_file.c anyways. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230530183941.7223-21-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/qemu-file.h | 1 - migration/qemu-file.c | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/migration/qemu-file.h b/migration/qemu-file.h index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -XXX,XX +XXX,XX @@ void qemu_file_skip(QEMUFile *f, int size); * accounting information tracks the total migration traffic. */ void qemu_file_credit_transfer(QEMUFile *f, size_t size); -int qemu_file_get_error_obj(QEMUFile *f, Error **errp); int qemu_file_get_error_obj_any(QEMUFile *f1, QEMUFile *f2, Error **errp); void qemu_file_set_error_obj(QEMUFile *f, int ret, Error *err); void qemu_file_set_error(QEMUFile *f, int ret); diff --git a/migration/qemu-file.c b/migration/qemu-file.c index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -XXX,XX +XXX,XX @@ void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks) * is not 0. * */ -int qemu_file_get_error_obj(QEMUFile *f, Error **errp) +static int qemu_file_get_error_obj(QEMUFile *f, Error **errp) { if (errp) { *errp = f->last_error_obj ? error_copy(f->last_error_obj) : NULL; -- 2.40.1 This is how everything else in QEMUFile is structured. As a bonus they are three less lines of code. Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20230530183941.7223-17-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/qemu-file.h | 1 - migration/qemu-file.c | 12 ------------ migration/rdma.c | 39 +++++++++++++++++++-------------------- 3 files changed, 19 insertions(+), 33 deletions(-) diff --git a/migration/qemu-file.h b/migration/qemu-file.h index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -XXX,XX +XXX,XX @@ uint64_t qemu_file_transferred_noflush(QEMUFile *f); */ void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size, bool may_free); -bool qemu_file_mode_is_not_valid(const char *mode); #include "migration/qemu-file-types.h" diff --git a/migration/qemu-file.c b/migration/qemu-file.c index XXXXXXX..XXXXXXX 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -XXX,XX +XXX,XX @@ int qemu_file_shutdown(QEMUFile *f) return 0; } -bool qemu_file_mode_is_not_valid(const char *mode) -{ - if (mode == NULL || - (mode[0] != 'r' && mode[0] != 'w') || - mode[1] != 'b' || mode[2] != 0) { - fprintf(stderr, "qemu_fopen: Argument validity check failed\n"); - return true; - } - - return false; -} - static QEMUFile *qemu_file_new_impl(QIOChannel *ioc, bool is_writable) { QEMUFile *f; diff --git a/migration/rdma.c b/migration/rdma.c index XXXXXXX..XXXXXXX 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -XXX,XX +XXX,XX @@ static void qio_channel_rdma_register_types(void) type_init(qio_channel_rdma_register_types); -static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode) +static QEMUFile *rdma_new_input(RDMAContext *rdma) { - QIOChannelRDMA *rioc; + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); - if (qemu_file_mode_is_not_valid(mode)) { - return NULL; - } + rioc->file = qemu_file_new_input(QIO_CHANNEL(rioc)); + rioc->rdmain = rdma; + rioc->rdmaout = rdma->return_path; + qemu_file_set_hooks(rioc->file, &rdma_read_hooks); - rioc = QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); + return rioc->file; +} - if (mode[0] == 'w') { - rioc->file = qemu_file_new_output(QIO_CHANNEL(rioc)); - rioc->rdmaout = rdma; - rioc->rdmain = rdma->return_path; - qemu_file_set_hooks(rioc->file, &rdma_write_hooks); - } else { - rioc->file = qemu_file_new_input(QIO_CHANNEL(rioc)); - rioc->rdmain = rdma; - rioc->rdmaout = rdma->return_path; - qemu_file_set_hooks(rioc->file, &rdma_read_hooks); - } +static QEMUFile *rdma_new_output(RDMAContext *rdma) +{ + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); + + rioc->file = qemu_file_new_output(QIO_CHANNEL(rioc)); + rioc->rdmaout = rdma; + rioc->rdmain = rdma->return_path; + qemu_file_set_hooks(rioc->file, &rdma_write_hooks); return rioc->file; } @@ -XXX,XX +XXX,XX @@ static void rdma_accept_incoming_migration(void *opaque) return; } - f = qemu_fopen_rdma(rdma, "rb"); + f = rdma_new_input(rdma); if (f == NULL) { - fprintf(stderr, "RDMA ERROR: could not qemu_fopen_rdma\n"); + fprintf(stderr, "RDMA ERROR: could not open RDMA for input\n"); qemu_rdma_cleanup(rdma); return; } @@ -XXX,XX +XXX,XX @@ void rdma_start_outgoing_migration(void *opaque, trace_rdma_start_outgoing_migration_after_rdma_connect(); - s->to_dst_file = qemu_fopen_rdma(rdma, "wb"); + s->to_dst_file = rdma_new_output(rdma); migrate_fd_connect(s, NULL); return; return_path_err: -- 2.40.1 The following changes since commit ac0cc20ad2fe0b8df2e5d9458e90a095ac711ab1: Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging (2026-05-01 14:41:49 -0400) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags/next-pull-request for you to fetch changes up to 348c67cb16ecb479994b0c957699f906bd40a6c2: tests/qtest/migration: Fix A-B file build (2026-05-05 12:35:25 -0400) ---------------------------------------------------------------- Migration and mem pull request - Fabiano's fix on migrate_set_parameter crash with multifd & zerocopy - Pranav's fix on postcopy stucking at device state when ack lost - Samuel's new migration parameter x-rdma-chunk-size for RDMA - PeterX's vfio/migration series to report remaining data and fix downtime calc - PeterM's MemoryRegionOps .impl cleanup series - Fabiano's fix to build a-b migration bootfiles for all archs ---------------------------------------------------------------- CJ Chen (2): hw/riscv: iommu-trap: remove .impl.unaligned = true system/memory: assert on invalid MemoryRegionOps .unaligned combo Fabiano Rosas (2): migration: Use QAPI_CLONE_MEMBERS in migrate_params_test_apply tests/qtest/migration: Fix A-B file build Peter Maydell (2): hw/npcm7xx_fiu: Specify .impl for npcm7xx_fiu_flash_ops hw/xtensa/mx_pic: Specify xtensa_mx_pic_ops .impl settings Peter Xu (15): migration: Fix low possibility downtime violation migration/qapi: Rename MigrationStats to MigrationRAMStats vfio/migration: Cache stop size in VFIOMigration migration/treewide: Merge @state_pending_{exact|estimate} APIs migration: Use the new save_query_pending() API directly migration: Introduce stopcopy_bytes in save_query_pending() vfio/migration: Fix incorrect reporting for VFIO pending data migration: Move iteration counter out of RAM migration: Introduce a helper to return switchover bw estimate migration: Calculate expected downtime on demand migration: Fix calculation of expected_downtime to take VFIO info migration: Remember total dirty bytes in mig_stats migration/qapi: Introduce system-wide "remaining" reports migration/qapi: Update unit for avail-switchover-bandwidth vfio/migration: Add tracepoints for precopy/stopcopy query ioctls Pranav Tyagi (1): migration: Fix blocking in POSTCOPY_DEVICE during package load Samuel Zhang (1): migration/rdma: add x-rdma-chunk-size parameter docs/about/removed-features.rst | 2 +- docs/devel/migration/main.rst | 9 +- docs/devel/migration/vfio.rst | 9 +- qapi/migration.json | 45 +- hw/vfio/vfio-migration-internal.h | 8 + include/migration/register.h | 59 +- migration/migration-stats.h | 20 +- migration/migration.h | 3 +- migration/options.h | 1 + migration/savevm.h | 7 +- tests/qtest/migration/aarch64/a-b-kernel.h | 7 +- tests/qtest/migration/bootfile.h | 11 + tests/qtest/migration/i386/a-b-bootblock.h | 8 +- tests/qtest/migration/ppc64/a-b-kernel.h | 8 +- tests/qtest/migration/s390x/a-b-bios.h | 611 +++++++++++++-------- hw/riscv/riscv-iommu.c | 1 - hw/s390x/s390-stattrib.c | 9 +- hw/ssi/npcm7xx_fiu.c | 5 + hw/vfio/migration.c | 125 +++-- hw/xtensa/mx_pic.c | 27 + migration/block-dirty-bitmap.c | 10 +- migration/migration-hmp-cmds.c | 16 + migration/migration.c | 225 +++++--- migration/options.c | 59 +- migration/ram.c | 40 +- migration/rdma.c | 30 +- migration/savevm.c | 42 +- system/memory.c | 1 + tests/qtest/migration/s390x/a-b-bios.c | 66 ++- hw/vfio/trace-events | 5 +- migration/trace-events | 3 +- tests/qtest/migration/Makefile | 11 +- tests/qtest/migration/aarch64/Makefile | 4 +- tests/qtest/migration/aarch64/a-b-kernel.S | 2 +- tests/qtest/migration/i386/Makefile | 4 +- tests/qtest/migration/i386/a-b-bootblock.S | 2 +- tests/qtest/migration/ppc64/Makefile | 6 +- tests/qtest/migration/ppc64/a-b-kernel.S | 2 +- tests/qtest/migration/s390x/Makefile | 8 +- 39 files changed, 957 insertions(+), 554 deletions(-) -- 2.53.0 From: Pranav Tyagi <prtyagi@redhat.com> The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong. This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side. Fixes: 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state") Signed-off-by: Pranav Tyagi <prtyagi@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Link: https://lore.kernel.org/r/20260423094438.43556-1-prtyagi@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration.h | 1 - migration/migration.c | 48 ++++++++++++++++++++++++++++--------------- 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -XXX,XX +XXX,XX @@ struct MigrationState { bool rdma_migration; bool postcopy_package_loaded; - QemuEvent postcopy_package_loaded_event; GSource *hup_source; diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ int migrate_init(MigrationState *s, Error **errp) migration_reset_vfio_bytes_transferred(); s->postcopy_package_loaded = false; - qemu_event_reset(&s->postcopy_package_loaded_event); return 0; } @@ -XXX,XX +XXX,XX @@ static void *source_return_path_thread(void *opaque) if (tmp32 == QEMU_VM_PING_PACKAGED_LOADED) { trace_source_return_path_thread_postcopy_package_loaded(); ms->postcopy_package_loaded = true; - qemu_event_set(&ms->postcopy_package_loaded_event); + migration_rp_kick(ms); } break; @@ -XXX,XX +XXX,XX @@ out: trace_source_return_path_thread_bad_end(); } - if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) { + if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER || + ms->state == MIGRATION_STATUS_POSTCOPY_DEVICE) { /* - * this will be extremely unlikely: that we got yet another network - * issue during recovering of the 1st network failure.. during this - * period the main migration thread can be waiting on rp_sem for - * this thread to sync with the other side. + * The migration thread can get stuck waiting for rp_sem if the + * return path fails to sync with the destination. This handles + * two specific cases: * - * When this happens, explicitly kick the migration thread out of - * RECOVER stage and back to PAUSED, so the admin can try - * everything again. + * POSTCOPY_RECOVER: A failure occurs during a recovery attempt. + * We kick the migration thread back to PAUSED so the admin can + * retry. + * + * POSTCOPY_DEVICE: The MIG_RP_MSG_PONG is lost due to a + * network failure or destination crash. We kick the migration + * thread out of its wait so it can fail the migration and safely + * resume the VM on the source. */ migration_rp_kick(ms); } @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) if (s->state == MIGRATION_STATUS_POSTCOPY_DEVICE && (s->postcopy_package_loaded || complete_ready)) { /* - * If package has been loaded, the event is set and we will - * immediatelly transition to POSTCOPY_ACTIVE. If we are ready for - * completion, we need to wait for destination to load the postcopy - * package before actually completing. + * We will immediately transition to POSTCOPY_ACTIVE. + * If we are ready for completion, we need to wait for + * destination to load the postcopy package before actually + * completing. */ - qemu_event_wait(&s->postcopy_package_loaded_event); + while (!s->postcopy_package_loaded) { + if (migration_rp_wait(s)) { + /* + * Error happened. Migration thread was stuck waiting in + * POSTCOPY_DEVICE for rp_sem which was never set. + */ + migrate_set_state(&s->state, + MIGRATION_STATUS_POSTCOPY_DEVICE, + MIGRATION_STATUS_FAILING); + return MIG_ITERATE_BREAK; + } + } + /* Acknowledgement received from the destination */ migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE, MIGRATION_STATUS_POSTCOPY_ACTIVE); } @@ -XXX,XX +XXX,XX @@ static void migration_instance_finalize(Object *obj) qemu_sem_destroy(&ms->rp_state.rp_pong_acks); qemu_sem_destroy(&ms->postcopy_qemufile_src_sem); error_free(ms->error); - qemu_event_destroy(&ms->postcopy_package_loaded_event); } static void migration_instance_init(Object *obj) @@ -XXX,XX +XXX,XX @@ static void migration_instance_init(Object *obj) qemu_sem_init(&ms->wait_unplug_sem, 0); qemu_sem_init(&ms->postcopy_qemufile_src_sem, 0); qemu_mutex_init(&ms->qemu_file_lock); - qemu_event_init(&ms->postcopy_package_loaded_event, 0); } /* -- 2.53.0 From: Fabiano Rosas <farosas@suse.de> Use QAPI_CLONE_MEMBERS instead of making an assignment. The QAPI method makes the handling of the TLS strings more intuitive because it clones them as well. This also fixes a segfault when a NULL TLS option is accessed as part of a validation check for another option (e.g. in the zero-copy + multifd compression case). Details follow: Currently, after copying s->parameters to the temporary MigrationParameters object before migrate_params_check(), the references in temporary object to the TLS options are dropped, either because: a) the user set a new option, in which case that's fine as s->parameters still holds the reference to the old memory or, b) the user did not set a new option, in which case keeping the references in the temporary object would later cause them to be freed along with it, leading to double-free when s->parameters is also freed later on. In this second case, it was overlooked that the TLS options can be accessed already during migrate_params_check() as part of validation of another option. Those pointers should not have been cleared. Using QAPI_CLONE_MEMBERS fixes the issue because the temporary object is not stealing a reference from s->parameters anymore. Cc: qemu-stable <qemu-stable@nongnu.org> Fixes: aed97f0563 ("migration: Normalize tls arguments") Reported-by: Maciej S. Szmigiero <mail@maciej.szmigiero.name> Link: https://lore.kernel.org/r/a65a1049-9f19-460a-8e27-a62bb30d2727@maciej.szmigiero.name Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de> Tested-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> Link: https://lore.kernel.org/r/20260414223718.23965-1-farosas@suse.de Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/options.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp) static void migrate_params_test_apply(MigrationParameters *params, MigrationParameters *dest) { - *dest = migrate_get_current()->parameters; + MigrationState *s = migrate_get_current(); - /* TODO use QAPI_CLONE() instead of duplicating it inline */ + QAPI_CLONE_MEMBERS(MigrationParameters, dest, &s->parameters); if (params->has_throttle_trigger_threshold) { dest->throttle_trigger_threshold = params->throttle_trigger_threshold; @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrationParameters *params, } if (params->tls_creds) { + qapi_free_StrOrNull(dest->tls_creds); dest->tls_creds = QAPI_CLONE(StrOrNull, params->tls_creds); - } else { - /* clear the reference, it's owned by s->parameters */ - dest->tls_creds = NULL; } if (params->tls_hostname) { + qapi_free_StrOrNull(dest->tls_hostname); dest->tls_hostname = QAPI_CLONE(StrOrNull, params->tls_hostname); - } else { - /* clear the reference, it's owned by s->parameters */ - dest->tls_hostname = NULL; } if (params->tls_authz) { + qapi_free_StrOrNull(dest->tls_authz); dest->tls_authz = QAPI_CLONE(StrOrNull, params->tls_authz); - } else { - /* clear the reference, it's owned by s->parameters */ - dest->tls_authz = NULL; } if (params->has_max_bandwidth) { @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrationParameters *params, } if (params->has_block_bitmap_mapping) { - dest->has_block_bitmap_mapping = true; - dest->block_bitmap_mapping = params->block_bitmap_mapping; + qapi_free_BitmapMigrationNodeAliasList(dest->block_bitmap_mapping); + dest->block_bitmap_mapping = QAPI_CLONE(BitmapMigrationNodeAliasList, + params->block_bitmap_mapping); } if (params->has_x_vcpu_dirty_limit_period) { @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrationParameters *params, } if (params->has_cpr_exec_command) { - dest->cpr_exec_command = params->cpr_exec_command; + qapi_free_strList(dest->cpr_exec_command); + dest->cpr_exec_command = QAPI_CLONE(strList, params->cpr_exec_command); } } @@ -XXX,XX +XXX,XX @@ void qmp_migrate_set_parameters(MigrationParameters *params, Error **errp) } migrate_tls_opts_free(&tmp); + qapi_free_BitmapMigrationNodeAliasList(tmp.block_bitmap_mapping); + qapi_free_strList(tmp.cpr_exec_command); } -- 2.53.0 From: Samuel Zhang <guoqing.zhang@amd.com> The default 1MB RDMA chunk size causes slow live migration because each chunk triggers a write_flush (ibv_post_send). For 8GB RAM, 1MB chunk size produces ~15000 flushes vs ~3700 with 1024MB chunk size. Add x-rdma-chunk-size parameter to configure the RDMA chunk size for faster migration. Usage: `migrate_set_parameter x-rdma-chunk-size 1024M` Performance with RDMA live migration of 8GB RAM VM: | x-rdma-chunk-size (B) | time (s) | throughput (MB/s) | |-----------------------|----------|-------------------| | 1M (default) | 37.915 | 1,007 | | 32M | 17.880 | 2,260 | | 1024M | 4.368 | 17,529 | Signed-off-by: Samuel Zhang <guoqing.zhang@amd.com> Acked-by: Markus Armbruster <armbru@redhat.com> Acked-by: Li Zhijian <lizhijian@fujitsu.com> Tested-by: Li Zhijian <lizhijian@fujitsu.com> Acked-by: Fabiano Rosas <farosas@suse.de> Acked-by: Peter Xu <peterx@redhat.com> Link: https://lore.kernel.org/r/20260427031401.3895523-1-guoqing.zhang@amd.com Signed-off-by: Peter Xu <peterx@redhat.com> --- qapi/migration.json | 13 +++++++++++-- migration/options.h | 1 + migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 33 ++++++++++++++++++++++++++++++++- migration/rdma.c | 30 ++++++++++++++++-------------- 5 files changed, 71 insertions(+), 17 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # # Features: # -# @unstable: Members @x-checkpoint-delay and +# @unstable: Members @x-checkpoint-delay, @x-rdma-chunk-size, and # @x-vcpu-dirty-limit-period are experimental. # # Since: 2.4 @@ -XXX,XX +XXX,XX @@ 'mode', 'zero-page-detection', 'direct-io', + { 'name': 'x-rdma-chunk-size', 'features': [ 'unstable' ] }, 'cpr-exec-command'] } ## @@ -XXX,XX +XXX,XX @@ # is @cpr-exec. The first list element is the program's filename, # the remainder its arguments. (Since 10.2) # +# @x-rdma-chunk-size: RDMA memory registration chunk size in bytes. +# Default is 1MiB. Must be a power of 2 in the range +# [1MiB, 1024MiB]. Only applies when migrating via RDMA. +# Must be set to the same value on both source and destination +# before migration starts. (Since 11.1) +# # Features: # -# @unstable: Members @x-checkpoint-delay and +# @unstable: Members @x-checkpoint-delay, @x-rdma-chunk-size, and # @x-vcpu-dirty-limit-period are experimental. # # Since: 2.4 @@ -XXX,XX +XXX,XX @@ '*mode': 'MigMode', '*zero-page-detection': 'ZeroPageDetection', '*direct-io': 'bool', + '*x-rdma-chunk-size': { 'type': 'uint64', + 'features': [ 'unstable' ] }, '*cpr-exec-command': [ 'str' ]} } ## diff --git a/migration/options.h b/migration/options.h index XXXXXXX..XXXXXXX 100644 --- a/migration/options.h +++ b/migration/options.h @@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void); const char *migrate_tls_hostname(void); uint64_t migrate_xbzrle_cache_size(void); ZeroPageDetection migrate_zero_page_detection(void); +uint64_t migrate_rdma_chunk_size(void); /* parameters helpers */ diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) params->direct_io ? "on" : "off"); } + if (params->has_x_rdma_chunk_size) { + monitor_printf(mon, "%s: %" PRIu64 " bytes\n", + MigrationParameter_str( + MIGRATION_PARAMETER_X_RDMA_CHUNK_SIZE), + params->x_rdma_chunk_size); + } + assert(params->has_cpr_exec_command); monitor_print_cpr_exec_command(mon, params->cpr_exec_command); } @@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_direct_io = true; visit_type_bool(v, param, &p->direct_io, &err); break; + case MIGRATION_PARAMETER_X_RDMA_CHUNK_SIZE: + p->has_x_rdma_chunk_size = true; + visit_type_size(v, param, &p->x_rdma_chunk_size, &err); + break; case MIGRATION_PARAMETER_CPR_EXEC_COMMAND: { /* * NOTE: g_autofree will only auto g_free() the strv array when diff --git a/migration/options.c b/migration/options.c index XXXXXXX..XXXXXXX 100644 --- a/migration/options.c +++ b/migration/options.c @@ -XXX,XX +XXX,XX @@ #include "qemu/osdep.h" #include "qemu/error-report.h" +#include "qemu/units.h" #include "exec/target_page.h" #include "qapi/clone-visitor.h" #include "qapi/error.h" @@ -XXX,XX +XXX,XX @@ const PropertyInfo qdev_prop_StrOrNull; #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ +#define DEFAULT_MIGRATE_X_RDMA_CHUNK_SIZE MiB const Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, @@ -XXX,XX +XXX,XX @@ const Property migration_properties[] = { DEFINE_PROP_ZERO_PAGE_DETECTION("zero-page-detection", MigrationState, parameters.zero_page_detection, ZERO_PAGE_DETECTION_MULTIFD), + DEFINE_PROP_UINT64("x-rdma-chunk-size", MigrationState, + parameters.x_rdma_chunk_size, + DEFAULT_MIGRATE_X_RDMA_CHUNK_SIZE), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -XXX,XX +XXX,XX @@ ZeroPageDetection migrate_zero_page_detection(void) return s->parameters.zero_page_detection; } +uint64_t migrate_rdma_chunk_size(void) +{ + MigrationState *s = migrate_get_current(); + uint64_t size = s->parameters.x_rdma_chunk_size; + + assert(MiB <= size && size <= GiB && is_power_of_2(size)); + return size; +} + /* parameters helpers */ AnnounceParameters *migrate_announce_params(void) @@ -XXX,XX +XXX,XX @@ static void migrate_mark_all_params_present(MigrationParameters *p) &p->has_announce_step, &p->has_block_bitmap_mapping, &p->has_x_vcpu_dirty_limit_period, &p->has_vcpu_dirty_limit, &p->has_mode, &p->has_zero_page_detection, &p->has_direct_io, - &p->has_cpr_exec_command, + &p->has_x_rdma_chunk_size, &p->has_cpr_exec_command, }; len = ARRAY_SIZE(has_fields); @@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp) return false; } + if (params->has_x_rdma_chunk_size && + (params->x_rdma_chunk_size < MiB || + params->x_rdma_chunk_size > GiB || + !is_power_of_2(params->x_rdma_chunk_size))) { + error_setg(errp, "Option x_rdma_chunk_size expects " + "a power of 2 in the range 1MiB to 1024MiB"); + return false; + } + return true; } @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrationParameters *params, dest->direct_io = params->direct_io; } + if (params->has_x_rdma_chunk_size) { + dest->x_rdma_chunk_size = params->x_rdma_chunk_size; + } + if (params->has_cpr_exec_command) { qapi_free_strList(dest->cpr_exec_command); dest->cpr_exec_command = QAPI_CLONE(strList, params->cpr_exec_command); @@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrationParameters *params) s->parameters.direct_io = params->direct_io; } + if (params->has_x_rdma_chunk_size) { + s->parameters.x_rdma_chunk_size = params->x_rdma_chunk_size; + } + if (params->has_cpr_exec_command) { qapi_free_strList(s->parameters.cpr_exec_command); s->parameters.cpr_exec_command = diff --git a/migration/rdma.c b/migration/rdma.c index XXXXXXX..XXXXXXX 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -XXX,XX +XXX,XX @@ #define RDMA_RESOLVE_TIMEOUT_MS 10000 /* Do not merge data if larger than this. */ -#define RDMA_MERGE_MAX (2 * 1024 * 1024) -#define RDMA_SIGNALED_SEND_MAX (RDMA_MERGE_MAX / 4096) +static inline uint64_t rdma_merge_max(void) +{ + return migrate_rdma_chunk_size() * 2; +} -#define RDMA_REG_CHUNK_SHIFT 20 /* 1 MB */ +#define RDMA_SIGNALED_SEND_MAX 512 /* * This is only for non-live state being migrated. @@ -XXX,XX +XXX,XX @@ static int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head, static inline uint64_t ram_chunk_index(const uint8_t *start, const uint8_t *host) { - return ((uintptr_t) host - (uintptr_t) start) >> RDMA_REG_CHUNK_SHIFT; + return ((uintptr_t) host - (uintptr_t) start) / migrate_rdma_chunk_size(); } static inline uint8_t *ram_chunk_start(const RDMALocalBlock *rdma_ram_block, uint64_t i) { return (uint8_t *)(uintptr_t)(rdma_ram_block->local_host_addr + - (i << RDMA_REG_CHUNK_SHIFT)); + (i * migrate_rdma_chunk_size())); } static inline uint8_t *ram_chunk_end(const RDMALocalBlock *rdma_ram_block, uint64_t i) { uint8_t *result = ram_chunk_start(rdma_ram_block, i) + - (1UL << RDMA_REG_CHUNK_SHIFT); + migrate_rdma_chunk_size(); if (result > (rdma_ram_block->local_host_addr + rdma_ram_block->length)) { result = rdma_ram_block->local_host_addr + rdma_ram_block->length; @@ -XXX,XX +XXX,XX @@ static int qemu_rdma_write_one(RDMAContext *rdma, struct ibv_send_wr *bad_wr; int reg_result_idx, ret, count = 0; uint64_t chunk, chunks; + uint64_t chunk_size = migrate_rdma_chunk_size(); uint8_t *chunk_start, *chunk_end; RDMALocalBlock *block = &(rdma->local_ram_blocks.block[current_index]); RDMARegister reg; @@ -XXX,XX +XXX,XX @@ retry: chunk_start = ram_chunk_start(block, chunk); if (block->is_ram_block) { - chunks = length / (1UL << RDMA_REG_CHUNK_SHIFT); + chunks = length / chunk_size; - if (chunks && ((length % (1UL << RDMA_REG_CHUNK_SHIFT)) == 0)) { + if (chunks && ((length % chunk_size) == 0)) { chunks--; } } else { - chunks = block->length / (1UL << RDMA_REG_CHUNK_SHIFT); + chunks = block->length / chunk_size; - if (chunks && ((block->length % (1UL << RDMA_REG_CHUNK_SHIFT)) == 0)) { + if (chunks && ((block->length % chunk_size) == 0)) { chunks--; } } trace_qemu_rdma_write_one_top(chunks + 1, - (chunks + 1) * - (1UL << RDMA_REG_CHUNK_SHIFT) / 1024 / 1024); + (chunks + 1) * chunk_size / 1024 / 1024); chunk_end = ram_chunk_end(block, chunk + chunks); @@ -XXX,XX +XXX,XX @@ static int qemu_rdma_write(RDMAContext *rdma, rdma->current_length += len; /* flush it if buffer is too large */ - if (rdma->current_length >= RDMA_MERGE_MAX) { + if (rdma->current_length >= rdma_merge_max()) { return qemu_rdma_write_flush(rdma, errp); } @@ -XXX,XX +XXX,XX @@ int rdma_registration_handle(QEMUFile *f) } else { chunk = reg->key.chunk; host_addr = block->local_host_addr + - (reg->key.chunk * (1UL << RDMA_REG_CHUNK_SHIFT)); + (reg->key.chunk * migrate_rdma_chunk_size()); /* Check for particularly bad chunk value */ if (host_addr < (void *)block->local_host_addr) { error_report("rdma: bad chunk for block %s" -- 2.53.0 When QEMU queried the estimated version of pending data and thinks it's ready to converge, it'll send another accurate query to make sure of it. It is needed to make sure we collect the latest reports and that equation still holds true. However we missed one tiny little difference here on "<" v.s. "<=" when comparing pending_size (A) to threshold_size (B).. QEMU src only re-query if A<B, but will kickoff switchover if A<=B. I think it means it is possible to happen if A (as an estimate only so far) accidentally equals to B, then re-query won't happen and switchover will proceed without considering new dirtied data. It turns out it was an accident in my commit 7aaa1fc072 when refactoring the code around. Fix this by using the same equation in both places. Fixes: 7aaa1fc072 ("migration: Rewrite the migration complete detect logic") Cc: qemu-stable@nongnu.org Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-3-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) * postcopy started, so ESTIMATE should always match with EXACT * during postcopy phase. */ - if (pending_size < s->threshold_size) { + if (pending_size <= s->threshold_size) { qemu_savevm_state_pending_exact(&must_precopy, &can_postcopy); pending_size = must_precopy + can_postcopy; trace_migrate_pending_exact(pending_size, must_precopy, -- 2.53.0 This stats is only about RAM, make it accurate. This paves way for statistics for all devices. Thanks to Markus, who pointed out that docs/devel/qapi-code-gen.rst has a section "Compatibility considerations" stated: Since type names are not visible in the Client JSON Protocol, types may be freely renamed. Even certain refactorings are invisible, such as splitting members from one type into a common base type. Hence this change is not ABI violation according to the document. While at it, touch up the lines to make it read better, correct the restriction on migration status being 'active' or 'completed': over time we grew too many new status that will also report "ram" section. Cc: Daniel P. Berrangé <berrange@redhat.com> Cc: devel@lists.libvirt.org Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-4-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- docs/about/removed-features.rst | 2 +- qapi/migration.json | 10 +++++----- migration/migration-stats.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/about/removed-features.rst b/docs/about/removed-features.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/about/removed-features.rst +++ b/docs/about/removed-features.rst @@ -XXX,XX +XXX,XX @@ was superseded by ``sections``. ``query-migrate`` return value member ``skipped`` (removed in 9.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -Member ``skipped`` of the ``MigrationStats`` struct hasn't been used +Member ``skipped`` of the ``MigrationRAMStats`` struct hasn't been used for more than 10 years. Removed with no replacement. ``migrate`` command option ``inc`` (removed in 9.1) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ { 'include': 'sockets.json' } ## -# @MigrationStats: +# @MigrationRAMStats: # # Detailed migration status. # @@ -XXX,XX +XXX,XX @@ # # Since: 0.14 ## -{ 'struct': 'MigrationStats', +{ 'struct': 'MigrationRAMStats', 'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' , 'duplicate': 'int', 'normal': 'int', @@ -XXX,XX +XXX,XX @@ # If this field is not returned, no migration process has been # initiated # -# @ram: `MigrationStats` containing detailed migration status, only -# returned if status is 'active' or 'completed'(since 1.2) +# @ram: Detailed migration RAM statistics, only returned if migration +# is in progress or completed (since 1.2) # # @xbzrle-cache: `XBZRLECacheStats` containing detailed XBZRLE # migration statistics, only returned if XBZRLE feature is on and @@ -XXX,XX +XXX,XX @@ # Since: 0.14 ## { 'struct': 'MigrationInfo', - 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationStats', + 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationRAMStats', '*vfio': 'VfioStats', '*xbzrle-cache': 'XBZRLECacheStats', '*total-time': 'int', diff --git a/migration/migration-stats.h b/migration/migration-stats.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -XXX,XX +XXX,XX @@ /* * These are the ram migration statistic counters. It is loosely - * based on MigrationStats. + * based on MigrationRAMStats. */ typedef struct { /* -- 2.53.0 Add a field to cache stop size. Note that there's an initial value change in vfio_save_setup for the stop size default, but it shouldn't matter if it is followed with a math of MIN() against VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE. Document that all the three sizes we read from VFIO's uAPI on dirty or stop sizes are estimates, so QEMU needs to always remember they can be anything. Reviewed-by: Avihai Horon <avihaih@nvidia.com> Link: https://lore.kernel.org/r/20260421202110.306051-5-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/vfio/vfio-migration-internal.h | 8 +++++ hw/vfio/migration.c | 50 ++++++++++++++++++------------- 2 files changed, 38 insertions(+), 20 deletions(-) diff --git a/hw/vfio/vfio-migration-internal.h b/hw/vfio/vfio-migration-internal.h index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/vfio-migration-internal.h +++ b/hw/vfio/vfio-migration-internal.h @@ -XXX,XX +XXX,XX @@ typedef struct VFIOMigration { void *data_buffer; size_t data_buffer_size; uint64_t mig_flags; + /* + * NOTE: all three sizes cached are reported from VFIO's uAPI, which + * are defined as estimate only. QEMU should not trust these values + * but only use them to do best-effort estimates. Always be prepared + * that these sizes may either grow or even shrink in reality while + * read()ing from the VFIO fds. + */ uint64_t precopy_init_size; uint64_t precopy_dirty_size; + uint64_t stopcopy_size; bool multifd_transfer; VFIOMultifd *multifd; bool initial_data_sent; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -XXX,XX +XXX,XX @@ */ #define VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE (1 * MiB) +/* + * Migration size of VFIO devices can be as little as a few KBs or as big as + * many GBs. This value should be big enough to cover the worst case. + */ +#define VFIO_MIG_STOP_COPY_SIZE (100 * GiB) + static unsigned long bytes_transferred; static const char *mig_state_to_str(enum vfio_device_mig_state state) @@ -XXX,XX +XXX,XX @@ static void vfio_migration_cleanup(VFIODevice *vbasedev) migration->data_fd = -1; } -static int vfio_query_stop_copy_size(VFIODevice *vbasedev, - uint64_t *stop_copy_size) +static int vfio_query_stop_copy_size(VFIODevice *vbasedev) { uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_mig_data_size), @@ -XXX,XX +XXX,XX @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev, struct vfio_device_feature *feature = (struct vfio_device_feature *)buf; struct vfio_device_feature_mig_data_size *mig_data_size = (struct vfio_device_feature_mig_data_size *)feature->data; + VFIOMigration *migration = vbasedev->migration; feature->argsz = sizeof(buf); feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIG_DATA_SIZE; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { + /* + * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE + * is reported so downtime limit won't be violated. + */ + migration->stopcopy_size = VFIO_MIG_STOP_COPY_SIZE; return -errno; } - *stop_copy_size = mig_data_size->stop_copy_length; + migration->stopcopy_size = mig_data_size->stop_copy_length; return 0; } @@ -XXX,XX +XXX,XX @@ static void vfio_update_estimated_pending_data(VFIOMigration *migration, return; } + /* + * The total size remaining requires separate accounting. Do not trust + * the counter, so what we have read() may be more than what reported. + */ + if (migration->stopcopy_size > data_size) { + migration->stopcopy_size -= data_size; + } else { + migration->stopcopy_size = 0; + } + if (migration->precopy_init_size) { uint64_t init_size = MIN(migration->precopy_init_size, data_size); @@ -XXX,XX +XXX,XX @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - uint64_t stop_copy_size = VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE; int ret; if (!vfio_multifd_setup(vbasedev, false, errp)) { @@ -XXX,XX +XXX,XX @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); - vfio_query_stop_copy_size(vbasedev, &stop_copy_size); + vfio_query_stop_copy_size(vbasedev); migration->data_buffer_size = MIN(VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE, - stop_copy_size); + migration->stopcopy_size); migration->data_buffer = g_try_malloc0(migration->data_buffer_size); if (!migration->data_buffer) { error_setg(errp, "%s: Failed to allocate migration data buffer", @@ -XXX,XX +XXX,XX @@ static void vfio_state_pending_estimate(void *opaque, uint64_t *must_precopy, migration->precopy_dirty_size); } -/* - * Migration size of VFIO devices can be as little as a few KBs or as big as - * many GBs. This value should be big enough to cover the worst case. - */ -#define VFIO_MIG_STOP_COPY_SIZE (100 * GiB) - static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, uint64_t *can_postcopy) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE; - /* - * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE is - * reported so downtime limit won't be violated. - */ - vfio_query_stop_copy_size(vbasedev, &stop_copy_size); - *must_precopy += stop_copy_size; + vfio_query_stop_copy_size(vbasedev); + *must_precopy += migration->stopcopy_size; if (vfio_device_state_is_precopy(vbasedev)) { vfio_query_precopy_size(migration); } trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy, - stop_copy_size, migration->precopy_init_size, + migration->stopcopy_size, + migration->precopy_init_size, migration->precopy_dirty_size); } -- 2.53.0 These two APIs are a slight duplication. For example, there're a few users that directly pass in the same function. It might also be error prone to provide two hooks, so that it's easier to happen one module report different things via the two hooks. In reality, they should always report the same thing, only about whether we should use a fast-path when the slow path might be too slow, as QEMU may query these information quite frequently during migration process. Merge it into one API, provide a bool showing if the query is an exact query or not. No functional change intended. Export qemu_savevm_query_pending(). We should use the new API here provided when there're new users to do the query. This will happen very soon. Cc: Halil Pasic <pasic@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Eric Farman <farman@linux.ibm.com> Cc: Matthew Rosato <mjrosato@linux.ibm.com> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: Ilya Leoshkevich <iii@linux.ibm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Cornelia Huck <cohuck@redhat.com> Cc: Eric Blake <eblake@redhat.com> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Cc: John Snow <jsnow@redhat.com> Reviewed-by: Jason J. Herne <jjherne@linux.ibm.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Avihai Horon <avihaih@nvidia.com> Acked-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Link: https://lore.kernel.org/r/20260421202110.306051-6-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- docs/devel/migration/main.rst | 9 ++---- docs/devel/migration/vfio.rst | 9 ++---- include/migration/register.h | 52 ++++++++++++---------------------- migration/savevm.h | 3 ++ hw/s390x/s390-stattrib.c | 9 +++--- hw/vfio/migration.c | 48 ++++++++++++++----------------- migration/block-dirty-bitmap.c | 10 +++---- migration/ram.c | 33 +++++++-------------- migration/savevm.c | 42 +++++++++++++-------------- hw/vfio/trace-events | 3 +- 10 files changed, 86 insertions(+), 132 deletions(-) diff --git a/docs/devel/migration/main.rst b/docs/devel/migration/main.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/devel/migration/main.rst +++ b/docs/devel/migration/main.rst @@ -XXX,XX +XXX,XX @@ An iterative device must provide: - A ``load_setup`` function that initialises the data structures on the destination. - - A ``state_pending_exact`` function that indicates how much more - data we must save. The core migration code will use this to - determine when to pause the CPUs and complete the migration. - - - A ``state_pending_estimate`` function that indicates how much more - data we must save. When the estimated amount is smaller than the - threshold, we call ``state_pending_exact``. + - A ``save_query_pending`` function that indicates how much more + data we must save. - A ``save_live_iterate`` function should send a chunk of data until the point that stream bandwidth limits tell it to stop. Each call diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -XXX,XX +XXX,XX @@ VFIO implements the device hooks for the iterative approach as follows: * A ``load_setup`` function that sets the VFIO device on the destination in _RESUMING state. -* A ``state_pending_estimate`` function that reports an estimate of the - remaining pre-copy data that the vendor driver has yet to save for the VFIO - device. - -* A ``state_pending_exact`` function that reads pending_bytes from the vendor - driver, which indicates the amount of data that the vendor driver has yet to - save for the VFIO device. +* A ``save_query_pending`` function that reports the remaining data that + the vendor driver has yet to save for the VFIO device. * An ``is_active_iterate`` function that indicates ``save_live_iterate`` is active only when the VFIO device is in pre-copy states. diff --git a/include/migration/register.h b/include/migration/register.h index XXXXXXX..XXXXXXX 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -XXX,XX +XXX,XX @@ #include "hw/core/vmstate-if.h" +typedef struct MigPendingData { + /* Amount of pending bytes can be transferred in precopy or stopcopy */ + uint64_t precopy_bytes; + /* Amount of pending bytes can be transferred in postcopy */ + uint64_t postcopy_bytes; +} MigPendingData; + /** * struct SaveVMHandlers: handler structure to finely control * migration of complex subsystems and devices, such as RAM, block and @@ -XXX,XX +XXX,XX @@ typedef struct SaveVMHandlers { bool (*save_postcopy_prepare)(QEMUFile *f, void *opaque, Error **errp); /** - * @state_pending_estimate - * - * This estimates the remaining data to transfer + * @save_query_pending * - * Sum of @can_postcopy and @must_postcopy is the whole amount of - * pending data. - * - * @opaque: data pointer passed to register_savevm_live() - * @must_precopy: amount of data that must be migrated in precopy - * or in stopped state, i.e. that must be migrated - * before target start. - * @can_postcopy: amount of data that can be migrated in postcopy - * or in stopped state, i.e. after target start. - * Some can also be migrated during precopy (RAM). - * Some must be migrated after source stops - * (block-dirty-bitmap) - */ - void (*state_pending_estimate)(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy); - - /** - * @state_pending_exact + * This estimates the remaining data to transfer on the source side. * - * This calculates the exact remaining data to transfer + * When @exact is true, a module must report accurate results. When + * @exact is false, a module may report estimates. * - * Sum of @can_postcopy and @must_postcopy is the whole amount of - * pending data. + * It's highly recommended that modules implement a faster version of + * the query path (for example, by proper caching on the counters) if + * an accurate query will be time-consuming. * * @opaque: data pointer passed to register_savevm_live() - * @must_precopy: amount of data that must be migrated in precopy - * or in stopped state, i.e. that must be migrated - * before target start. - * @can_postcopy: amount of data that can be migrated in postcopy - * or in stopped state, i.e. after target start. - * Some can also be migrated during precopy (RAM). - * Some must be migrated after source stops - * (block-dirty-bitmap) + * @pending: pointer to a MigPendingData struct + * @exact: set to true for an accurate (slow) query */ - void (*state_pending_exact)(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy); + void (*save_query_pending)(void *opaque, MigPendingData *pending, + bool exact); /** * @load_state diff --git a/migration/savevm.h b/migration/savevm.h index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -XXX,XX +XXX,XX @@ #ifndef MIGRATION_SAVEVM_H #define MIGRATION_SAVEVM_H +#include "migration/register.h" + #define QEMU_VM_FILE_MAGIC 0x5145564d #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002 #define QEMU_VM_FILE_VERSION 0x00000003 @@ -XXX,XX +XXX,XX @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy); void qemu_savevm_state_cleanup(void); void qemu_savevm_state_complete_postcopy(QEMUFile *f); int qemu_savevm_state_complete_precopy(MigrationState *s); +void qemu_savevm_query_pending(MigPendingData *pending, bool exact); void qemu_savevm_state_pending_exact(uint64_t *must_precopy, uint64_t *can_postcopy); void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c index XXXXXXX..XXXXXXX 100644 --- a/hw/s390x/s390-stattrib.c +++ b/hw/s390x/s390-stattrib.c @@ -XXX,XX +XXX,XX @@ static int cmma_save_setup(QEMUFile *f, void *opaque, Error **errp) return 0; } -static void cmma_state_pending(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy) +static void cmma_state_pending(void *opaque, MigPendingData *pending, + bool exact) { S390StAttribState *sas = S390_STATTRIB(opaque); S390StAttribClass *sac = S390_STATTRIB_GET_CLASS(sas); long long res = sac->get_dirtycount(sas); if (res >= 0) { - *must_precopy += res; + pending->precopy_bytes += res; } } @@ -XXX,XX +XXX,XX @@ static SaveVMHandlers savevm_s390_stattrib_handlers = { .save_setup = cmma_save_setup, .save_live_iterate = cmma_save_iterate, .save_complete = cmma_save_complete, - .state_pending_exact = cmma_state_pending, - .state_pending_estimate = cmma_state_pending, + .save_query_pending = cmma_state_pending, .save_cleanup = cmma_save_cleanup, .load_state = cmma_load, .is_active = cmma_active, diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -XXX,XX +XXX,XX @@ static void vfio_save_cleanup(void *opaque) trace_vfio_save_cleanup(vbasedev->name); } -static void vfio_state_pending_estimate(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy) +static void vfio_state_pending_sync(VFIODevice *vbasedev) { - VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - if (!vfio_device_state_is_precopy(vbasedev)) { - return; - } - - *must_precopy += - migration->precopy_init_size + migration->precopy_dirty_size; + vfio_query_stop_copy_size(vbasedev); - trace_vfio_state_pending_estimate(vbasedev->name, *must_precopy, - *can_postcopy, - migration->precopy_init_size, - migration->precopy_dirty_size); + if (vfio_device_state_is_precopy(vbasedev)) { + vfio_query_precopy_size(migration); + } } -static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy) +static void vfio_state_pending(void *opaque, MigPendingData *pending, + bool exact) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; + uint64_t remain; - vfio_query_stop_copy_size(vbasedev); - *must_precopy += migration->stopcopy_size; - - if (vfio_device_state_is_precopy(vbasedev)) { - vfio_query_precopy_size(migration); + if (exact) { + vfio_state_pending_sync(vbasedev); + remain = migration->stopcopy_size; + } else { + if (!vfio_device_state_is_precopy(vbasedev)) { + return; + } + remain = migration->precopy_init_size + migration->precopy_dirty_size; } - trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy, - migration->stopcopy_size, - migration->precopy_init_size, - migration->precopy_dirty_size); + pending->precopy_bytes += remain; + + trace_vfio_state_pending(vbasedev->name, migration->stopcopy_size, + migration->precopy_init_size, + migration->precopy_dirty_size, exact); } static bool vfio_is_active_iterate(void *opaque) @@ -XXX,XX +XXX,XX @@ static const SaveVMHandlers savevm_vfio_handlers = { .save_prepare = vfio_save_prepare, .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, - .state_pending_estimate = vfio_state_pending_estimate, - .state_pending_exact = vfio_state_pending_exact, + .save_query_pending = vfio_state_pending, .is_active_iterate = vfio_is_active_iterate, .save_live_iterate = vfio_save_iterate, .save_complete = vfio_save_complete_precopy, diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index XXXXXXX..XXXXXXX 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -XXX,XX +XXX,XX @@ static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque) return 0; } -static void dirty_bitmap_state_pending(void *opaque, - uint64_t *must_precopy, - uint64_t *can_postcopy) +static void dirty_bitmap_state_pending(void *opaque, MigPendingData *data, + bool exact) { DBMSaveState *s = &((DBMState *)opaque)->save; SaveBitmapState *dbms; @@ -XXX,XX +XXX,XX @@ static void dirty_bitmap_state_pending(void *opaque, trace_dirty_bitmap_state_pending(pending); - *can_postcopy += pending; + data->postcopy_bytes += pending; } /* First occurrence of this bitmap. It should be created if doesn't exist */ @@ -XXX,XX +XXX,XX @@ static SaveVMHandlers savevm_dirty_bitmap_handlers = { .save_setup = dirty_bitmap_save_setup, .save_complete = dirty_bitmap_save_complete, .has_postcopy = dirty_bitmap_has_postcopy, - .state_pending_exact = dirty_bitmap_state_pending, - .state_pending_estimate = dirty_bitmap_state_pending, + .save_query_pending = dirty_bitmap_state_pending, .save_live_iterate = dirty_bitmap_save_iterate, .is_active_iterate = dirty_bitmap_is_active_iterate, .load_state = dirty_bitmap_load, diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ static int ram_save_complete(QEMUFile *f, void *opaque) return qemu_fflush(f); } -static void ram_state_pending_estimate(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy) -{ - RAMState **temp = opaque; - RAMState *rs = *temp; - - uint64_t remaining_size = rs->migration_dirty_pages * TARGET_PAGE_SIZE; - - if (migrate_postcopy_ram()) { - /* We can do postcopy, and all the data is postcopiable */ - *can_postcopy += remaining_size; - } else { - *must_precopy += remaining_size; - } -} - -static void ram_state_pending_exact(void *opaque, uint64_t *must_precopy, - uint64_t *can_postcopy) +static void ram_state_pending(void *opaque, MigPendingData *pending, + bool exact) { RAMState **temp = opaque; RAMState *rs = *temp; uint64_t remaining_size; - if (!migration_in_postcopy()) { + /* + * Sync is not needed either with: (1) a fast query, or (2) after + * postcopy has started (no new dirty will generate anymore). + */ + if (exact && !migration_in_postcopy()) { bql_lock(); WITH_RCU_READ_LOCK_GUARD() { migration_bitmap_sync_precopy(false); @@ -XXX,XX +XXX,XX @@ static void ram_state_pending_exact(void *opaque, uint64_t *must_precopy, if (migrate_postcopy_ram()) { /* We can do postcopy, and all the data is postcopiable */ - *can_postcopy += remaining_size; + pending->postcopy_bytes += remaining_size; } else { - *must_precopy += remaining_size; + pending->precopy_bytes += remaining_size; } } @@ -XXX,XX +XXX,XX @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_iterate = ram_save_iterate, .save_complete = ram_save_complete, .has_postcopy = ram_has_postcopy, - .state_pending_exact = ram_state_pending_exact, - .state_pending_estimate = ram_state_pending_estimate, + .save_query_pending = ram_state_pending, .load_state = ram_load, .save_cleanup = ram_save_cleanup, .load_setup = ram_load_setup, diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ int qemu_savevm_state_complete_precopy(MigrationState *s) return qemu_fflush(f); } -/* Give an estimate of the amount left to be transferred, - * the result is split into the amount for units that can and - * for units that can't do postcopy. - */ -void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, - uint64_t *can_postcopy) +void qemu_savevm_query_pending(MigPendingData *pending, bool exact) { SaveStateEntry *se; - *must_precopy = 0; - *can_postcopy = 0; + pending->precopy_bytes = 0; + pending->postcopy_bytes = 0; QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { - if (!se->ops || !se->ops->state_pending_estimate) { + if (!se->ops || !se->ops->save_query_pending) { continue; } if (!qemu_savevm_state_active(se)) { continue; } - se->ops->state_pending_estimate(se->opaque, must_precopy, can_postcopy); + se->ops->save_query_pending(se->opaque, pending, exact); } } +void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, + uint64_t *can_postcopy) +{ + MigPendingData pending; + + qemu_savevm_query_pending(&pending, false); + + *must_precopy = pending.precopy_bytes; + *can_postcopy = pending.postcopy_bytes; +} + void qemu_savevm_state_pending_exact(uint64_t *must_precopy, uint64_t *can_postcopy) { - SaveStateEntry *se; + MigPendingData pending; - *must_precopy = 0; - *can_postcopy = 0; + qemu_savevm_query_pending(&pending, true); - QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { - if (!se->ops || !se->ops->state_pending_exact) { - continue; - } - if (!qemu_savevm_state_active(se)) { - continue; - } - se->ops->state_pending_exact(se->opaque, must_precopy, can_postcopy); - } + *must_precopy = pending.precopy_bytes; + *can_postcopy = pending.postcopy_bytes; } void qemu_savevm_state_cleanup(void) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -XXX,XX +XXX,XX @@ vfio_save_device_config_state(const char *name) " (%s)" vfio_save_iterate(const char *name, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy initial size %"PRIu64" precopy dirty size %"PRIu64 vfio_save_iterate_start(const char *name) " (%s)" vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data buffer size %"PRIu64 -vfio_state_pending_estimate(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy %"PRIu64" postcopy %"PRIu64" precopy initial size %"PRIu64" precopy dirty size %"PRIu64 -vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy %"PRIu64" postcopy %"PRIu64" stopcopy size %"PRIu64" precopy initial size %"PRIu64" precopy dirty size %"PRIu64 +vfio_state_pending(const char *name, uint64_t stopcopy_size, uint64_t precopy_init_size, uint64_t precopy_dirty_size, bool exact) " (%s) stopcopy size %"PRIu64" precopy initial size %"PRIu64" precopy dirty size %"PRIu64 " exact %d" vfio_vmstate_change(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" vfio_vmstate_change_prepare(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" -- 2.53.0 It's easier to use the new API directly in the migration iterations. This also paves way for follow up patches to add new data to report directly to the iterator function. When at it, merge the tracepoints too into one. No functional change intended. Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Avihai Horon <avihaih@nvidia.com> Link: https://lore.kernel.org/r/20260421202110.306051-7-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/savevm.h | 4 ---- migration/migration.c | 16 +++++++--------- migration/savevm.c | 23 ++--------------------- migration/trace-events | 3 +-- 4 files changed, 10 insertions(+), 36 deletions(-) diff --git a/migration/savevm.h b/migration/savevm.h index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -XXX,XX +XXX,XX @@ void qemu_savevm_state_cleanup(void); void qemu_savevm_state_complete_postcopy(QEMUFile *f); int qemu_savevm_state_complete_precopy(MigrationState *s); void qemu_savevm_query_pending(MigPendingData *pending, bool exact); -void qemu_savevm_state_pending_exact(uint64_t *must_precopy, - uint64_t *can_postcopy); -void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, - uint64_t *can_postcopy); int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy); bool qemu_savevm_state_postcopy_prepare(QEMUFile *f, Error **errp); void qemu_savevm_state_end(QEMUFile *f); diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ typedef enum { */ static MigIterateState migration_iteration_run(MigrationState *s) { - uint64_t must_precopy, can_postcopy, pending_size; Error *local_err = NULL; bool in_postcopy = (s->state == MIGRATION_STATUS_POSTCOPY_DEVICE || s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE); bool can_switchover = migration_can_switchover(s); + MigPendingData pending = { }; + uint64_t pending_size; bool complete_ready; /* Fast path - get the estimated amount of pending data */ - qemu_savevm_state_pending_estimate(&must_precopy, &can_postcopy); - pending_size = must_precopy + can_postcopy; - trace_migrate_pending_estimate(pending_size, must_precopy, can_postcopy); + qemu_savevm_query_pending(&pending, false); + pending_size = pending.precopy_bytes + pending.postcopy_bytes; if (in_postcopy) { /* @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) * during postcopy phase. */ if (pending_size <= s->threshold_size) { - qemu_savevm_state_pending_exact(&must_precopy, &can_postcopy); - pending_size = must_precopy + can_postcopy; - trace_migrate_pending_exact(pending_size, must_precopy, - can_postcopy); + qemu_savevm_query_pending(&pending, true); + pending_size = pending.precopy_bytes + pending.postcopy_bytes; } /* Should we switch to postcopy now? */ - if (must_precopy <= s->threshold_size && + if (pending.precopy_bytes <= s->threshold_size && can_switchover && qatomic_read(&s->start_postcopy)) { if (postcopy_start(s, &local_err)) { migrate_error_propagate(s, error_copy(local_err)); diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ void qemu_savevm_query_pending(MigPendingData *pending, bool exact) } se->ops->save_query_pending(se->opaque, pending, exact); } -} - -void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, - uint64_t *can_postcopy) -{ - MigPendingData pending; - - qemu_savevm_query_pending(&pending, false); - - *must_precopy = pending.precopy_bytes; - *can_postcopy = pending.postcopy_bytes; -} - -void qemu_savevm_state_pending_exact(uint64_t *must_precopy, - uint64_t *can_postcopy) -{ - MigPendingData pending; - - qemu_savevm_query_pending(&pending, true); - *must_precopy = pending.precopy_bytes; - *can_postcopy = pending.postcopy_bytes; + trace_qemu_savevm_query_pending(exact, pending->precopy_bytes, + pending->postcopy_bytes); } void qemu_savevm_state_cleanup(void) diff --git a/migration/trace-events b/migration/trace-events index XXXXXXX..XXXXXXX 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -XXX,XX +XXX,XX @@ qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" qemu_savevm_send_packaged(void) "" +qemu_savevm_query_pending(bool exact, uint64_t precopy, uint64_t postcopy) "exact=%d, precopy=%"PRIu64", postcopy=%"PRIu64 loadvm_state_switchover_ack_needed(unsigned int switchover_ack_pending_num) "Switchover ack pending num=%u" loadvm_state_setup(void) "" loadvm_state_cleanup(void) "" @@ -XXX,XX +XXX,XX @@ migration_cleanup(void) "" migrate_error(const char *error_desc) "error=%s" migration_cancel(void) "" migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at 0x%zx len 0x%zx" -migrate_pending_exact(uint64_t size, uint64_t pre, uint64_t post) "exact pending size %" PRIu64 " (pre = %" PRIu64 " post=%" PRIu64 ")" -migrate_pending_estimate(uint64_t size, uint64_t pre, uint64_t post) "estimate pending size %" PRIu64 " (pre = %" PRIu64 " post=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migrate_send_rp_recv_bitmap(char *name, int64_t size) "block '%s' size 0x%"PRIi64 migration_completion_file_err(void) "" -- 2.53.0 Allow modules to report data that can only be migrated after VM is stopped. When this concept is introduced, we will need to account stopcopy size to be part of pending_size as before. However, when there're data only can be migrated in stopcopy phase, it means the old "pending_size" may not always be able to reach low enough to kickoff an slow version of query sync. It used to be almost guaranteed to happen as all prior iterative modules doesn't have stopcopy only data. VFIO may change that fact by having some data that must be copied during stop phase. So we need to make sure QEMU will kickoff a synchronized version of query pending when all precopy data is migrated. This might be important to VFIO to keep making progress even if the downtime cannot yet be satisfied. So far, this patch should introduce no functional change, as no module yet report stopcopy size. This paves way for VFIO to properly report its pending data sizes, which will start to include stop-only data. Reviewed-by: Avihai Horon <avihaih@nvidia.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-8-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- include/migration/register.h | 7 ++++ migration/migration.c | 65 ++++++++++++++++++++++++++++++------ migration/savevm.c | 10 ++++-- migration/trace-events | 2 +- 4 files changed, 70 insertions(+), 14 deletions(-) diff --git a/include/migration/register.h b/include/migration/register.h index XXXXXXX..XXXXXXX 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -XXX,XX +XXX,XX @@ typedef struct MigPendingData { uint64_t precopy_bytes; /* Amount of pending bytes can be transferred in postcopy */ uint64_t postcopy_bytes; + /* Amount of pending bytes can be transferred only in stopcopy */ + uint64_t stopcopy_bytes; + /* + * Total pending data, modules do not need to update this field, it + * will be automatically calculated by migration core API. + */ + uint64_t total_bytes; } MigPendingData; /** diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ typedef enum { MIG_ITERATE_BREAK, /* Break the loop */ } MigIterateState; +/* Are we ready to move to the next iteration phase? */ +static bool migration_iteration_next_ready(MigrationState *s, + MigPendingData *pending) +{ + /* + * If the estimated values already suggest us to switchover, mark this + * iteration finished, time to do a slow sync. + */ + if (pending->total_bytes <= s->threshold_size) { + return true; + } + + /* + * Since we may have modules reporting stop-only data, we also want to + * re-query with slow mode if all precopy data is moved over. This + * will also mark the current iteration done. + * + * This could happen when e.g. a module (like, VFIO) reports stopcopy + * size too large so it will never yet satisfy the downtime with the + * current setup (above check). Here, slow version of re-query helps + * because we keep trying the best to move whatever we have. + */ + if (pending->precopy_bytes == 0) { + return true; + } + + return false; +} + +static void migration_iteration_go_next(MigPendingData *pending) +{ + /* + * Do a slow sync will achieve this. TODO: move RAM iteration code + * into the core layer. + */ + qemu_savevm_query_pending(pending, true); +} + +static bool postcopy_should_start(MigrationState *s, MigPendingData *pending) +{ + /* If postcopy's switchver will violate user specified downtime, stop */ + if (pending->precopy_bytes + pending->stopcopy_bytes > s->threshold_size) { + return false; + } + + return qatomic_read(&s->start_postcopy); +} + /* * Return true if continue to the next iteration directly, false * otherwise. @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE); bool can_switchover = migration_can_switchover(s); MigPendingData pending = { }; - uint64_t pending_size; bool complete_ready; /* Fast path - get the estimated amount of pending data */ qemu_savevm_query_pending(&pending, false); - pending_size = pending.precopy_bytes + pending.postcopy_bytes; if (in_postcopy) { /* @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) * postcopy completion doesn't rely on can_switchover, because when * POSTCOPY_ACTIVE it means switchover already happened. */ - complete_ready = !pending_size; + complete_ready = !pending.total_bytes; if (s->state == MIGRATION_STATUS_POSTCOPY_DEVICE && (s->postcopy_package_loaded || complete_ready)) { /* @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) * postcopy started, so ESTIMATE should always match with EXACT * during postcopy phase. */ - if (pending_size <= s->threshold_size) { - qemu_savevm_query_pending(&pending, true); - pending_size = pending.precopy_bytes + pending.postcopy_bytes; + if (migration_iteration_next_ready(s, &pending)) { + migration_iteration_go_next(&pending); } /* Should we switch to postcopy now? */ - if (pending.precopy_bytes <= s->threshold_size && - can_switchover && qatomic_read(&s->start_postcopy)) { + if (can_switchover && postcopy_should_start(s, &pending)) { if (postcopy_start(s, &local_err)) { migrate_error_propagate(s, error_copy(local_err)); error_report_err(local_err); @@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s) * (2) Pending size is no more than the threshold specified * (which was calculated from expected downtime) */ - complete_ready = can_switchover && (pending_size <= s->threshold_size); + complete_ready = can_switchover && + (pending.total_bytes <= s->threshold_size); } if (complete_ready) { - trace_migration_thread_low_pending(pending_size); + trace_migration_thread_low_pending(pending.total_bytes); migration_completion(s); return MIG_ITERATE_BREAK; } diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ void qemu_savevm_query_pending(MigPendingData *pending, bool exact) { SaveStateEntry *se; - pending->precopy_bytes = 0; - pending->postcopy_bytes = 0; + memset(pending, 0, sizeof(*pending)); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { if (!se->ops || !se->ops->save_query_pending) { @@ -XXX,XX +XXX,XX @@ void qemu_savevm_query_pending(MigPendingData *pending, bool exact) se->ops->save_query_pending(se->opaque, pending, exact); } + pending->total_bytes = pending->precopy_bytes + + pending->stopcopy_bytes + pending->postcopy_bytes; + trace_qemu_savevm_query_pending(exact, pending->precopy_bytes, - pending->postcopy_bytes); + pending->stopcopy_bytes, + pending->postcopy_bytes, + pending->total_bytes); } void qemu_savevm_state_cleanup(void) diff --git a/migration/trace-events b/migration/trace-events index XXXXXXX..XXXXXXX 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -XXX,XX +XXX,XX @@ qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" qemu_savevm_send_packaged(void) "" -qemu_savevm_query_pending(bool exact, uint64_t precopy, uint64_t postcopy) "exact=%d, precopy=%"PRIu64", postcopy=%"PRIu64 +qemu_savevm_query_pending(bool exact, uint64_t precopy, uint64_t stopcopy, uint64_t postcopy, uint64_t total) "exact=%d, precopy=%"PRIu64", stopcopy=%"PRIu64", postcopy=%"PRIu64", total=%"PRIu64 loadvm_state_switchover_ack_needed(unsigned int switchover_ack_pending_num) "Switchover ack pending num=%u" loadvm_state_setup(void) "" loadvm_state_cleanup(void) "" -- 2.53.0 VFIO reports different things in its fast/slow version of query pending results. It was because it wants to make sure precopy data can reach 0, which is needed to make sure sync queries will happen periodically over time. Now with stopcopy size reporting facility it doesn't need this hack anymore. Fix this by reporting the same values in fast/slow versions of query pending request, except that the slow version will do a slow sync with the hardwares. When at it, removing the special casing for vfio_device_state_is_precopy() which may reporting nothing in a fast query. Then ther reporting will be consistent to VFIO devices that do not support precopy phase. Copy stable might be too much; just skip it and skip the Fixes. Reviewed-by: Avihai Horon <avihaih@nvidia.com> Tested-by: Avihai Horon <avihaih@nvidia.com> Link: https://lore.kernel.org/r/20260421202110.306051-9-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/vfio/migration.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -XXX,XX +XXX,XX @@ static void vfio_state_pending(void *opaque, MigPendingData *pending, { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - uint64_t remain; + uint64_t precopy_size, stopcopy_size; if (exact) { vfio_state_pending_sync(vbasedev); - remain = migration->stopcopy_size; + } + + precopy_size = + migration->precopy_init_size + migration->precopy_dirty_size; + + if (migration->stopcopy_size > precopy_size) { + stopcopy_size = migration->stopcopy_size - precopy_size; } else { - if (!vfio_device_state_is_precopy(vbasedev)) { - return; - } - remain = migration->precopy_init_size + migration->precopy_dirty_size; + stopcopy_size = 0; } - pending->precopy_bytes += remain; + pending->precopy_bytes += precopy_size; + pending->stopcopy_bytes += stopcopy_size; trace_vfio_state_pending(vbasedev->name, migration->stopcopy_size, migration->precopy_init_size, -- 2.53.0 It used to hide in RAM dirty sync path. Now with more modules being able to slow sync on dirty information, keeping it there may not be good anymore because it's not RAM's own concept for iterations: all modules should follow. More importantly, mgmt may try to query dirty info (to make policy decisions like adjusting downtime) by listening to iteration count changes via QMP events. So we must make sure the boost of iterations only happens _after_ the dirty sync operations with whatever form (RAM's dirty bitmap sync, or VFIO's different ioctls to fetch latest dirty info from kernel). Move this to core migration path to manage, together with the event generation, so that it can be well ordered with the sync operations for all modules. This brings a good side effect that we should have an old issue regarding to cpu_throttle_dirty_sync_timer_tick() which can randomly boost iteration counts (because it invokes sync ops). Now it won't, which is actually the right behavior. Said that, we have code (not only QEMU, but likely mgmt too) assuming the 1st iteration will always shows dirty count to 1. Make it initialized with 1 this time, because we'll miss the dirty sync for setup() on boosting this counter now. Reviewed-by: Hyman Huang <yong.huang@smartx.com> Reviewed-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-10-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration-stats.h | 3 ++- migration/migration.c | 29 ++++++++++++++++++++++++++--- migration/ram.c | 6 ------ 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/migration/migration-stats.h b/migration/migration-stats.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -XXX,XX +XXX,XX @@ typedef struct { */ uint64_t dirty_pages_rate; /* - * Number of times we have synchronized guest bitmaps. + * Number of times we have synchronized guest bitmaps. This always + * starts from 1 for the 1st iteration. */ uint64_t dirty_sync_count; /* diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ int migrate_init(MigrationState *s, Error **errp) s->threshold_size = 0; s->switchover_acked = false; s->rdma_migration = false; + /* - * set mig_stats memory to zero for a new migration + * set mig_stats memory to zero for a new migration.. except the + * iteration counter, which we want to make sure it returns 1 for the + * first iteration. */ memset(&mig_stats, 0, sizeof(mig_stats)); + mig_stats.dirty_sync_count = 1; + migration_reset_vfio_bytes_transferred(); s->postcopy_package_loaded = false; @@ -XXX,XX +XXX,XX @@ static bool migration_iteration_next_ready(MigrationState *s, static void migration_iteration_go_next(MigPendingData *pending) { /* - * Do a slow sync will achieve this. TODO: move RAM iteration code - * into the core layer. + * Do a slow sync first before boosting the iteration count. */ qemu_savevm_query_pending(pending, true); + + /* + * Boost dirty sync count to reflect we finished one iteration. + * + * NOTE: we need to make sure when this happens (together with the + * event sent below) all modules have slow-synced the pending data + * above. That means a write mem barrier, but qatomic_add() should be + * enough. + * + * It's because a mgmt could wait on the iteration event to query again + * on pending data for policy changes (e.g. downtime adjustments). The + * ordering will make sure the query will fetch the latest results from + * all the modules. + */ + qatomic_add(&mig_stats.dirty_sync_count, 1); + + if (migrate_events()) { + qapi_event_send_migration_pass(mig_stats.dirty_sync_count); + } } static bool postcopy_should_start(MigrationState *s, MigPendingData *pending) diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) RAMBlock *block; int64_t end_time; - qatomic_add(&mig_stats.dirty_sync_count, 1); - if (!rs->time_last_bitmap_sync) { rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); } @@ -XXX,XX +XXX,XX @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) rs->num_dirty_pages_period = 0; rs->bytes_xfer_prev = migration_transferred_bytes(); } - if (migrate_events()) { - uint64_t generation = qatomic_read(&mig_stats.dirty_sync_count); - qapi_event_send_migration_pass(generation); - } } void migration_bitmap_sync_precopy(bool last_stage) -- 2.53.0 Add a helper migration_get_switchover_bw() to return an estimate of switchover bandwidth. Use it to simplify the current code. This will be used in later to remove expected_downtime. When at it, remove two qatomic_read() to shrink the lines because atomic ops are not needed when it's always the same thread who does the updates. Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-11-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration.c | 48 +++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value) migrate_send_rp_message(mis, MIG_RP_MSG_RESUME_ACK, sizeof(buf), &buf); } +/* + * Returns the estimated switchover bandwidth (unit: bytes / seconds) + */ +static double migration_get_switchover_bw(MigrationState *s) +{ + uint64_t switchover_bw = migrate_avail_switchover_bandwidth(); + + if (switchover_bw) { + /* If user specified, prioritize this value and don't estimate */ + return (double)switchover_bw; + } + + return s->mbps / 8 * 1000 * 1000; +} + bool migration_is_running(void) { MigrationState *s = current_migration; @@ -XXX,XX +XXX,XX @@ static void migration_update_counters(MigrationState *s, { uint64_t transferred, transferred_pages, time_spent; uint64_t current_bytes; /* bytes transferred since the beginning */ - uint64_t switchover_bw; - /* Expected bandwidth when switching over to destination QEMU */ - double expected_bw_per_ms; - double bandwidth; + double switchover_bw_per_ms; if (current_time < s->iteration_start_time + BUFFER_DELAY) { return; } - switchover_bw = migrate_avail_switchover_bandwidth(); current_bytes = migration_transferred_bytes(); transferred = current_bytes - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; - bandwidth = (double)transferred / time_spent; - - if (switchover_bw) { - /* - * If the user specified a switchover bandwidth, let's trust the - * user so that can be more accurate than what we estimated. - */ - expected_bw_per_ms = (double)switchover_bw / 1000; - } else { - /* If the user doesn't specify bandwidth, we use the estimated */ - expected_bw_per_ms = bandwidth; - } - - s->threshold_size = expected_bw_per_ms * migrate_downtime_limit(); - s->mbps = (((double) transferred * 8.0) / ((double) time_spent / 1000.0)) / 1000.0 / 1000.0; + /* NOTE: only update this after bandwidth (s->mbps) updated */ + switchover_bw_per_ms = migration_get_switchover_bw(s) / 1000; + s->threshold_size = switchover_bw_per_ms * migrate_downtime_limit(); + transferred_pages = ram_get_total_transferred_pages() - s->iteration_initial_pages; s->pages_per_second = (double) transferred_pages / @@ -XXX,XX +XXX,XX @@ static void migration_update_counters(MigrationState *s, * if we haven't sent anything, we don't want to * recalculate. 10000 is a small enough number for our purposes */ - if (qatomic_read(&mig_stats.dirty_pages_rate) && - transferred > 10000) { + if (mig_stats.dirty_pages_rate && transferred > 10000) { s->expected_downtime = - qatomic_read(&mig_stats.dirty_bytes_last_sync) / expected_bw_per_ms; + mig_stats.dirty_bytes_last_sync / switchover_bw_per_ms; } migration_rate_reset(); @@ -XXX,XX +XXX,XX @@ static void migration_update_counters(MigrationState *s, trace_migrate_transferred(transferred, time_spent, /* Both in unit bytes/ms */ - bandwidth, switchover_bw / 1000, + (uint64_t)s->mbps, + (uint64_t)switchover_bw_per_ms, s->threshold_size); } -- 2.53.0 This value does not need to be calculated as frequent. Only calculate it on demand when query-migrate happened. With that we can remove the variable in MigrationState. This paves way for fixing this value to include all modules (not only RAM but others too). Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-12-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration.h | 2 +- migration/migration.c | 25 ++++++++++++------------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -XXX,XX +XXX,XX @@ struct MigrationState { /* Timestamp when VM is down (ms) to migrate the last stuff */ int64_t downtime_start; int64_t downtime; - int64_t expected_downtime; bool capabilities[MIGRATION_CAPABILITY__MAX]; int64_t setup_time; @@ -XXX,XX +XXX,XX @@ void migration_cancel(void); void migration_populate_vfio_info(MigrationInfo *info); void migration_reset_vfio_bytes_transferred(void); void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page); +int64_t migration_downtime_calc_expected(MigrationState *s); /* * Migration thread waiting for return path thread. Return non-zero if an diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ static bool migrate_show_downtime(MigrationState *s) return (s->state == MIGRATION_STATUS_COMPLETED) || migration_in_postcopy(); } +/* Return expected downtime (unit: milliseconds) */ +int64_t migration_downtime_calc_expected(MigrationState *s) +{ + if (mig_stats.dirty_sync_count <= 1) { + return migrate_downtime_limit(); + } + + return mig_stats.dirty_bytes_last_sync / + migration_get_switchover_bw(s) * 1000; +} + static void populate_time_info(MigrationInfo *info, MigrationState *s) { info->has_status = true; @@ -XXX,XX +XXX,XX @@ static void populate_time_info(MigrationInfo *info, MigrationState *s) info->downtime = s->downtime; } else { info->has_expected_downtime = true; - info->expected_downtime = s->expected_downtime; + info->expected_downtime = migration_downtime_calc_expected(s); } } @@ -XXX,XX +XXX,XX @@ int migrate_init(MigrationState *s, Error **errp) s->mbps = 0.0; s->pages_per_second = 0.0; s->downtime = 0; - s->expected_downtime = 0; s->setup_time = 0; s->start_postcopy = false; s->migration_thread_running = false; @@ -XXX,XX +XXX,XX @@ static void migration_update_counters(MigrationState *s, s->pages_per_second = (double) transferred_pages / (((double) time_spent / 1000.0)); - /* - * if we haven't sent anything, we don't want to - * recalculate. 10000 is a small enough number for our purposes - */ - if (mig_stats.dirty_pages_rate && transferred > 10000) { - s->expected_downtime = - mig_stats.dirty_bytes_last_sync / switchover_bw_per_ms; - } - migration_rate_reset(); update_iteration_initial_status(s); @@ -XXX,XX +XXX,XX @@ void migration_start_outgoing(MigrationState *s) bool resume = (s->state == MIGRATION_STATUS_POSTCOPY_RECOVER_SETUP); int ret; - s->expected_downtime = migrate_downtime_limit(); - if (resume) { /* This is a resumed migration */ rate_limit = migrate_max_postcopy_bandwidth(); -- 2.53.0 QEMU will provide an expected downtime for the whole system during migration, by remembering the total dirty RAM that we synced the last time, divides the estimated switchover bandwidth. That was flawed when VFIO is taking into account: consider there is a VFIO GPU device that contains GBs of data to migrate during stop phase. Those will not be accounted in this math. Fix it by updating dirty_bytes_last_sync properly only when we go to the next iteration, rather than hide this update in the RAM code. Meanwhile, fetch the total (rather than RAM-only) portion of dirty bytes, so as to include GPU device states too. Update the comment of the field to reflect its new meaning. Now after this change, the expected-downtime to be read from query-migrate should be very accurate even with VFIO devices involved. Tested-by: Cédric Le Goater <clg@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-13-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration-stats.h | 8 +++----- migration/migration.c | 11 ++++++++--- migration/ram.c | 1 - 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/migration/migration-stats.h b/migration/migration-stats.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -XXX,XX +XXX,XX @@ */ typedef struct { /* - * Number of bytes that were dirty last time that we synced with - * the guest memory. We use that to calculate the downtime. As - * the remaining dirty amounts to what we know that is still dirty - * since last iteration, not counting what the guest has dirtied - * since we synchronized bitmaps. + * Number of bytes that were reported dirty after the latest + * system-wise synchronization of dirty information. It is used to do + * best-effort estimation on expected downtime. */ uint64_t dirty_bytes_last_sync; /* diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ static void migration_iteration_go_next(MigPendingData *pending) */ qemu_savevm_query_pending(pending, true); + /* + * Update the dirty information for the whole system for this + * iteration. This value is used to calculate expected downtime. + */ + qatomic_set(&mig_stats.dirty_bytes_last_sync, pending->total_bytes); + /* * Boost dirty sync count to reflect we finished one iteration. * * NOTE: we need to make sure when this happens (together with the * event sent below) all modules have slow-synced the pending data - * above. That means a write mem barrier, but qatomic_add() should be - * enough. + * above and updated corresponding fields (e.g. dirty_bytes_last_sync). * * It's because a mgmt could wait on the iteration event to query again * on pending data for policy changes (e.g. downtime adjustments). The * ordering will make sure the query will fetch the latest results from - * all the modules. + * all the modules on everything. */ qatomic_add(&mig_stats.dirty_sync_count, 1); diff --git a/migration/ram.c b/migration/ram.c index XXXXXXX..XXXXXXX 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -XXX,XX +XXX,XX @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) RAMBLOCK_FOREACH_NOT_IGNORED(block) { ramblock_sync_dirty_bitmap(rs, block); } - qatomic_set(&mig_stats.dirty_bytes_last_sync, ram_bytes_remaining()); } } -- 2.53.0 Introduce this new counter to remember the total dirty bytes for the whole system. It will be used for query-migrate command to fetch system-wise remaining data. A prior attempt was made to not use this counter but query directly from all the modules in a QMP handler, but it exposed some complexity not only on migration state machine race conditions (where the query may be invoked anytime of the state machine), or on locking implications (where some of the query hooks may take BQL, which is illegal at least in a QMP handler). For more information, see: https://lore.kernel.org/r/aeZMtxqrKWAMKzdN@x1.local This oneliner will resolve everything, except that it is not as accurate. The hope is it is a worthwhile trade-off solution, after knowing above challenges. Now, there is one more reason we should make each invocation of save_live_iterate() to be lightweight, because this counter will only get updated once for each loop over all save_live_iterate() hooks when present. But that's always the goal. Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-14-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- migration/migration-stats.h | 7 +++++++ migration/savevm.c | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/migration/migration-stats.h b/migration/migration-stats.h index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -XXX,XX +XXX,XX @@ typedef struct { * best-effort estimation on expected downtime. */ uint64_t dirty_bytes_last_sync; + /* + * Number of bytes that were reported dirty now. This is an estimate + * value and will be updated every time migration thread queries from + * modules in an iteration loop. It is used to provide best-effort + * estimation on total remaining data. + */ + uint64_t dirty_bytes_total; /* * Number of pages dirtied per second. */ diff --git a/migration/savevm.c b/migration/savevm.c index XXXXXXX..XXXXXXX 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -XXX,XX +XXX,XX @@ void qemu_savevm_query_pending(MigPendingData *pending, bool exact) pending->total_bytes = pending->precopy_bytes + pending->stopcopy_bytes + pending->postcopy_bytes; + /* + * Update system remaining dirty bytes whenever QEMU queries. It will + * make the value to be not as accurate, but should still be pretty + * close to reality when this got invoked frequently while iterating. + */ + mig_stats.dirty_bytes_total = pending->total_bytes; + trace_qemu_savevm_query_pending(exact, pending->precopy_bytes, pending->stopcopy_bytes, pending->postcopy_bytes, -- 2.53.0 Currently, mgmt can only query for remaining RAM using QMP command "query-migrate" and monitor the "ram" section. There's no way to report system-wide remaining data including VFIO devices. It was not a problem before, because for a very long time RAM was the only part that matters. After VFIO migrations landed upstream, it may not be enough. There can be GPU devices that contain GBs of device states. Mgmt may want to know how much remaining for special devices like VFIO, because all of them will be accounted as VM data to migrate and will contribute to downtime in the switchover phase. Add a new "remaining" field in query-migrate results on the top level, reflecting system-wide remaining data, which will include everything like VFIO devices. This information will be useful for mgmt to implement generic way of stall detection that covers all system resources. For example, when system-wide remaining data (especially, if sampled right after each migration iteration) does not decrease anymore for a relatively long period of time, then it may imply there is a challenge of converging, mgmt can react based on how this value changes over time. Before this patch, "expected_downtime" almost played this role. For example, by monitoring "expected_downtime" at the beginning of each iteration can in most cases also reflect the progress of migration system-wide. Said that, "expected_downtime" was always calculated based on a bandwidth value that can fluctuate if avail-switchover-bandwidth is not used. This new "remaining" field will remove that part of uncertainty for mgmt no matter if avail-switchover-bandwidth is used by the mgmt. With the new field, HMP "info migrate" now reports this: (qemu) info migrate Status: active Time (ms): total=12080, setup=14, exp_down=300 Remaining: 1.36 GiB <--- this is the new line RAM info: Throughput (Mbps): 840.50 Sizes: pagesize=4 KiB, total=4.02 GiB Transfers: transferred=1.18 GiB, remain=1.36 GiB Channels: precopy=1.18 GiB, multifd=0 B, postcopy=0 B Page Types: normal=307923, zero=388148 Page Rates (pps): transfer=25660 Others: dirty_syncs=1 When VFIO is not involved, the value reported in the new field should be approximately the same as reported in the "remaining" field of the RAM section. It is only approximately because the system-wide remaining data is a cached value, which gets frequently updated by migration core. OTOH, the RAM's remaining data is accurate. When VFIO is involved, the new value reported should normally be larger, because it will include the size of VFIO remaining data too. Cc: Aseef Imran <aimran@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Dr. David Alan Gilbert <dave@treblig.org> Acked-by: Markus Armbruster <armbru@redhat.com> # QAPI schema Link: https://lore.kernel.org/r/20260421202110.306051-15-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- qapi/migration.json | 4 ++++ migration/migration-hmp-cmds.c | 5 +++++ migration/migration.c | 7 +++++++ 3 files changed, 16 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # average memory load of the virtual CPU indirectly. Note that # zero means guest doesn't dirty memory. (Since 8.1) # +# @remaining: amount of bytes remaining to be migrated system-wide, +# includes both RAM and all devices (like VFIO). (Since 11.1) +# # Features: # # @unstable: Members @postcopy-latency, @postcopy-vcpu-latency, @@ -XXX,XX +XXX,XX @@ ## { 'struct': 'MigrationInfo', 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationRAMStats', + '*remaining': 'size', '*vfio': 'VfioStats', '*xbzrle-cache': 'XBZRLECacheStats', '*total-time': 'int', diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) } } + if (info->has_remaining) { + g_autofree char *remaining = size_to_str(info->remaining); + monitor_printf(mon, "Remaining: \t\t%s\n", remaining); + } + if (info->has_socket_address) { SocketAddressList *addr; diff --git a/migration/migration.c b/migration/migration.c index XXXXXXX..XXXXXXX 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -XXX,XX +XXX,XX @@ static void populate_time_info(MigrationInfo *info, MigrationState *s) } } +static void populate_global_info(MigrationInfo *info, MigrationState *s) +{ + info->has_remaining = true; + info->remaining = qatomic_read(&mig_stats.dirty_bytes_total); +} + static void populate_ram_info(MigrationInfo *info, MigrationState *s) { size_t page_size = qemu_target_page_size(); @@ -XXX,XX +XXX,XX @@ static void fill_source_migration_info(MigrationInfo *info) /* TODO add some postcopy stats */ populate_time_info(info, s); populate_ram_info(info, s); + populate_global_info(info, s); migration_populate_vfio_info(info); break; case MIGRATION_STATUS_COLO: -- 2.53.0 Add ", in bytes per second". Unfortunately indentations need to be updated completely, but no change on the rest. Cc: Markus Armbruster <armbru@redhat.com> Suggested-by: Juraj Marcin <jmarcin@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Acked-by: Markus Armbruster <armbru@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-16-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- qapi/migration.json | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -XXX,XX +XXX,XX @@ # (Since 2.8) # # @avail-switchover-bandwidth: to set the available bandwidth that -# migration can use during switchover phase. **Note:** this does -# not limit the bandwidth during switchover, but only for -# calculations when making decisions to switchover. By default, -# this value is zero, which means QEMU will estimate the bandwidth -# automatically. This can be set when the estimated value is not -# accurate, while the user is able to guarantee such bandwidth is -# available when switching over. When specified correctly, this -# can make the switchover decision much more accurate. -# (Since 8.2) +# migration can use during switchover phase, in bytes per +# second. **Note:** this does not limit the bandwidth during +# switchover, but only for calculations when making decisions to +# switchover. By default, this value is zero, which means QEMU +# will estimate the bandwidth automatically. This can be set +# when the estimated value is not accurate, while the user is +# able to guarantee such bandwidth is available when switching +# over. When specified correctly, this can make the switchover +# decision much more accurate. (Since 8.2) # # @downtime-limit: set maximum tolerated downtime for migration. # maximum downtime in milliseconds (Since 2.8) -- 2.53.0 Add two tracepoints for both precopy and stopcopy query ioctls. When at it, add one warn_report_once() for each of them when it fails. Reviewed-by: Avihai Horon <avihaih@nvidia.com> Tested-by: Cédric Le Goater <clg@redhat.com> Reviewed-by: Cédric Le Goater <clg@redhat.com> Link: https://lore.kernel.org/r/20260421202110.306051-17-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/vfio/migration.c | 35 +++++++++++++++++++++++++---------- hw/vfio/trace-events | 2 ++ 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -XXX,XX +XXX,XX @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev) struct vfio_device_feature_mig_data_size *mig_data_size = (struct vfio_device_feature_mig_data_size *)feature->data; VFIOMigration *migration = vbasedev->migration; + int ret; feature->argsz = sizeof(buf); feature->flags = @@ -XXX,XX +XXX,XX @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev) * is reported so downtime limit won't be violated. */ migration->stopcopy_size = VFIO_MIG_STOP_COPY_SIZE; - return -errno; + ret = -errno; + warn_report_once("VFIO device %s ioctl(VFIO_DEVICE_FEATURE) on " + "VFIO_DEVICE_FEATURE_MIG_DATA_SIZE failed (%d)", + vbasedev->name, ret); + } else { + migration->stopcopy_size = mig_data_size->stop_copy_length; + ret = 0; } - migration->stopcopy_size = mig_data_size->stop_copy_length; + trace_vfio_query_stop_copy_size(vbasedev->name, + migration->stopcopy_size, ret); - return 0; + return ret; } static int vfio_query_precopy_size(VFIOMigration *migration) @@ -XXX,XX +XXX,XX @@ static int vfio_query_precopy_size(VFIOMigration *migration) struct vfio_precopy_info precopy = { .argsz = sizeof(precopy), }; - - migration->precopy_init_size = 0; - migration->precopy_dirty_size = 0; + int ret; if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)) { - return -errno; + migration->precopy_init_size = 0; + migration->precopy_dirty_size = 0; + ret = -errno; + warn_report_once("VFIO device %s ioctl(VFIO_MIG_GET_PRECOPY_INFO) " + "failed (%d)", migration->vbasedev->name, ret); + } else { + migration->precopy_init_size = precopy.initial_bytes; + migration->precopy_dirty_size = precopy.dirty_bytes; + ret = 0; } - migration->precopy_init_size = precopy.initial_bytes; - migration->precopy_dirty_size = precopy.dirty_bytes; + trace_vfio_query_precopy_size(migration->vbasedev->name, + migration->precopy_init_size, + migration->precopy_dirty_size, ret); - return 0; + return ret; } /* Returns the size of saved data on success and -errno on error */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index XXXXXXX..XXXXXXX 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -XXX,XX +XXX,XX @@ vfio_migration_realize(const char *name) " (%s)" vfio_migration_set_device_state(const char *name, const char *state) " (%s) state %s" vfio_migration_set_state(const char *name, const char *new_state, const char *recover_state) " (%s) new state %s, recover state %s" vfio_migration_state_notifier(const char *name, int state) " (%s) state %d" +vfio_query_precopy_size(const char *name, uint64_t init_size, uint64_t dirty_size, int ret) " (%s) init %"PRIu64" dirty %"PRIu64" ret %d" +vfio_query_stop_copy_size(const char *name, uint64_t size, int ret) " (%s) stopcopy size %"PRIu64" ret %d" vfio_save_block(const char *name, int data_size) " (%s) data_size %d" vfio_save_block_precopy_empty_hit(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" -- 2.53.0 From: CJ Chen <cjchen@igel.co.jp> The riscv_iommu_trap_ops MemoryRegionOps specifies that unaligned accesses are not valid for this device but that it does implement them. This doesn't make much sense, and we want to add an assertion that registered MRs don't specify this invalid combination of settings. Drop .impl.unaligned = true, with no behaviour change. Signed-off-by: CJ Chen <cjchen@igel.co.jp> Acked-by: Tomoyuki Hirose <hrstmyk811m@gmail.com> Reported-by: Tomoyuki Hirose <hrstmyk811m@gmail.com> Reviewed-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com> [PMM: reworded commit message] Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Link: https://lore.kernel.org/r/20260428093339.2087081-2-peter.maydell@linaro.org Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/riscv/riscv-iommu.c | 1 - 1 file changed, 1 deletion(-) diff --git a/hw/riscv/riscv-iommu.c b/hw/riscv/riscv-iommu.c index XXXXXXX..XXXXXXX 100644 --- a/hw/riscv/riscv-iommu.c +++ b/hw/riscv/riscv-iommu.c @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps riscv_iommu_trap_ops = { .impl = { .min_access_size = 4, .max_access_size = 8, - .unaligned = true, }, .valid = { .min_access_size = 4, -- 2.53.0 From: Peter Maydell <peter.maydell@linaro.org> Currently npcm7xx_fiu_flash_ops provides no .impl substruct; this means that it gets the default of "implements 1, 2 and 4 byte aligned accesses". This is more constrained than the device permits in its .valid substruct, and also narrower than the functions are written to handle. Add a .impl substruct matching the .valid substruct; this means that all guest accesses are handled directly by the read and write functions, and are never synthesized by the memory subsystem performing multiple accesses to the device (which would not behave correctly, as these read and write fucntions have side effects). Based-on-a-patch-by: CJ Chen <cjchen@igel.co.jp> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Link: https://lore.kernel.org/r/20260428093339.2087081-3-peter.maydell@linaro.org Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/ssi/npcm7xx_fiu.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/ssi/npcm7xx_fiu.c b/hw/ssi/npcm7xx_fiu.c index XXXXXXX..XXXXXXX 100644 --- a/hw/ssi/npcm7xx_fiu.c +++ b/hw/ssi/npcm7xx_fiu.c @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps npcm7xx_fiu_flash_ops = { .read = npcm7xx_fiu_flash_read, .write = npcm7xx_fiu_flash_write, .endianness = DEVICE_LITTLE_ENDIAN, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + .unaligned = true, + }, .valid = { .min_access_size = 1, .max_access_size = 8, -- 2.53.0 From: Peter Maydell <peter.maydell@linaro.org> The xtensa mx-pic interrupt controller has a rather odd register setup, where some registers are 32 bits but are decoded at offsets only one apart from each other. The QEMU implementation handles this correctly, but it did not set .impl.unaligned = true. This has worked up til now because QEMU has entirely ignored .impl.unaligned, and just allowed through unaligned accesses when .valid.unaligned is set. To allow the possibility of properly implementing synthesis of unaligned accesses by the memory subsystem when they are valid but the device doesn't implement them, and for clarity of intention, state explicitly that this MR's read and write functions directly handle unaligned accesses, by setting .impl.unaligned = true. While we are adjusting the MemoryRegionOps, we set also the minimum and maximum allowed access sizes. Since the only way to get at this device is via the CPU's RER and WER instructions, which always operate at 32-bit sizes (see the HELPER(rer) and HELPER(wer) functions in target/xtensa/op_helper.c), we know we will always get 32-bit accesses. Specify explicitly that that is what is valid and implemented for the MR. Add a comment to clarify that the hardware behaviour here is not "true memory-mapped registers", so the odd-looking implementation is correct. Based-on-a-patch-by: CJ Chen <cjchen@igel.co.jp> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Max Filippov <jcmvbkbc@gmail.com> Link: https://lore.kernel.org/r/20260428093339.2087081-4-peter.maydell@linaro.org Signed-off-by: Peter Xu <peterx@redhat.com> --- hw/xtensa/mx_pic.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/hw/xtensa/mx_pic.c b/hw/xtensa/mx_pic.c index XXXXXXX..XXXXXXX 100644 --- a/hw/xtensa/mx_pic.c +++ b/hw/xtensa/mx_pic.c @@ -XXX,XX +XXX,XX @@ struct XtensaMxPic { } cpu[MX_MAX_CPU]; }; +/* + * Note that decode for these registers is rather strange by the usual + * MMIO standards -- the MIROUT and MIPICAUSE areas can be read and + * written at 32-bit length, returning different values for each byte + * offset, because the low bits of the address are treated as selecting + * an IRQ or a processor: + * + * 00nn 0...0p..p Interrupt Routing, route IRQ n to processor p + * 01pp 0...0d..d 16 bits (d) 'ored' as single IPI to processor p + * + * This is because (like x86 IO port in/out accesses) the offset is + * not a memory-mapped address but is really a register number, + * accessed via the Xtensa RER/WER "external register" instructions. + * + * We set .valid and .impl to both allow unaligned = true to permit + * these byte-offsets. Because this device is not a true memory mapped + * device but is accessible only via the Xtensa RER/WER "external + * register" interface, all accesses are guaranteed 32 bits. + */ + static uint64_t xtensa_mx_pic_ext_reg_read(void *opaque, hwaddr offset, unsigned size) { @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps xtensa_mx_pic_ops = { .read = xtensa_mx_pic_ext_reg_read, .write = xtensa_mx_pic_ext_reg_write, .endianness = DEVICE_NATIVE_ENDIAN, + .impl = { + .min_access_size = 4, + .max_access_size = 4, + .unaligned = true, + }, .valid = { + .min_access_size = 4, + .max_access_size = 4, .unaligned = true, }, }; -- 2.53.0 From: CJ Chen <cjchen@igel.co.jp> When it comes to this pattern: .valid.unaligned = false and impl.unaligned = true, is effectlvely contradictory. The .valid structure indicates that unaligned access should be rejected at the access validation phase, yet .impl suggests the underlying device implementation can handle unaligned operations. As a result, the upper-layer code will never even reach the .impl logic. Add an assertion that the MemoryRegionOps doesn't specify this invalid combination. Signed-off-by: CJ Chen <cjchen@igel.co.jp> Tested-by: CJ Chen <cjchen@igel.co.jp> Suggested-by: Peter Xu <peterx@redhat.com> Acked-by: Tomoyuki Hirose <hrstmyk811m@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> [PMM: tweaked commit message] Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Link: https://lore.kernel.org/r/20260428093339.2087081-5-peter.maydell@linaro.org Signed-off-by: Peter Xu <peterx@redhat.com> --- system/memory.c | 1 + 1 file changed, 1 insertion(+) diff --git a/system/memory.c b/system/memory.c index XXXXXXX..XXXXXXX 100644 --- a/system/memory.c +++ b/system/memory.c @@ -XXX,XX +XXX,XX @@ void memory_region_init_io(MemoryRegion *mr, Object *owner, const MemoryRegionOps *ops, void *opaque, const char *name, uint64_t size) { + g_assert(!ops || !(ops->impl.unaligned && !ops->valid.unaligned)); memory_region_init(mr, owner, name, size); memory_region_set_ops(mr, ops, opaque); } -- 2.53.0 From: Fabiano Rosas <farosas@suse.de> The generation of the guest workloads for migration-test is all broken: 1) We moved code around and forgot to update the includes from the guest workloads at qtest/migration/<ARCH>/. 2) Since this code doesn't build by default due to the need for cross compilation, we also broke the s390x code entirely by removing the libc implementation from sclp.c. 3) The build adds a blank line at the end of the header files due to how the sed command used to remove extra variable declarations is written. 4) Must be too long since we last built this thing that powerpc compilers would output BE by default. Nowadays we need -mbig-endian to force it. 5) Format of comments needs fixing to please checkpatch. Fix everything. Most changes are trivial, except for s390x where I had to reimplement some of the routines from sclp.c. Compiling that object file in doesn't work because of its various includes that this standalone code can't provide. Testing this change (assuming x86 host): $ cd build $ pushd tests/qtest/migration $ make clean $ make i386 $ make CROSS_PREFIX=your-favorite-cross-compiler- s390x $ make CROSS_PREFIX=your-favorite-cross-compiler- aarch64 $ make CROSS_PREFIX=your-favorite-cross-compiler- ppc64 $ popd $ make -j$(nproc) # must build the tests again $ QTEST_QEMU_BINARY=./qemu-system-<ARCH> migration-test Fixes: 9f4278837d ("pc-bios/s390-ccw: Use the libc from SLOF and remove sclp prints") Fixes: e1803dabdc ("tests/qtest/migration: Move common test code") Signed-off-by: Fabiano Rosas <farosas@suse.de> Link: https://lore.kernel.org/r/20260429231025.30818-1-farosas@suse.de Signed-off-by: Peter Xu <peterx@redhat.com> --- tests/qtest/migration/aarch64/a-b-kernel.h | 7 +- tests/qtest/migration/bootfile.h | 11 + tests/qtest/migration/i386/a-b-bootblock.h | 8 +- tests/qtest/migration/ppc64/a-b-kernel.h | 8 +- tests/qtest/migration/s390x/a-b-bios.h | 611 +++++++++++++-------- tests/qtest/migration/s390x/a-b-bios.c | 66 ++- tests/qtest/migration/Makefile | 11 +- tests/qtest/migration/aarch64/Makefile | 4 +- tests/qtest/migration/aarch64/a-b-kernel.S | 2 +- tests/qtest/migration/i386/Makefile | 4 +- tests/qtest/migration/i386/a-b-bootblock.S | 2 +- tests/qtest/migration/ppc64/Makefile | 6 +- tests/qtest/migration/ppc64/a-b-kernel.S | 2 +- tests/qtest/migration/s390x/Makefile | 8 +- 14 files changed, 492 insertions(+), 258 deletions(-) diff --git a/tests/qtest/migration/aarch64/a-b-kernel.h b/tests/qtest/migration/aarch64/a-b-kernel.h index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/aarch64/a-b-kernel.h +++ b/tests/qtest/migration/aarch64/a-b-kernel.h @@ -XXX,XX +XXX,XX @@ -/* This file is automatically generated from the assembly file in - * tests/migration/aarch64. Edit that file and then run "make all" - * inside tests/migration to update, and then remember to send both +/* + * This file is automatically generated from the assembly file in + * tests/qtest/migration/aarch64. Edit that file and then run "make" + * inside tests/qtest/migration to update, and then remember to send both * the header and the assembler differences in your patch submission. */ unsigned char aarch64_kernel[] = { diff --git a/tests/qtest/migration/bootfile.h b/tests/qtest/migration/bootfile.h index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/bootfile.h +++ b/tests/qtest/migration/bootfile.h @@ -XXX,XX +XXX,XX @@ #ifndef BOOTFILE_H #define BOOTFILE_H +/* + * This file is included from qtest, but also from the assembly code + * that generates the header file containing the guest workload. Don't + * expose the function declarations and non-standard types to it. + */ +#ifdef __ASSEMBLER__ +#define MIGRATION_GUEST_CODE +#endif + /* Common */ #define TEST_MEM_PAGE_SIZE 4096 @@ -XXX,XX +XXX,XX @@ */ #define ARM_TEST_MAX_KERNEL_SIZE (512 * 1024) +#ifndef MIGRATION_GUEST_CODE void bootfile_delete(void); char *bootfile_create(const char *arch, const char *dir, bool suspend_me); char *bootfile_get(void); +#endif #endif /* BOOTFILE_H */ diff --git a/tests/qtest/migration/i386/a-b-bootblock.h b/tests/qtest/migration/i386/a-b-bootblock.h index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/i386/a-b-bootblock.h +++ b/tests/qtest/migration/i386/a-b-bootblock.h @@ -XXX,XX +XXX,XX @@ -/* This file is automatically generated from the assembly file in - * tests/migration/i386. Edit that file and then run "make all" - * inside tests/migration to update, and then remember to send both +/* + * This file is automatically generated from the assembly file in + * tests/qtest/migration/i386. Edit that file and then run "make" + * inside tests/qtest/migration to update, and then remember to send both * the header and the assembler differences in your patch submission. */ unsigned char x86_bootsect[] = { @@ -XXX,XX +XXX,XX @@ unsigned char x86_bootsect[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x55, 0xaa }; - #define SYM_do_zero 0x00007c3d #define SYM_gdt 0x00007ca0 #define SYM_gdtdesc 0x00007cb8 diff --git a/tests/qtest/migration/ppc64/a-b-kernel.h b/tests/qtest/migration/ppc64/a-b-kernel.h index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/ppc64/a-b-kernel.h +++ b/tests/qtest/migration/ppc64/a-b-kernel.h @@ -XXX,XX +XXX,XX @@ -/* This file is automatically generated from the assembly file in - * tests/migration/ppc64. Edit that file and then run "make all" - * inside tests/migration to update, and then remember to send both +/* + * This file is automatically generated from the assembly file in + * tests/qtest/migration/ppc64. Edit that file and then run "make" + * inside tests/qtest/migration to update, and then remember to send both * the header and the assembler differences in your patch submission. */ unsigned char ppc64_kernel[] = { @@ -XXX,XX +XXX,XX @@ unsigned char ppc64_kernel[] = { 0x38, 0xa0, 0x00, 0x01, 0x38, 0xc0, 0x00, 0x42, 0x78, 0xc6, 0xc1, 0xc6, 0x44, 0x00, 0x00, 0x22, 0x4b, 0xff, 0xff, 0xcc }; - diff --git a/tests/qtest/migration/s390x/a-b-bios.h b/tests/qtest/migration/s390x/a-b-bios.h index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/s390x/a-b-bios.h +++ b/tests/qtest/migration/s390x/a-b-bios.h @@ -XXX,XX +XXX,XX @@ -/* This file is automatically generated from the a-b-bios.c file in - * tests/migration/s390x. Edit that file and then run "make all" - * inside tests/migration to update, and then remember to send both +/* + * This file is automatically generated from the assembly file in + * tests/qtest/migration/s390x. Edit that file and then run "make" + * inside tests/qtest/migration to update, and then remember to send both * the header and the assembler differences in your patch submission. */ unsigned char s390x_elf[] = { 0x7f, 0x45, 0x4c, 0x46, 0x02, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x16, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xa8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x09, 0x80, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x98, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x38, 0x00, 0x07, 0x00, 0x40, - 0x00, 0x0d, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x04, + 0x00, 0x10, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x88, 0x00, 0x00, 0x00, 0x00, @@ -XXX,XX +XXX,XX @@ unsigned char s390x_elf[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xac, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xac, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x88, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x88, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x06, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xb8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x17, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0xb8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x38, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x01, 0x18, 0x48, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0e, 0xd8, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x1e, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x30, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x01, 0x21, 0x28, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x07, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0xb8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0xb8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x20, + 0x00, 0x00, 0x0e, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0xd8, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x64, 0x74, 0xe5, 0x51, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x64, 0x74, 0xe5, 0x52, 0x00, 0x00, 0x00, 0x04, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xb8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x17, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0xb8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x38, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x01, 0x38, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0e, 0xd8, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x1e, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x28, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x28, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x2f, 0x6c, 0x69, 0x62, 0x2f, 0x6c, 0x64, 0x36, 0x34, 0x2e, 0x73, 0x6f, - 0x2e, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x2e, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x02, 0x48, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x03, 0xa8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0c, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0xf0, 0xeb, 0xef, 0xf0, 0x70, - 0x00, 0x24, 0xa7, 0xfb, 0xff, 0x60, 0xc0, 0xe5, 0x00, 0x00, 0x01, 0x5f, - 0xc0, 0x20, 0x00, 0x00, 0x02, 0xa8, 0xc0, 0xe5, 0x00, 0x00, 0x01, 0x75, - 0xa5, 0x2e, 0x00, 0x10, 0xa7, 0x19, 0x63, 0x00, 0x92, 0x00, 0x20, 0x00, - 0xa7, 0x2b, 0x10, 0x00, 0xa7, 0x17, 0xff, 0xfc, 0xa5, 0x1e, 0x00, 0x10, - 0xa7, 0x29, 0x63, 0x00, 0xe3, 0x30, 0x10, 0x00, 0x00, 0x90, 0xa7, 0x3a, - 0x00, 0x01, 0x42, 0x30, 0x10, 0x00, 0xa7, 0x1b, 0x10, 0x00, 0xa7, 0x27, - 0xff, 0xf7, 0xc0, 0x20, 0x00, 0x00, 0x02, 0x8a, 0xc0, 0xe5, 0x00, 0x00, - 0x01, 0x56, 0xa7, 0xf4, 0xff, 0xeb, 0x07, 0x07, 0xc0, 0xf0, 0x00, 0x00, - 0x4e, 0x5c, 0xc0, 0x20, 0x00, 0x00, 0x00, 0x7d, 0xe3, 0x20, 0x20, 0x00, - 0x00, 0x04, 0xc0, 0x30, 0x00, 0x00, 0x96, 0xa3, 0xb9, 0x0b, 0x00, 0x32, - 0xb9, 0x02, 0x00, 0x33, 0xa7, 0x84, 0x00, 0x19, 0xa7, 0x3b, 0xff, 0xff, - 0xeb, 0x43, 0x00, 0x08, 0x00, 0x0c, 0xb9, 0x02, 0x00, 0x44, 0xb9, 0x04, - 0x00, 0x12, 0xa7, 0x84, 0x00, 0x09, 0xd7, 0xff, 0x10, 0x00, 0x10, 0x00, - 0x41, 0x10, 0x11, 0x00, 0xa7, 0x47, 0xff, 0xfb, 0xc0, 0x20, 0x00, 0x00, - 0x00, 0x0d, 0x44, 0x30, 0x20, 0x00, 0xc0, 0x20, 0x00, 0x00, 0x00, 0x5b, - 0xd2, 0x0f, 0x01, 0xd0, 0x20, 0x00, 0xa7, 0xf4, 0xff, 0xa1, 0xd7, 0x00, - 0x10, 0x00, 0x10, 0x00, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x50, 0xb2, 0xb2, - 0x10, 0x00, 0xa7, 0xf4, 0x00, 0x00, 0xeb, 0x00, 0xf0, 0x00, 0x00, 0x25, - 0x96, 0x02, 0xf0, 0x06, 0xeb, 0x00, 0xf0, 0x00, 0x00, 0x2f, 0xc0, 0x10, - 0x00, 0x00, 0x00, 0x2a, 0xe3, 0x10, 0x01, 0xb8, 0x00, 0x24, 0xc0, 0x10, - 0x00, 0x00, 0x00, 0x4b, 0xd2, 0x07, 0x01, 0xb0, 0x10, 0x00, 0xc0, 0x10, - 0x00, 0x00, 0x00, 0x3d, 0xb2, 0xb2, 0x10, 0x00, 0xeb, 0x66, 0xf0, 0x00, - 0x00, 0x25, 0x96, 0xff, 0xf0, 0x04, 0xeb, 0x66, 0xf0, 0x00, 0x00, 0x2f, - 0xc0, 0x10, 0x00, 0x00, 0x00, 0x1a, 0xe3, 0x10, 0x01, 0xf8, 0x00, 0x24, - 0xc0, 0x10, 0x00, 0x00, 0x00, 0x36, 0xd2, 0x07, 0x01, 0xf0, 0x10, 0x00, - 0xc0, 0x10, 0x00, 0x00, 0x00, 0x24, 0xb2, 0xb2, 0x10, 0x00, 0xeb, 0x00, - 0xf0, 0x00, 0x00, 0x25, 0x94, 0xfd, 0xf0, 0x06, 0xeb, 0x00, 0xf0, 0x00, - 0x00, 0x2f, 0x07, 0xfe, 0xeb, 0x66, 0xf0, 0x00, 0x00, 0x25, 0x94, 0x00, - 0xf0, 0x04, 0xeb, 0x66, 0xf0, 0x00, 0x00, 0x2f, 0x07, 0xfe, 0x07, 0x07, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0xf0, 0x00, 0x02, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x07, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x70, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0x00, + 0xeb, 0xdf, 0xf0, 0x68, 0x00, 0x24, 0xc0, 0xd0, 0x00, 0x00, 0x01, 0x05, + 0xa7, 0xfb, 0xff, 0x60, 0xc0, 0x10, 0x00, 0x00, 0x96, 0xc0, 0xa7, 0x28, + 0x00, 0x1c, 0x40, 0x20, 0x10, 0x00, 0xa5, 0x2c, 0x00, 0x04, 0xe3, 0x20, + 0x10, 0x0a, 0x00, 0x24, 0xa7, 0x28, 0x00, 0x40, 0x40, 0x20, 0x10, 0x12, + 0x58, 0x20, 0xd0, 0x00, 0xb2, 0x20, 0x00, 0x21, 0xb2, 0x22, 0x00, 0x30, + 0x88, 0x30, 0x00, 0x1c, 0xc0, 0xe5, 0x00, 0x00, 0x00, 0x63, 0xa7, 0x29, + 0x00, 0x41, 0xc0, 0xe5, 0x00, 0x00, 0x00, 0xbf, 0xa5, 0x2e, 0x00, 0x10, + 0xa7, 0x19, 0x63, 0x00, 0x92, 0x00, 0x20, 0x00, 0xa7, 0x2b, 0x10, 0x00, + 0xa7, 0x17, 0xff, 0xfc, 0xa5, 0x1e, 0x00, 0x10, 0xa7, 0x29, 0x63, 0x00, + 0xe3, 0x30, 0x10, 0x00, 0x00, 0x90, 0xa7, 0x3a, 0x00, 0x01, 0x42, 0x30, + 0x10, 0x00, 0xa7, 0x1b, 0x10, 0x00, 0xa7, 0x27, 0xff, 0xf7, 0xa7, 0x29, + 0x00, 0x42, 0xc0, 0xe5, 0x00, 0x00, 0x00, 0xa1, 0xa7, 0xf4, 0xff, 0xec, + 0xc0, 0xf0, 0x00, 0x00, 0x56, 0x30, 0xc0, 0x20, 0x00, 0x00, 0x0e, 0x7d, + 0xe3, 0x20, 0x20, 0x00, 0x00, 0x04, 0xc0, 0x30, 0x00, 0x00, 0x9e, 0x77, + 0xb9, 0x0b, 0x00, 0x32, 0xb9, 0x02, 0x00, 0x33, 0xa7, 0x84, 0x00, 0x19, + 0xa7, 0x3b, 0xff, 0xff, 0xeb, 0x43, 0x00, 0x08, 0x00, 0x0c, 0xb9, 0x02, + 0x00, 0x44, 0xb9, 0x04, 0x00, 0x12, 0xa7, 0x84, 0x00, 0x09, 0xd7, 0xff, + 0x10, 0x00, 0x10, 0x00, 0x41, 0x10, 0x11, 0x00, 0xa7, 0x47, 0xff, 0xfb, + 0xc0, 0x20, 0x00, 0x00, 0x00, 0x0d, 0x44, 0x30, 0x20, 0x00, 0xc0, 0x20, + 0x00, 0x00, 0x00, 0x57, 0xd2, 0x0f, 0x01, 0xd0, 0x20, 0x00, 0xa7, 0xf4, + 0xff, 0x89, 0xd7, 0x00, 0x10, 0x00, 0x10, 0x00, 0xc0, 0x10, 0x00, 0x00, + 0x00, 0x4c, 0xb2, 0xb2, 0x10, 0x00, 0xa7, 0xf4, 0x00, 0x00, 0xeb, 0x00, + 0xf0, 0x00, 0x00, 0x25, 0x96, 0x02, 0xf0, 0x06, 0xeb, 0x00, 0xf0, 0x00, + 0x00, 0x2f, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x2a, 0xe3, 0x10, 0x01, 0xb8, + 0x00, 0x24, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x47, 0xd2, 0x07, 0x01, 0xb0, + 0x10, 0x00, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x39, 0xb2, 0xb2, 0x10, 0x00, + 0xeb, 0x66, 0xf0, 0x00, 0x00, 0x25, 0x96, 0xff, 0xf0, 0x04, 0xeb, 0x66, + 0xf0, 0x00, 0x00, 0x2f, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x1a, 0xe3, 0x10, + 0x01, 0xf8, 0x00, 0x24, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x32, 0xd2, 0x07, + 0x01, 0xf0, 0x10, 0x00, 0xc0, 0x10, 0x00, 0x00, 0x00, 0x20, 0xb2, 0xb2, + 0x10, 0x00, 0xeb, 0x00, 0xf0, 0x00, 0x00, 0x25, 0x94, 0xfd, 0xf0, 0x06, + 0xeb, 0x00, 0xf0, 0x00, 0x00, 0x2f, 0x07, 0xfe, 0xeb, 0x66, 0xf0, 0x00, + 0x00, 0x25, 0x94, 0x00, 0xf0, 0x04, 0xeb, 0x66, 0xf0, 0x00, 0x00, 0x2f, + 0x07, 0xfe, 0x07, 0x07, 0x00, 0x02, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x02, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x03, 0x02, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, 0xeb, 0xbf, 0xf0, 0x58, - 0x00, 0x24, 0xc0, 0x10, 0x00, 0x00, 0x4e, 0x0d, 0xa7, 0xfb, 0xff, 0x60, - 0xb2, 0x20, 0x00, 0x21, 0xb2, 0x22, 0x00, 0xb0, 0x88, 0xb0, 0x00, 0x1c, - 0xc0, 0xe5, 0xff, 0xff, 0xff, 0x91, 0xa7, 0xbe, 0x00, 0x03, 0xa7, 0x84, - 0x00, 0x13, 0xa7, 0xbe, 0x00, 0x02, 0xa7, 0x28, 0x00, 0x00, 0xa7, 0x74, - 0x00, 0x04, 0xa7, 0x28, 0xff, 0xfe, 0xe3, 0x40, 0xf1, 0x10, 0x00, 0x04, - 0xb9, 0x14, 0x00, 0x22, 0xeb, 0xbf, 0xf0, 0xf8, 0x00, 0x04, 0x07, 0xf4, - 0xa7, 0x28, 0xff, 0xff, 0xa7, 0xf4, 0xff, 0xf5, 0x07, 0x07, 0x07, 0x07, - 0xeb, 0xbf, 0xf0, 0x58, 0x00, 0x24, 0xc0, 0xd0, 0x00, 0x00, 0x01, 0x25, - 0xa7, 0xfb, 0xff, 0x60, 0xa7, 0xb9, 0x00, 0x00, 0xa7, 0x19, 0x00, 0x00, - 0xc0, 0x40, 0x00, 0x00, 0x4d, 0xd8, 0xa7, 0x3b, 0x00, 0x01, 0xa7, 0x37, - 0x00, 0x23, 0xc0, 0x20, 0x00, 0x00, 0x4d, 0xd1, 0x18, 0x31, 0xa7, 0x1a, - 0x00, 0x06, 0x40, 0x10, 0x20, 0x08, 0xa7, 0x3a, 0x00, 0x0e, 0xa7, 0x18, - 0x1a, 0x00, 0x40, 0x30, 0x20, 0x00, 0x92, 0x00, 0x20, 0x02, 0x40, 0x10, - 0x20, 0x0a, 0xe3, 0x20, 0xd0, 0x00, 0x00, 0x04, 0xc0, 0xe5, 0xff, 0xff, - 0xff, 0xac, 0xe3, 0x40, 0xf1, 0x10, 0x00, 0x04, 0xb9, 0x04, 0x00, 0x2b, - 0xeb, 0xbf, 0xf0, 0xf8, 0x00, 0x04, 0x07, 0xf4, 0xb9, 0x04, 0x00, 0x51, - 0xa7, 0x5b, 0x00, 0x01, 0xa7, 0x09, 0x0f, 0xf7, 0xb9, 0x21, 0x00, 0x50, - 0xa7, 0x24, 0xff, 0xd7, 0x41, 0xeb, 0x20, 0x00, 0x95, 0x0a, 0xe0, 0x00, - 0xa7, 0x74, 0x00, 0x08, 0x41, 0x11, 0x40, 0x0e, 0x92, 0x0d, 0x10, 0x00, - 0xb9, 0x04, 0x00, 0x15, 0x43, 0x5b, 0x20, 0x00, 0x42, 0x51, 0x40, 0x0e, - 0xa7, 0xbb, 0x00, 0x01, 0x41, 0x10, 0x10, 0x01, 0xa7, 0xf4, 0xff, 0xbf, - 0xc0, 0x50, 0x00, 0x00, 0x00, 0xd8, 0xc0, 0x10, 0x00, 0x00, 0x4d, 0x8d, - 0xa7, 0x48, 0x00, 0x1c, 0x40, 0x40, 0x10, 0x00, 0x50, 0x20, 0x10, 0x0c, - 0xa7, 0x48, 0x00, 0x04, 0xe3, 0x20, 0x50, 0x00, 0x00, 0x04, 0x40, 0x40, - 0x10, 0x0a, 0x50, 0x30, 0x10, 0x10, 0xc0, 0xf4, 0xff, 0xff, 0xff, 0x6b, - 0xa7, 0x39, 0x00, 0x40, 0xa7, 0x29, 0x00, 0x00, 0xc0, 0xf4, 0xff, 0xff, - 0xff, 0xe4, 0x07, 0x07, 0xb9, 0x04, 0x00, 0x13, 0xa7, 0x2a, 0xff, 0xff, - 0xb9, 0x04, 0x00, 0x34, 0xa7, 0x48, 0x00, 0x01, 0x15, 0x24, 0xa7, 0x24, - 0x00, 0x07, 0xb9, 0x04, 0x00, 0x21, 0xc0, 0xf4, 0xff, 0xff, 0xff, 0x7f, - 0xa7, 0x29, 0xff, 0xff, 0x07, 0xfe, 0x07, 0x07, 0xa7, 0x39, 0x00, 0x00, - 0x41, 0x13, 0x20, 0x00, 0x95, 0x00, 0x10, 0x00, 0xa7, 0x74, 0x00, 0x05, - 0xc0, 0xf4, 0xff, 0xff, 0xff, 0x70, 0xa7, 0x3b, 0x00, 0x01, 0xa7, 0xf4, - 0xff, 0xf5, 0x07, 0x07, 0xeb, 0xbf, 0xf0, 0x58, 0x00, 0x24, 0xc0, 0xd0, - 0x00, 0x00, 0x00, 0x95, 0xa7, 0xfb, 0xff, 0x60, 0xb9, 0x04, 0x00, 0xb2, - 0xa7, 0x19, 0x00, 0x20, 0xc0, 0x20, 0x00, 0x00, 0x4d, 0x40, 0x92, 0x00, - 0x20, 0x00, 0xa7, 0x2b, 0x00, 0x01, 0xa7, 0x17, 0xff, 0xfc, 0xc0, 0x10, - 0x00, 0x00, 0x4d, 0x37, 0xa7, 0x28, 0x10, 0x00, 0x40, 0x20, 0x10, 0x00, - 0xe3, 0x20, 0xd0, 0x00, 0x00, 0x04, 0xc0, 0xe5, 0xff, 0xff, 0xff, 0x1d, - 0x12, 0x22, 0xa7, 0x74, 0x00, 0x19, 0xa7, 0x19, 0x00, 0x00, 0xc0, 0x40, - 0x00, 0x00, 0x00, 0x79, 0xa7, 0x39, 0x00, 0x08, 0xc0, 0x20, 0x00, 0x00, - 0x4d, 0x2c, 0x41, 0x21, 0x20, 0x00, 0xe3, 0x20, 0x20, 0x00, 0x00, 0x90, - 0x43, 0x22, 0x40, 0x00, 0x42, 0x21, 0xb0, 0x00, 0xa7, 0x1b, 0x00, 0x01, - 0xa7, 0x37, 0xff, 0xf2, 0xe3, 0x40, 0xf1, 0x10, 0x00, 0x04, 0xeb, 0xbf, - 0xf0, 0xf8, 0x00, 0x04, 0x07, 0xf4, 0x07, 0x07, 0xeb, 0xaf, 0xf0, 0x50, - 0x00, 0x24, 0xc0, 0xd0, 0x00, 0x00, 0x00, 0x55, 0xa7, 0xfb, 0xff, 0x60, - 0xa7, 0x19, 0x0f, 0xf8, 0xb9, 0x21, 0x00, 0x31, 0xb9, 0x04, 0x00, 0xa2, - 0xa7, 0xc4, 0x00, 0x2a, 0xa7, 0xb9, 0x0f, 0xf8, 0xc0, 0x10, 0x00, 0x00, - 0x4c, 0xf6, 0xa7, 0x28, 0x10, 0x00, 0x40, 0x20, 0x10, 0x00, 0x92, 0x00, - 0x10, 0x02, 0xe3, 0x20, 0xd0, 0x00, 0x00, 0x04, 0xc0, 0xe5, 0xff, 0xff, - 0xfe, 0xda, 0xa7, 0xbb, 0x00, 0x01, 0xa7, 0x19, 0x00, 0x00, 0xa7, 0xb7, - 0x00, 0x17, 0xc0, 0x10, 0x00, 0x00, 0x4c, 0xe1, 0xe3, 0x40, 0xf1, 0x10, - 0x00, 0x04, 0xe3, 0x20, 0x10, 0x08, 0x00, 0x91, 0xa7, 0x2a, 0xff, 0xf9, - 0xb9, 0x14, 0x00, 0x22, 0xeb, 0xaf, 0xf0, 0xf0, 0x00, 0x04, 0x07, 0xf4, - 0xb9, 0x04, 0x00, 0xb3, 0xa7, 0xf4, 0xff, 0xd8, 0xc0, 0x20, 0x00, 0x00, - 0x4c, 0xcc, 0x41, 0x31, 0xa0, 0x00, 0x41, 0x21, 0x20, 0x00, 0xa7, 0x1b, - 0x00, 0x01, 0xd2, 0x00, 0x30, 0x00, 0x20, 0x0f, 0xa7, 0xf4, 0xff, 0xdd, - 0x07, 0x07, 0x07, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x76, 0x00, 0x05, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x78, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x02, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x77, 0x00, 0x05, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x20, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x3c, 0x28, 0x2b, 0x7c, 0x26, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x21, 0x24, 0x2a, 0x29, 0x3b, 0x2e, - 0x2d, 0x2f, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2c, - 0x25, 0x5f, 0x3e, 0x3f, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x60, 0x3a, 0x23, 0x40, 0x27, 0x3d, 0x22, 0x2e, 0x61, 0x62, 0x63, - 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, 0x70, 0x71, 0x72, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, - 0x79, 0x7a, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x2e, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, - 0x51, 0x52, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x53, 0x54, - 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, 0x2e, - 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x2e, 0x2e, - 0x2e, 0x2e, 0x2e, 0x2e, 0x41, 0x00, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x6f, 0xff, 0xfe, 0xf5, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xd8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x02, 0x28, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xf8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x15, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x30, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x09, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x00, 0x6f, 0xff, 0xff, 0xfb, - 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, - 0x6f, 0xff, 0xff, 0xf9, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x80, 0x00, 0x00, 0x00, 0x07, 0xfe, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, + 0xc0, 0x50, 0x00, 0x00, 0x00, 0x20, 0xc0, 0x10, 0x00, 0x00, 0x95, 0xe1, + 0x42, 0x20, 0x10, 0x0e, 0xa7, 0x28, 0x00, 0x0f, 0x40, 0x20, 0x10, 0x00, + 0x58, 0x20, 0x50, 0x04, 0x50, 0x20, 0x10, 0x08, 0x92, 0x00, 0x10, 0x02, + 0x58, 0x20, 0x50, 0x00, 0xb2, 0x20, 0x00, 0x21, 0xb2, 0x22, 0x00, 0x30, + 0x88, 0x30, 0x00, 0x1c, 0xc0, 0xf4, 0xff, 0xff, 0xff, 0x85, 0x07, 0x07, + 0x07, 0x07, 0x07, 0x07, 0x00, 0x76, 0x00, 0x05, 0x00, 0x07, 0x1a, 0x00, + 0x00, 0x78, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, @@ -XXX,XX +XXX,XX @@ unsigned char s390x_elf[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x17, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x47, 0x43, 0x43, 0x3a, - 0x20, 0x28, 0x55, 0x62, 0x75, 0x6e, 0x74, 0x75, 0x20, 0x31, 0x31, 0x2e, - 0x34, 0x2e, 0x30, 0x2d, 0x31, 0x75, 0x62, 0x75, 0x6e, 0x74, 0x75, 0x31, - 0x7e, 0x32, 0x32, 0x2e, 0x30, 0x34, 0x29, 0x20, 0x31, 0x31, 0x2e, 0x34, - 0x2e, 0x30, 0x00, 0x00, 0x2e, 0x73, 0x68, 0x73, 0x74, 0x72, 0x74, 0x61, - 0x62, 0x00, 0x2e, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x70, 0x00, 0x2e, 0x67, - 0x6e, 0x75, 0x2e, 0x68, 0x61, 0x73, 0x68, 0x00, 0x2e, 0x64, 0x79, 0x6e, - 0x73, 0x79, 0x6d, 0x00, 0x2e, 0x64, 0x79, 0x6e, 0x73, 0x74, 0x72, 0x00, - 0x2e, 0x72, 0x65, 0x6c, 0x61, 0x2e, 0x64, 0x79, 0x6e, 0x00, 0x2e, 0x74, - 0x65, 0x78, 0x74, 0x00, 0x2e, 0x72, 0x6f, 0x64, 0x61, 0x74, 0x61, 0x00, - 0x2e, 0x64, 0x79, 0x6e, 0x61, 0x6d, 0x69, 0x63, 0x00, 0x2e, 0x67, 0x6f, - 0x74, 0x00, 0x2e, 0x62, 0x73, 0x73, 0x00, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, - 0x65, 0x6e, 0x74, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xc8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xc8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x6f, 0xff, 0xfe, 0xf5, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x50, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x20, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0b, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x15, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x02, 0x58, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, + 0x00, 0x00, 0x00, 0x00, 0x6f, 0xff, 0xff, 0xfb, 0x00, 0x00, 0x00, 0x00, + 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x6f, 0xff, 0xff, 0xf9, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x13, 0x6f, 0xff, 0xff, 0xf6, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1e, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0x00, + 0x47, 0x43, 0x43, 0x3a, 0x20, 0x28, 0x53, 0x55, 0x53, 0x45, 0x20, 0x4c, + 0x69, 0x6e, 0x75, 0x78, 0x29, 0x20, 0x31, 0x32, 0x2e, 0x33, 0x2e, 0x30, + 0x00, 0x00, 0x2e, 0x73, 0x68, 0x73, 0x74, 0x72, 0x74, 0x61, 0x62, 0x00, + 0x2e, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x70, 0x00, 0x2e, 0x67, 0x6e, 0x75, + 0x2e, 0x68, 0x61, 0x73, 0x68, 0x00, 0x2e, 0x64, 0x79, 0x6e, 0x73, 0x79, + 0x6d, 0x00, 0x2e, 0x64, 0x79, 0x6e, 0x73, 0x74, 0x72, 0x00, 0x2e, 0x72, + 0x65, 0x6c, 0x61, 0x2e, 0x64, 0x79, 0x6e, 0x00, 0x2e, 0x74, 0x65, 0x78, + 0x74, 0x00, 0x2e, 0x72, 0x6f, 0x64, 0x61, 0x74, 0x61, 0x00, 0x2e, 0x65, + 0x68, 0x5f, 0x66, 0x72, 0x61, 0x6d, 0x65, 0x00, 0x2e, 0x64, 0x79, 0x6e, + 0x61, 0x6d, 0x69, 0x63, 0x00, 0x2e, 0x67, 0x6f, 0x74, 0x00, 0x2e, 0x64, + 0x61, 0x74, 0x61, 0x00, 0x2e, 0x62, 0x73, 0x73, 0x00, 0x2e, 0x63, 0x6f, + 0x6d, 0x6d, 0x65, 0x6e, 0x74, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x01, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xd8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x03, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1d, - 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xf8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x01, 0xf8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, - 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, - 0x00, 0x00, 0x00, 0x25, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x28, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x28, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x2d, 0x00, 0x00, 0x00, 0x04, + 0x00, 0x00, 0x01, 0xc8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xc8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, + 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xd8, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x28, + 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, + 0x00, 0x00, 0x00, 0x13, 0x6f, 0xff, 0xff, 0xf6, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1d, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x02, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x30, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x03, + 0x00, 0x00, 0x02, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x20, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x05, + 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x25, + 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x50, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x02, 0x50, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x2d, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x58, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x58, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x02, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x70, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x37, - 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x48, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x02, 0x48, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x40, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3d, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x78, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x04, 0x78, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x88, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x88, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x01, 0x24, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x88, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x88, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, 0x06, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x17, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xb8, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x20, 0x00, 0x00, 0x00, 0x04, + 0x00, 0x00, 0x1e, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0e, 0xd8, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x10, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x4e, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x58, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0xd8, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x08, 0xd8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f, 0xe8, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x0f, 0xe8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x53, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, + 0x00, 0x00, 0x00, 0x5d, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0xf0, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x01, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x58, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0xf0, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, - 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x63, 0x00, 0x00, 0x00, 0x08, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x08, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x68, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x10, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x09, 0x1b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x21, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x71, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00 }; diff --git a/tests/qtest/migration/s390x/a-b-bios.c b/tests/qtest/migration/s390x/a-b-bios.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/s390x/a-b-bios.c +++ b/tests/qtest/migration/s390x/a-b-bios.c @@ -XXX,XX +XXX,XX @@ * option) any later version. */ -#define LOADPARM_LEN 8 /* Needed for sclp.h */ +/* for sclp.h */ +#define LOADPARM_LEN 8 +typedef unsigned int uint32_t; +typedef unsigned short uint16_t; +typedef unsigned char uint8_t; -#include <libc.h> -#include <s390-ccw.h> #include <sclp.h> -char stack[0x8000] __attribute__((aligned(4096))); - #define START_ADDRESS (1024 * 1024) #define END_ADDRESS (100 * 1024 * 1024) +/* at pc-bios/s390-ccw/start.S */ + +void __attribute__((weak)) consume_sclp_int(void) +{ +} + +static char _sccb[4096] __attribute__((__aligned__(4096))); +char stack[0x8000] __attribute__((aligned(4096))); + +static void sclp_service_call(unsigned int command) +{ + int cc; + + asm volatile( + " .insn rre,0xb2200000,%1,%2\n" /* servc %1,%2 */ + " ipm %0\n" + " srl %0,28" + : "=&d" (cc) : "d" (command), "a" (__pa(_sccb)) + : "cc", "memory"); + consume_sclp_int(); +} + +static void sclp_setup(void) +{ + WriteEventMask *sccb = (void *)_sccb; + + sccb->h.length = sizeof(WriteEventMask); + + sccb->mask_length = sizeof(unsigned int); + sccb->cp_receive_mask = 0; + sccb->cp_send_mask = SCLP_EVENT_MASK_MSG_ASCII; + + sclp_service_call(SCLP_CMD_WRITE_EVENT_MASK); +} + +static void putc(const char c) +{ + WriteEventData *sccb = (void *)_sccb; + int len = 1; + + sccb->data[0] = c; + + sccb->h.length = sizeof(WriteEventData) + len; + sccb->h.function_code = SCLP_FC_NORMAL_WRITE; + + sccb->ebh.length = sizeof(EventBufferHeader) + len; + sccb->ebh.type = SCLP_EVENT_ASCII_CONSOLE_DATA; + sccb->ebh.flags = 0; + + sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA); +} + void main(void) { unsigned long addr; sclp_setup(); - sclp_print("A"); + putc('A'); /* * Make sure all of the pages have consistent contents before incrementing @@ -XXX,XX +XXX,XX @@ void main(void) for (addr = START_ADDRESS; addr < END_ADDRESS; addr += 4096) { *(volatile char *)addr += 1; /* Change pages */ } - sclp_print("B"); + putc('B'); } } diff --git a/tests/qtest/migration/Makefile b/tests/qtest/migration/Makefile index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/Makefile +++ b/tests/qtest/migration/Makefile @@ -XXX,XX +XXX,XX @@ TARGET_LIST = i386 aarch64 s390x ppc64 -SRC_PATH = ../.. +SRC_PATH = ../../../.. .PHONY: help $(TARGET_LIST) help: @@ -XXX,XX +XXX,XX @@ help: @echo " Possible targets are: $(TARGET_LIST)" override define __note -/* This file is automatically generated from the assembly file in - * tests/migration/$@. Edit that file and then run "make all" - * inside tests/migration to update, and then remember to send both +/* + * This file is automatically generated from the assembly file in + * tests/qtest/migration/$@. Edit that file and then run "make" + * inside tests/qtest/migration to update, and then remember to send both * the header and the assembler differences in your patch submission. */ endef export __note $(TARGET_LIST): - $(MAKE) CROSS_PREFIX=$(CROSS_PREFIX) -C $@ + $(MAKE) SRC_PATH=$(SRC_PATH) CROSS_PREFIX=$(CROSS_PREFIX) -C $@ clean: for target in $(TARGET_LIST); do \ diff --git a/tests/qtest/migration/aarch64/Makefile b/tests/qtest/migration/aarch64/Makefile index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/aarch64/Makefile +++ b/tests/qtest/migration/aarch64/Makefile @@ -XXX,XX +XXX,XX @@ all: a-b-kernel.h a-b-kernel.h: aarch64.kernel echo "$$__note" > $@ - xxd -i $< | sed -e 's/.*int.*//' >> $@ + xxd -i $< | sed -e '/.*int.*/d' >> $@ aarch64.kernel: aarch64.elf $(CROSS_PREFIX)objcopy -O binary $< $@ @@ -XXX,XX +XXX,XX @@ aarch64.elf: a-b-kernel.S $(CROSS_PREFIX)gcc -o $@ -nostdlib -Wl,--build-id=none $< clean: - $(RM) *.kernel *.elf + $(RM) *.kernel *.elf *.h diff --git a/tests/qtest/migration/aarch64/a-b-kernel.S b/tests/qtest/migration/aarch64/a-b-kernel.S index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/aarch64/a-b-kernel.S +++ b/tests/qtest/migration/aarch64/a-b-kernel.S @@ -XXX,XX +XXX,XX @@ # pc-relative address. Also the branch instructions should use relative # addresses only. -#include "../migration-test.h" +#include "../bootfile.h" .section .text diff --git a/tests/qtest/migration/i386/Makefile b/tests/qtest/migration/i386/Makefile index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/i386/Makefile +++ b/tests/qtest/migration/i386/Makefile @@ -XXX,XX +XXX,XX @@ all: a-b-bootblock.h a-b-bootblock.h: x86.bootsect x86.o echo "$$__note" > header.tmp - xxd -i $< | sed -e 's/.*int.*//' >> header.tmp + xxd -i $< | sed -e '/.*int.*/d' >> header.tmp nm x86.o | awk '{print "#define SYM_"$$3" 0x"$$1}' >> header.tmp mv header.tmp $@ @@ -XXX,XX +XXX,XX @@ x86.o: a-b-bootblock.S $(CROSS_PREFIX)gcc -I.. -m32 -march=i486 -c $< -o $@ clean: - @rm -rf *.boot *.o *.bootsect + @rm -rf *.boot *.o *.bootsect *.h diff --git a/tests/qtest/migration/i386/a-b-bootblock.S b/tests/qtest/migration/i386/a-b-bootblock.S index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/i386/a-b-bootblock.S +++ b/tests/qtest/migration/i386/a-b-bootblock.S @@ -XXX,XX +XXX,XX @@ # # Author: dgilbert@redhat.com -#include "migration-test.h" +#include "../bootfile.h" #define ACPI_ENABLE 0xf1 #define ACPI_PORT_SMI_CMD 0xb2 diff --git a/tests/qtest/migration/ppc64/Makefile b/tests/qtest/migration/ppc64/Makefile index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/ppc64/Makefile +++ b/tests/qtest/migration/ppc64/Makefile @@ -XXX,XX +XXX,XX @@ all: a-b-kernel.h a-b-kernel.h: ppc64.kernel echo "$$__note" > $@ - xxd -i $< | sed -e 's/.*int.*//' >> $@ + xxd -i $< | sed -e '/.*int.*/d' >> $@ ppc64.kernel: ppc64.elf $(CROSS_PREFIX)objcopy -O binary -S $< $@ ppc64.elf: a-b-kernel.S - $(CROSS_PREFIX)gcc -static -o $@ -nostdlib -Wl,--build-id=none $< + $(CROSS_PREFIX)gcc -mbig-endian -static -o $@ -nostdlib -Wl,--build-id=none $< clean: - $(RM) *.kernel *.elf + $(RM) *.kernel *.elf *.h diff --git a/tests/qtest/migration/ppc64/a-b-kernel.S b/tests/qtest/migration/ppc64/a-b-kernel.S index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/ppc64/a-b-kernel.S +++ b/tests/qtest/migration/ppc64/a-b-kernel.S @@ -XXX,XX +XXX,XX @@ # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. -#include "../migration-test.h" +#include "../bootfile.h" .section .text diff --git a/tests/qtest/migration/s390x/Makefile b/tests/qtest/migration/s390x/Makefile index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/migration/s390x/Makefile +++ b/tests/qtest/migration/s390x/Makefile @@ -XXX,XX +XXX,XX @@ .PHONY: all clean all: a-b-bios.h -fwdir=../../../pc-bios/s390-ccw +fwdir=$(SRC_PATH)/pc-bios/s390-ccw CFLAGS+=-ffreestanding -fno-delete-null-pointer-checks -fPIE -Os \ -msoft-float -march=z900 -fno-asynchronous-unwind-tables \ @@ -XXX,XX +XXX,XX @@ CFLAGS+=-ffreestanding -fno-delete-null-pointer-checks -fPIE -Os \ a-b-bios.h: s390x.elf echo "$$__note" > header.tmp - xxd -i $< | sed -e 's/.*int.*//' >> header.tmp + xxd -i $< | sed -e '/.*int.*/d' >> header.tmp mv header.tmp $@ # We use common-page-size=16 to avoid big padding in the ELF file s390x.elf: a-b-bios.c $(CROSS_PREFIX)gcc $(CFLAGS) -I$(fwdir) $(fwdir)/start.S \ - $(fwdir)/sclp.c -Wl,-zcommon-page-size=16 -o $@ $< + -Wl,-zcommon-page-size=16 -o $@ $< $(CROSS_PREFIX)strip $@ clean: - @rm -rf *.elf *.o + @rm -rf *.elf *.o *.h -- 2.53.0