Hi, Steve
I have saw the discussion about the fork+exec mode below:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg815956.html

And I am still very curious and I want to discuss about the possibility to support both fork+exec and exec in cpr framework.

1.Why
fork+exec could have some advantages and also drawbacks versus execvp() directly.
Advantages
i) fork+exec give the chance to fallback to original process even after we do exec which is important for workload seamless if any error happens.
ii) smaller downtime since we could remove the vm_start() downtime out of the frozen window.
Drawbacks
i）need more codes to handle including fork，address/ports conflict between parent and child.
ii）more complex life cycle management for the two processes.

2.How
The cpr framework is flexible and scalable, and maybe we can make use of most codes to support both execvp and fork+exec mode.
However, we may need to do more work compared to execvp method.
i) do fork mode in a thread like migration thread
ii) make parent and child talk to each other through socket or anonymous pipe
iii）make use of sharing mechanism of fds in cpr framework including memfd, vfio and devices fds
iv）deal with the conflict about the socket address and port like vnc （do by reuse port and pass the different args by cprexec）
v) do life cycle managements for two qemu processes and need parent exit and reconnection for the child at last by the management service

Please tell me if I am missing something important:)

-- 
Regards.
Chuan

On 7/30/2021 9:10 AM, Zheng Chuan wrote:
> Hi, Steve
> I have saw the discussion about the fork+exec mode below:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg815956.html
> 
> And I am still very curious and I want to discuss about the possibility to support both fork+exec and exec in cpr framework.
> 
> 1.Why
> fork+exec could have some advantages and also drawbacks versus execvp() directly.
> Advantages
> i) fork+exec give the chance to fallback to original process even after we do exec which is important for workload seamless if any error happens.
> ii) smaller downtime since we could remove the vm_start() downtime out of the frozen window.
> Drawbacks
> i）need more codes to handle including fork，address/ports conflict between parent and child.
> ii）more complex life cycle management for the two processes.
> 
> 2.How
> The cpr framework is flexible and scalable, and maybe we can make use of most codes to support both execvp and fork+exec mode.
> However, we may need to do more work compared to execvp method.
> i) do fork mode in a thread like migration thread
> ii) make parent and child talk to each other through socket or anonymous pipe
> iii）make use of sharing mechanism of fds in cpr framework including memfd, vfio and devices fds
> iv）deal with the conflict about the socket address and port like vnc （do by reuse port and pass the different args by cprexec）
> v) do life cycle managements for two qemu processes and need parent exit and reconnection for the child at last by the management service
> 
> Please tell me if I am missing something important:)

Hi Zheng, that list sounds about right.  However, additional kernel changes would be needed to 
change ownership of the vfio device descriptors. The vfio module records the mm of the creating
process, and does not allow other processes to unmap ranges.  Also, the mm->locked_vm would 
need to be transferred to the new process.

Fork+exec could be a new mode to the cprsave command.

- Steve

Hi, Steve.

On 2021/8/5 4:50, Steven Sistare wrote:
> On 7/30/2021 9:10 AM, Zheng Chuan wrote:
>> Hi, Steve
>> I have saw the discussion about the fork+exec mode below:
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg815956.html
>>
>> And I am still very curious and I want to discuss about the possibility to support both fork+exec and exec in cpr framework.
>>
>> 1.Why
>> fork+exec could have some advantages and also drawbacks versus execvp() directly.
>> Advantages
>> i) fork+exec give the chance to fallback to original process even after we do exec which is important for workload seamless if any error happens.
>> ii) smaller downtime since we could remove the vm_start() downtime out of the frozen window.
>> Drawbacks
>> i）need more codes to handle including fork，address/ports conflict between parent and child.
>> ii）more complex life cycle management for the two processes.
>>
>> 2.How
>> The cpr framework is flexible and scalable, and maybe we can make use of most codes to support both execvp and fork+exec mode.
>> However, we may need to do more work compared to execvp method.
>> i) do fork mode in a thread like migration thread
>> ii) make parent and child talk to each other through socket or anonymous pipe
>> iii）make use of sharing mechanism of fds in cpr framework including memfd, vfio and devices fds
>> iv）deal with the conflict about the socket address and port like vnc （do by reuse port and pass the different args by cprexec）
>> v) do life cycle managements for two qemu processes and need parent exit and reconnection for the child at last by the management service
>>
>> Please tell me if I am missing something important:)
> 
> Hi Zheng, that list sounds about right.  However, additional kernel changes would be needed to 
> change ownership of the vfio device descriptors. The vfio module records the mm of the creating
> process, and does not allow other processes to unmap ranges.  Also, the mm->locked_vm would 
> need to be transferred to the new process.
> 
Oh，That's right. Maybe we could update them along with VFIO_DMA_MAP_FLAG_VADDR.

> Fork+exec could be a new mode to the cprsave command.
>
Yeah, I would be very happy to see it in the future:)
> - Steve
> .
> 

-- 
Regards.
Chuan

kindly ping？

On 2021/7/30 21:10, Zheng Chuan wrote:
> Hi, Steve
> I have saw the discussion about the fork+exec mode below:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg815956.html
> 
> And I am still very curious and I want to discuss about the possibility to support both fork+exec and exec in cpr framework.
> 
> 1.Why
> fork+exec could have some advantages and also drawbacks versus execvp() directly.
> Advantages
> i) fork+exec give the chance to fallback to original process even after we do exec which is important for workload seamless if any error happens.
> ii) smaller downtime since we could remove the vm_start() downtime out of the frozen window.
> Drawbacks
> i）need more codes to handle including fork，address/ports conflict between parent and child.
> ii）more complex life cycle management for the two processes.
> 
> 2.How
> The cpr framework is flexible and scalable, and maybe we can make use of most codes to support both execvp and fork+exec mode.
> However, we may need to do more work compared to execvp method.
> i) do fork mode in a thread like migration thread
> ii) make parent and child talk to each other through socket or anonymous pipe
> iii）make use of sharing mechanism of fds in cpr framework including memfd, vfio and devices fds
> iv）deal with the conflict about the socket address and port like vnc （do by reuse port and pass the different args by cprexec）
> v) do life cycle managements for two qemu processes and need parent exit and reconnection for the child at last by the management service
> 
> Please tell me if I am missing something important:)
> 

-- 
Regards.
Chuan