> This version is a bit better than the previous one. But we're at version > 13 and I am still unable to even start a guest. Please, make sure that this > basic functionality works in v14, because this is plain waste of precious > review bandwidth. > > Anyway, as usual, I've uploaded my suggested fixes here: > > https://gitlab.com/MichalPrivoznik/libvirt/-/commits/sgx/ Sorry to hear it didn't work in your environment. We definitely tested it several times and it works well for both QEMU 6.2.0 and 7.0.0. Let me try to reproduce it with the domain xml you shared before. By my best guess, if you see "qemu-system-x86_64:***: invalid object type: memory-backend-epc" error, it means QEMU didn't get enough permission to launch SGX VM. Pls add /dev/sgx_vepc, /dev/sgx_enclave and /dev/sgx_provision to the "cgroup_device_acl" list in /etc/libvirt/qemu.conf. QEMU requires those access to assign EPC, but it was denied by libvirt’s cgroup controllers by default. cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", ... "/dev/sgx_vepc", "/dev/sgx_enclave”, "/dev/sgx_provision” ] Also in /etc/libvirt/qemu.conf, set the runtime user to uid 0, since QEMU needs to read and write to those sgx devices, like /dev/sgx_vepc. Unfortunately, it is owned by root with file mode 600, so QEMU has to launch as root. user = "+0" It would be really helpful if you can use these steps to see whether it resolve the issue. I will add a doc somewhere to include all steps are required for use to use sgx in libvirt. Thanks, Lin.
On Wed, Jul 20, 2022 at 11:12:56PM +0000, Yang, Lin A wrote: > > This version is a bit better than the previous one. But we're at version > > 13 and I am still unable to even start a guest. Please, make sure that this > > basic functionality works in v14, because this is plain waste of precious > > review bandwidth. > > > > Anyway, as usual, I've uploaded my suggested fixes here: > > > > https://gitlab.com/MichalPrivoznik/libvirt/-/commits/sgx/ > > Sorry to hear it didn't work in your environment. We definitely tested it > several times and it works well for both QEMU 6.2.0 and 7.0.0. > > Let me try to reproduce it with the domain xml you shared before. > > By my best guess, if you see "qemu-system-x86_64:***: > invalid object type: memory-backend-epc" error, it means QEMU didn't > get enough permission to launch SGX VM. > > Pls add /dev/sgx_vepc, /dev/sgx_enclave and /dev/sgx_provision to the > "cgroup_device_acl" list in /etc/libvirt/qemu.conf. QEMU requires those access > to assign EPC, but it was denied by libvirt’s cgroup controllers by default. > > cgroup_device_acl = [ > "/dev/null", "/dev/full", "/dev/zero", > ... > "/dev/sgx_vepc", > "/dev/sgx_enclave”, > "/dev/sgx_provision” > ] > > Also in /etc/libvirt/qemu.conf, set the runtime user to uid 0, since QEMU needs to > read and write to those sgx devices, like /dev/sgx_vepc. Unfortunately, it is owned > by root with file mode 600, so QEMU has to launch as root. > > user = "+0" > > It would be really helpful if you can use these steps to see whether it resolve > the issue. I will add a doc somewhere to include all steps are required for use to > use sgx in libvirt. The need to customize qemu.conf to change cgroups ACLs and set uid==0 makes this patch series unusal in the real world deployments. It cannot be merged with such problems existing. Are the /dev/sgx* fundamentally required to be restricted to root access only, or is it safe to make them accessible to non-root ? ie If a malicious user has access to those files, what is the impact they have ? Bear in mind that QEMU itself can be malicious if the guest compromises it. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 7/21/22 10:06, Daniel P. Berrangé wrote: > On Wed, Jul 20, 2022 at 11:12:56PM +0000, Yang, Lin A wrote: >>> This version is a bit better than the previous one. But we're at version >>> 13 and I am still unable to even start a guest. Please, make sure that this >>> basic functionality works in v14, because this is plain waste of precious >>> review bandwidth. >>> >>> Anyway, as usual, I've uploaded my suggested fixes here: >>> >>> https://gitlab.com/MichalPrivoznik/libvirt/-/commits/sgx/ >> >> Sorry to hear it didn't work in your environment. We definitely tested it >> several times and it works well for both QEMU 6.2.0 and 7.0.0. Alright, I finally made it work. The problem was with memfd backend. I'll post patch for that soon. >> >> Let me try to reproduce it with the domain xml you shared before. >> >> By my best guess, if you see "qemu-system-x86_64:***: >> invalid object type: memory-backend-epc" error, it means QEMU didn't >> get enough permission to launch SGX VM. >> >> Pls add /dev/sgx_vepc, /dev/sgx_enclave and /dev/sgx_provision to the >> "cgroup_device_acl" list in /etc/libvirt/qemu.conf. QEMU requires those access >> to assign EPC, but it was denied by libvirt’s cgroup controllers by default. >> >> cgroup_device_acl = [ >> "/dev/null", "/dev/full", "/dev/zero", >> ... >> "/dev/sgx_vepc", >> "/dev/sgx_enclave”, >> "/dev/sgx_provision” >> ] >> >> Also in /etc/libvirt/qemu.conf, set the runtime user to uid 0, since QEMU needs to >> read and write to those sgx devices, like /dev/sgx_vepc. Unfortunately, it is owned >> by root with file mode 600, so QEMU has to launch as root. >> >> user = "+0" >> >> It would be really helpful if you can use these steps to see whether it resolve >> the issue. I will add a doc somewhere to include all steps are required for use to >> use sgx in libvirt. > > The need to customize qemu.conf to change cgroups ACLs and set uid==0 makes > this patch series unusal in the real world deployments. It cannot be merged > with such problems existing. > Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for other devices, including NVDIMM and virtio-pmem types of <memory/>), it's more tricky with relabelling. By default, when available, libvirt creates a separate mount namespace for each QEMU process and creates a very small /dev there, with only those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up patches on top of this series) the /dev/sgx* nodes are created there AND I have another patch that sets DAC/SELinux label on them so that uid=0 is no longer needed. What I worry about though, is the case when this namespace feature is disabled. Then libvirt should not touch /dev/sgx* because that might compromise security in the system. > Are the /dev/sgx* fundamentally required to be restricted to root access > only, or is it safe to make them accessible to non-root ? ie If a malicious > user has access to those files, what is the impact they have ? Bear in mind > that QEMU itself can be malicious if the guest compromises it. If we get an agreement here, I can cleanup this v13 and post v14 that include all patches mentioned. Michal
On Thu, Jul 21, 2022 at 03:12:05PM +0200, Michal Prívozník wrote: > On 7/21/22 10:06, Daniel P. Berrangé wrote: > > On Wed, Jul 20, 2022 at 11:12:56PM +0000, Yang, Lin A wrote: > >>> This version is a bit better than the previous one. But we're at version > >>> 13 and I am still unable to even start a guest. Please, make sure that this > >>> basic functionality works in v14, because this is plain waste of precious > >>> review bandwidth. > >>> > >>> Anyway, as usual, I've uploaded my suggested fixes here: > >>> > >>> https://gitlab.com/MichalPrivoznik/libvirt/-/commits/sgx/ > >> > >> Sorry to hear it didn't work in your environment. We definitely tested it > >> several times and it works well for both QEMU 6.2.0 and 7.0.0. > > Alright, I finally made it work. The problem was with memfd backend. > I'll post patch for that soon. > > >> > >> Let me try to reproduce it with the domain xml you shared before. > >> > >> By my best guess, if you see "qemu-system-x86_64:***: > >> invalid object type: memory-backend-epc" error, it means QEMU didn't > >> get enough permission to launch SGX VM. > >> > >> Pls add /dev/sgx_vepc, /dev/sgx_enclave and /dev/sgx_provision to the > >> "cgroup_device_acl" list in /etc/libvirt/qemu.conf. QEMU requires those access > >> to assign EPC, but it was denied by libvirt’s cgroup controllers by default. > >> > >> cgroup_device_acl = [ > >> "/dev/null", "/dev/full", "/dev/zero", > >> ... > >> "/dev/sgx_vepc", > >> "/dev/sgx_enclave”, > >> "/dev/sgx_provision” > >> ] > >> > >> Also in /etc/libvirt/qemu.conf, set the runtime user to uid 0, since QEMU needs to > >> read and write to those sgx devices, like /dev/sgx_vepc. Unfortunately, it is owned > >> by root with file mode 600, so QEMU has to launch as root. > >> > >> user = "+0" > >> > >> It would be really helpful if you can use these steps to see whether it resolve > >> the issue. I will add a doc somewhere to include all steps are required for use to > >> use sgx in libvirt. > > > > The need to customize qemu.conf to change cgroups ACLs and set uid==0 makes > > this patch series unusal in the real world deployments. It cannot be merged > > with such problems existing. > > > > Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for > other devices, including NVDIMM and virtio-pmem types of <memory/>), > it's more tricky with relabelling. > > By default, when available, libvirt creates a separate mount namespace > for each QEMU process and creates a very small /dev there, with only > those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up > patches on top of this series) the /dev/sgx* nodes are created there AND > I have another patch that sets DAC/SELinux label on them so that uid=0 > is no longer needed. What I worry about though, is the case when this > namespace feature is disabled. Then libvirt should not touch /dev/sgx* > because that might compromise security in the system. That might in turn require the ability to pass in pre-opened FDs for the devices to QEMU. > > > Are the /dev/sgx* fundamentally required to be restricted to root access > > only, or is it safe to make them accessible to non-root ? ie If a malicious > > user has access to those files, what is the impact they have ? Bear in mind > > that QEMU itself can be malicious if the guest compromises it. > > If we get an agreement here, I can cleanup this v13 and post v14 that > include all patches mentioned. > > Michal > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 7/21/22 15:24, Daniel P. Berrangé wrote: > On Thu, Jul 21, 2022 at 03:12:05PM +0200, Michal Prívozník wrote: >> On 7/21/22 10:06, Daniel P. Berrangé wrote: >>> On Wed, Jul 20, 2022 at 11:12:56PM +0000, Yang, Lin A wrote: >>>>> This version is a bit better than the previous one. But we're at version >>>>> 13 and I am still unable to even start a guest. Please, make sure that this >>>>> basic functionality works in v14, because this is plain waste of precious >>>>> review bandwidth. >>>>> >>>>> Anyway, as usual, I've uploaded my suggested fixes here: >>>>> >>>>> https://gitlab.com/MichalPrivoznik/libvirt/-/commits/sgx/ >>>> >>>> Sorry to hear it didn't work in your environment. We definitely tested it >>>> several times and it works well for both QEMU 6.2.0 and 7.0.0. >> >> Alright, I finally made it work. The problem was with memfd backend. >> I'll post patch for that soon. >> >>>> >>>> Let me try to reproduce it with the domain xml you shared before. >>>> >>>> By my best guess, if you see "qemu-system-x86_64:***: >>>> invalid object type: memory-backend-epc" error, it means QEMU didn't >>>> get enough permission to launch SGX VM. >>>> >>>> Pls add /dev/sgx_vepc, /dev/sgx_enclave and /dev/sgx_provision to the >>>> "cgroup_device_acl" list in /etc/libvirt/qemu.conf. QEMU requires those access >>>> to assign EPC, but it was denied by libvirt’s cgroup controllers by default. >>>> >>>> cgroup_device_acl = [ >>>> "/dev/null", "/dev/full", "/dev/zero", >>>> ... >>>> "/dev/sgx_vepc", >>>> "/dev/sgx_enclave”, >>>> "/dev/sgx_provision” BTW: I can see in QEMU sources /dev/sgx_vepc and /dev/sgx_provision being opened, but not sgx_enclave. And I see the former two on my system but not the last one. Can you Yang, share more info on this please? >>>> ] >>>> >>>> Also in /etc/libvirt/qemu.conf, set the runtime user to uid 0, since QEMU needs to >>>> read and write to those sgx devices, like /dev/sgx_vepc. Unfortunately, it is owned >>>> by root with file mode 600, so QEMU has to launch as root. >>>> >>>> user = "+0" >>>> >>>> It would be really helpful if you can use these steps to see whether it resolve >>>> the issue. I will add a doc somewhere to include all steps are required for use to >>>> use sgx in libvirt. >>> >>> The need to customize qemu.conf to change cgroups ACLs and set uid==0 makes >>> this patch series unusal in the real world deployments. It cannot be merged >>> with such problems existing. >>> >> >> Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for >> other devices, including NVDIMM and virtio-pmem types of <memory/>), >> it's more tricky with relabelling. >> >> By default, when available, libvirt creates a separate mount namespace >> for each QEMU process and creates a very small /dev there, with only >> those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up >> patches on top of this series) the /dev/sgx* nodes are created there AND >> I have another patch that sets DAC/SELinux label on them so that uid=0 >> is no longer needed. What I worry about though, is the case when this >> namespace feature is disabled. Then libvirt should not touch /dev/sgx* >> because that might compromise security in the system. > > That might in turn require the ability to pass in pre-opened FDs for > the devices to QEMU. Yeah, that might be the perfect solution, but IIUC there's currently no way to achieve that, or is it? Is it something we should do in QEMU first? Michal
> BTW: I can see in QEMU sources /dev/sgx_vepc and /dev/sgx_provision being > opened, but not sgx_enclave. And I see the former two on my system but not > the last one. Can you Yang, share more info on this please? True, QEMU only need read and write access to /dev/sgx_vepc and /dev/sgx_provision. /dev/sgx_vepc allows userspace to allocate "raw" EPC without an associated enclave. The only known use case for raw EPC allocation is to expose EPC to a KVM guest, hence call it 'vepc'. /dev/sgx_enclave allows creating host enclave. It is not suitable for allocating EPC for KVM guest. Having separate device nodes, /dev/sgx_vepc and /dev/sgx_enclave, allows separate permission control for creating host SGX enclaves and KVM SGX guests. /dev/sgx_provision allows creating provisioning enclaves, which typically have more strict permissions than the plain enclave device /dev/sgx_enclave. Usually /dev/sgx_enclave and /dev/sgx_provision should exist together on your system. Set "CONFIG_X86_SGX=y" in Kconfig and enable SGX in bios should enable SGX driver and create /dev/sgx_enclave and /dev/sgx_provision device nodes. "CONFIG_X86_SGX_KVM=y" will create /dev/sgx_vepc device node. Regrading to permission control, one suggested way is making /dev/sgx_enclave is accessible to all userspace applications to create its enclave. Having strict permissions on /dev/sgx_vepc and /dev/sgx_provision only for user in specific group "XYZ". # ls -l /dev/sgx* crw-rw-rw- 1 root root 10, 125 Nov 16 2021 /dev/sgx_enclave crw-rw---- 1 root XYZ 10, 126 Nov 16 2021 /dev/sgx_provision crw-rw---- 1 root XYZ 10, 124 Nov 16 2021 /dev/sgx_vepc Instead of running QEMU by root, one straightforward way is admin create a dedicated "qemu" user and add it to "XYZ" group. In /etc/libvirt/qemu.conf, user = "qemu" Any concerns about this solution? Thanks, Lin.
On 7/22/22 08:57, Yang, Lin A wrote: >> BTW: I can see in QEMU sources /dev/sgx_vepc and /dev/sgx_provision being >> opened, but not sgx_enclave. And I see the former two on my system but not >> the last one. Can you Yang, share more info on this please? > > True, QEMU only need read and write access to /dev/sgx_vepc and /dev/sgx_provision. > > /dev/sgx_vepc allows userspace to allocate "raw" EPC without an associated enclave. > The only known use case for raw EPC allocation is to expose EPC to a KVM guest, > hence call it 'vepc'. > > /dev/sgx_enclave allows creating host enclave. It is not suitable for allocating EPC for > KVM guest. Having separate device nodes, /dev/sgx_vepc and /dev/sgx_enclave, > allows separate permission control for creating host SGX enclaves and KVM SGX guests. I see where the problem is now. My processor lacks X86_FEATURE_SGX_LC therefore, when SGX driver wants to register /dev/sgx_enclave inside of sgx_drv_init() (arch/x86/kernel/cpu/sgx/driver.c) the function exits early. > > /dev/sgx_provision allows creating provisioning enclaves, which typically have more > strict permissions than the plain enclave device /dev/sgx_enclave. > > Usually /dev/sgx_enclave and /dev/sgx_provision should exist together on your system. > Set "CONFIG_X86_SGX=y" in Kconfig and enable SGX in bios should enable SGX driver > and create /dev/sgx_enclave and /dev/sgx_provision device nodes. > "CONFIG_X86_SGX_KVM=y" will create /dev/sgx_vepc device node. > > Regrading to permission control, one suggested way is making /dev/sgx_enclave is > accessible to all userspace applications to create its enclave. Having strict permissions > on /dev/sgx_vepc and /dev/sgx_provision only for user in specific group "XYZ". > > # ls -l /dev/sgx* > crw-rw-rw- 1 root root 10, 125 Nov 16 2021 /dev/sgx_enclave > crw-rw---- 1 root XYZ 10, 126 Nov 16 2021 /dev/sgx_provision > crw-rw---- 1 root XYZ 10, 124 Nov 16 2021 /dev/sgx_vepc > > Instead of running QEMU by root, one straightforward way is admin create a > dedicated "qemu" user and add it to "XYZ" group. In /etc/libvirt/qemu.conf, > > user = "qemu" > > Any concerns about this solution? Well, as discussed with Daniel earlier, libvirt creates a separate mount namespace for each QEMU and inside it creates a very thin /dev with only a handful of nodes (per guest config). And what my patch does (and what we already do for /dev/sev) is mknod() /dev/sgx_provision and /dev/sgx_vepc inside that thin /dev and chown() it to the user under which QEMU is about to run. This namespace feature can be turned off though, in which case libvirt won't chown() those files (well, my patch is written that way). I think this is acceptable trade off between security and usability. Namespaces are enabled by default (if kernel supports them). Alright, so here's what we'll do. I'll polish my patches, fix up yours and send for review. Does this work for you? Michal
> Well, as discussed with Daniel earlier, libvirt creates a separate mount > namespace for each QEMU and inside it creates a very thin /dev with only a > handful of nodes (per guest config). And what my patch does (and what we > already do for /dev/sev) is mknod() /dev/sgx_provision and /dev/sgx_vepc inside > that thin /dev and chown() it to the user under which QEMU is about to run. > > This namespace feature can be turned off though, in which case libvirt won't > chown() those files (well, my patch is written that way). I think this is acceptable > trade off between security and usability. Namespaces are enabled by default (if > kernel supports them). > > Alright, so here's what we'll do. I'll polish my patches, fix up yours and send for > review. Does this work for you? Definitely Yes! This is awesome! Really appreciated your help. Good to know libvirt creates separate mount namespace and thin /dev for each QEMU. Lin.
On Thu, Jul 21, 2022 at 04:10:11PM +0200, Michal Prívozník wrote: > On 7/21/22 15:24, Daniel P. Berrangé wrote: > > On Thu, Jul 21, 2022 at 03:12:05PM +0200, Michal Prívozník wrote: > >> On 7/21/22 10:06, Daniel P. Berrangé wrote: > >> Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for > >> other devices, including NVDIMM and virtio-pmem types of <memory/>), > >> it's more tricky with relabelling. > >> > >> By default, when available, libvirt creates a separate mount namespace > >> for each QEMU process and creates a very small /dev there, with only > >> those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up > >> patches on top of this series) the /dev/sgx* nodes are created there AND > >> I have another patch that sets DAC/SELinux label on them so that uid=0 > >> is no longer needed. What I worry about though, is the case when this > >> namespace feature is disabled. Then libvirt should not touch /dev/sgx* > >> because that might compromise security in the system. > > > > That might in turn require the ability to pass in pre-opened FDs for > > the devices to QEMU. > > Yeah, that might be the perfect solution, but IIUC there's currently no > way to achieve that, or is it? Is it something we should do in QEMU first? The code uses 'qemu_open', so it should be possible already with FD passing, by using a /dev/fdset/NNN path. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 7/21/22 16:29, Daniel P. Berrangé wrote: > On Thu, Jul 21, 2022 at 04:10:11PM +0200, Michal Prívozník wrote: >> On 7/21/22 15:24, Daniel P. Berrangé wrote: >>> On Thu, Jul 21, 2022 at 03:12:05PM +0200, Michal Prívozník wrote: >>>> On 7/21/22 10:06, Daniel P. Berrangé wrote: >>>> Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for >>>> other devices, including NVDIMM and virtio-pmem types of <memory/>), >>>> it's more tricky with relabelling. >>>> >>>> By default, when available, libvirt creates a separate mount namespace >>>> for each QEMU process and creates a very small /dev there, with only >>>> those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up >>>> patches on top of this series) the /dev/sgx* nodes are created there AND >>>> I have another patch that sets DAC/SELinux label on them so that uid=0 >>>> is no longer needed. What I worry about though, is the case when this >>>> namespace feature is disabled. Then libvirt should not touch /dev/sgx* >>>> because that might compromise security in the system. >>> >>> That might in turn require the ability to pass in pre-opened FDs for >>> the devices to QEMU. >> >> Yeah, that might be the perfect solution, but IIUC there's currently no >> way to achieve that, or is it? Is it something we should do in QEMU first? > > The code uses 'qemu_open', so it should be possible already with > FD passing, by using a /dev/fdset/NNN path. But there's no attribute that libvirt provides a path to. How does FD passing work in such case then? Here's the SGX part of cmd line: -object '{"qom-type":"memory-backend-epc","id":"memepc0","prealloc":true,"prealloc-threads":16,"size":16777216,"host-nodes":[0],"policy":"bind"}' \ Michal
On Thu, Jul 21, 2022 at 05:09:15PM +0200, Michal Prívozník wrote: > On 7/21/22 16:29, Daniel P. Berrangé wrote: > > On Thu, Jul 21, 2022 at 04:10:11PM +0200, Michal Prívozník wrote: > >> On 7/21/22 15:24, Daniel P. Berrangé wrote: > >>> On Thu, Jul 21, 2022 at 03:12:05PM +0200, Michal Prívozník wrote: > >>>> On 7/21/22 10:06, Daniel P. Berrangé wrote: > >>>> Agreed. While libvirt can allow /dev/sgx* in CGroups (we do that for > >>>> other devices, including NVDIMM and virtio-pmem types of <memory/>), > >>>> it's more tricky with relabelling. > >>>> > >>>> By default, when available, libvirt creates a separate mount namespace > >>>> for each QEMU process and creates a very small /dev there, with only > >>>> those nodes that QEMU needs. Now, if libvirt is fixed (I have follow up > >>>> patches on top of this series) the /dev/sgx* nodes are created there AND > >>>> I have another patch that sets DAC/SELinux label on them so that uid=0 > >>>> is no longer needed. What I worry about though, is the case when this > >>>> namespace feature is disabled. Then libvirt should not touch /dev/sgx* > >>>> because that might compromise security in the system. > >>> > >>> That might in turn require the ability to pass in pre-opened FDs for > >>> the devices to QEMU. > >> > >> Yeah, that might be the perfect solution, but IIUC there's currently no > >> way to achieve that, or is it? Is it something we should do in QEMU first? > > > > The code uses 'qemu_open', so it should be possible already with > > FD passing, by using a /dev/fdset/NNN path. > > But there's no attribute that libvirt provides a path to. How does FD > passing work in such case then? Oh right, I forgot, so this can't be used as is. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
© 2016 - 2024 Red Hat, Inc.