Resolve a storage driver crash as a result of a long running
storageVolCreateXML when the virStorageVolPoolRefreshThread is
run as a result of when a storageVolUpload completed and ran the
virStoragePoolObjClearVols without checking if the creation
code was currently processing a buildVol after incrementing
the driver->asyncjob count.

The refreshThread will now check the pool asyncjob count before
attempting to pursue the pool refresh. Adjust the documentation
to describe the condition.

Crash from valgrind is as follows (with a bit of editing):

==21309== Invalid read of size 8
==21309==    at 0x153E47AF: storageBackendUpdateVolTargetInfo
==21309==    by 0x153E4C30: virStorageBackendUpdateVolInfo
==21309==    by 0x153E52DE: virStorageBackendVolRefreshLocal
==21309==    by 0x153DE29E: storageVolCreateXML
==21309==    by 0x562035B: virStorageVolCreateXML
==21309==    by 0x147366: remoteDispatchStorageVolCreateXML
...
==21309==  Address 0x2590a720 is 64 bytes inside a block of size 336 free'd
==21309==    at 0x4C2F2BB: free
==21309==    by 0x54CB9FA: virFree
==21309==    by 0x55BC800: virStorageVolDefFree
==21309==    by 0x55BF1D8: virStoragePoolObjClearVols
==21309==    by 0x153D967E: virStorageVolPoolRefreshThread
...
==21309==  Block was alloc'd at
==21309==    at 0x4C300A5: calloc
==21309==    by 0x54CB483: virAlloc
==21309==    by 0x55BDC1F: virStorageVolDefParseXML
==21309==    by 0x55BDC1F: virStorageVolDefParseNode
==21309==    by 0x55BE5A4: virStorageVolDefParse
==21309==    by 0x153DDFF1: storageVolCreateXML
==21309==    by 0x562035B: virStorageVolCreateXML
==21309==    by 0x147366: remoteDispatchStorageVolCreateXML
...

Signed-off-by: John Ferlan <jferlan@redhat.com>
---

 v1: https://www.redhat.com/archives/libvir-list/2017-November/msg00198.html

 Changes since v1:
   - From review, remove the retry if Asyncjobs > 0 logic and replace with
     a VIR_DEBUG indicating not doing refresh due to Asyncjob running and
     just goto cleanup.

   - Adjust the virStorageVolUpload docs to note that an attempt will be
     made to refresh the pool.

   - Drop patch 2 which added the Asyncjobs > 0 check to the SCSI pool
     refresh thread. Since a SCSI pool doesn't support buildVol, the
     asyncjob count cannot be anything other than zero, so it's pointless.

 src/libvirt-storage.c        | 3 ++-
 src/storage/storage_driver.c | 7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/src/libvirt-storage.c b/src/libvirt-storage.c
index 9ab0fe28ea..e4646cb80f 100644
--- a/src/libvirt-storage.c
+++ b/src/libvirt-storage.c
@@ -1632,7 +1632,8 @@ virStorageVolDownload(virStorageVolPtr vol,
  * another active stream is writing to the storage volume.
  *
  * When the data stream is closed whether the upload is successful
- * or not the target storage pool will be refreshed to reflect pool
+ * or not an attempt will be made to refresh the target storage pool
+ * if an asynchronous build is not running in order to reflect pool
  * and volume changes as a result of the upload. Depending on
  * the target volume storage backend and the source stream type
  * for a successful upload, the target volume may take on the
diff --git a/src/storage/storage_driver.c b/src/storage/storage_driver.c
index 2b7a299706..d209f5afb8 100644
--- a/src/storage/storage_driver.c
+++ b/src/storage/storage_driver.c
@@ -2267,6 +2267,13 @@ virStorageVolPoolRefreshThread(void *opaque)
         goto cleanup;
     def = virStoragePoolObjGetDef(obj);
 
+    /* If some thread is building a new volume in the pool, then we cannot
+     * clear out all vols and refresh the pool. So we'll just pass. */
+    if (virStoragePoolObjGetAsyncjobs(obj) > 0) {
+        VIR_DEBUG("Asyncjob in process, cannot refresh storage pool");
+        goto cleanup;
+    }
+
     if (!(backend = virStorageBackendForType(def->type)))
         goto cleanup;
 
-- 
2.13.6

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, Nov 16, 2017 at 09:06:23AM -0500, John Ferlan wrote:
>Resolve a storage driver crash as a result of a long running
>storageVolCreateXML when the virStorageVolPoolRefreshThread is
>run as a result of when a storageVolUpload completed and ran the
>virStoragePoolObjClearVols without checking if the creation
>code was currently processing a buildVol after incrementing
>the driver->asyncjob count.
>
>The refreshThread will now check the pool asyncjob count before
>attempting to pursue the pool refresh. Adjust the documentation
>to describe the condition.
>
>Crash from valgrind is as follows (with a bit of editing):
>
>==21309== Invalid read of size 8
>==21309==    at 0x153E47AF: storageBackendUpdateVolTargetInfo
>==21309==    by 0x153E4C30: virStorageBackendUpdateVolInfo
>==21309==    by 0x153E52DE: virStorageBackendVolRefreshLocal
>==21309==    by 0x153DE29E: storageVolCreateXML
>==21309==    by 0x562035B: virStorageVolCreateXML
>==21309==    by 0x147366: remoteDispatchStorageVolCreateXML
>...
>==21309==  Address 0x2590a720 is 64 bytes inside a block of size 336 free'd
>==21309==    at 0x4C2F2BB: free
>==21309==    by 0x54CB9FA: virFree
>==21309==    by 0x55BC800: virStorageVolDefFree
>==21309==    by 0x55BF1D8: virStoragePoolObjClearVols
>==21309==    by 0x153D967E: virStorageVolPoolRefreshThread
>...
>==21309==  Block was alloc'd at
>==21309==    at 0x4C300A5: calloc
>==21309==    by 0x54CB483: virAlloc
>==21309==    by 0x55BDC1F: virStorageVolDefParseXML
>==21309==    by 0x55BDC1F: virStorageVolDefParseNode
>==21309==    by 0x55BE5A4: virStorageVolDefParse
>==21309==    by 0x153DDFF1: storageVolCreateXML
>==21309==    by 0x562035B: virStorageVolCreateXML
>==21309==    by 0x147366: remoteDispatchStorageVolCreateXML
>...
>
>Signed-off-by: John Ferlan <jferlan@redhat.com>
>---
>
> v1: https://www.redhat.com/archives/libvir-list/2017-November/msg00198.html
>
> Changes since v1:
>   - From review, remove the retry if Asyncjobs > 0 logic and replace with
>     a VIR_DEBUG indicating not doing refresh due to Asyncjob running and
>     just goto cleanup.
>
>   - Adjust the virStorageVolUpload docs to note that an attempt will be
>     made to refresh the pool.
>
>   - Drop patch 2 which added the Asyncjobs > 0 check to the SCSI pool
>     refresh thread. Since a SCSI pool doesn't support buildVol, the
>     asyncjob count cannot be anything other than zero, so it's pointless.
>
> src/libvirt-storage.c        | 3 ++-
> src/storage/storage_driver.c | 7 +++++++
> 2 files changed, 9 insertions(+), 1 deletion(-)
>

ACK

Jan
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list