From nobody Tue Feb 10 06:43:55 2026 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1572293574; cv=none; d=zoho.com; s=zohoarc; b=Bbpbf3MOlNdBzM1qU6ZnpASe3czpXhOhtqZCDHdvn8GD6xLnalBtMkA3WUBwJ2loQq8viFBofnTvLFo9ubqAV35BFJvDXAs6Kf34joP3WGjZvIVQhxfihyiOmAofecIIdpYxJ97zeaaZHFKghAuaLVkDQINMdg2Y0ie4dDlOT/c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1572293574; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ofl43nwN8IKh3AF7wEAYBvmYefg7YGGmq8Ln0DllgiQ=; b=V8Tw17zdW4uQnbE3qpf6397aIOd/+ge7/X118oRwbBlzMbYNysYiSYhJxhSC/un+OYkuaK84M3Tk9aLJJ1HPIU91KGaYXTw6ZTdFH2/2wogl7URffxXE2RugwG1xjzKH3xWERkKA1H73AaCvUguPt7gMVyl9k6FUhs6WYqlaxso= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1572293574052854.3937679491648; Mon, 28 Oct 2019 13:12:54 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iPBMh-0001wr-JJ; Mon, 28 Oct 2019 20:11:47 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iPBMf-0001va-Or for xen-devel@lists.xenproject.org; Mon, 28 Oct 2019 20:11:45 +0000 Received: from mail-qk1-x741.google.com (unknown [2607:f8b0:4864:20::741]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 09a03874-f9bf-11e9-a531-bc764e2007e4; Mon, 28 Oct 2019 20:10:50 +0000 (UTC) Received: by mail-qk1-x741.google.com with SMTP id y81so9744857qkb.7 for ; Mon, 28 Oct 2019 13:10:50 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id u9sm1501115qke.50.2019.10.28.13.10.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 28 Oct 2019 13:10:46 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iPBLf-0001h3-HR; Mon, 28 Oct 2019 17:10:43 -0300 X-Inumbo-ID: 09a03874-f9bf-11e9-a531-bc764e2007e4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PQZNLb2n/VZGpEmak/Qy43FMq9i6D/cb88Nb6FZ4HCc=; b=JDfbNHty1ny9u0tolzz+R0bxyhO0ollb7F5vPylbs5XjRPew3ElqUgkRpnO1KqyNHJ lbmxT+WbuV2iqQKttXAinvJg6mLlpZOBkINKsbyZqgOCdsmgbxz+ymyca8lTAo2EIudp RdrM54tOjIVl+x7WKO8GaCpeaojoHs5YLsjcx7foxAOaiA94okZYV8o6vQfQCwuPbxr0 yggvPWtLeAKJS+9waHFzTuYqiUemPetepD5ewVTHRTeKTa3PYZPmq7T889b0g2RSfWRi iL7jN1WDPg49bHl8qlfraAoRYGBU7o23Ymndp/Dt+umlDMRAc3zH7TPYUwjq23HTYvtx Z/OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PQZNLb2n/VZGpEmak/Qy43FMq9i6D/cb88Nb6FZ4HCc=; b=P25bcr30HvILWKJlRCjw/0NtSClRFopptB0FWcP5bKzzChP59ed2Xi499HMbUIAYYN bl2FZVF2wAkgIxBpXzZ4+Tpkz2Kclmehg9a1p3ukMHJI5xGHiT6hGI67iAsCAPvedkWh zWm9Ump7rh47TmHO0syVDiSDsvGgSWn0PMHzN1GVpsu2+oTO5T7CUVweAq7yxcdoOAp7 SXlc8HzcBlJCzDACOy6IlgmCtm3GO6XejfsXwQzqnhWWIaAYEKGGQmv1RrtxhMs/mcc0 iekMkqBFPC7piqrHxQI2eRFVcy+Ad9wy7jrEzjW1ybuOrvIK+Zn5pPIsShy16yigZYlk zuvQ== X-Gm-Message-State: APjAAAUB7MZ3oPlfYkBVltZMVsjU0te6fG7iXShwoqNJU2aXF4X//C3j k5QFM4uMx4oYMx14udpFD8qAKZRvYgY= X-Google-Smtp-Source: APXvYqzMU7VQvTju2rb3w6e+2fL+RjV7hRaY2VBwnJrbzaMQL58Q1wBtGQ3o/J/TSiYTcAN3ImKl5A== X-Received: by 2002:a37:6643:: with SMTP id a64mr18555491qkc.144.1572293450333; Mon, 28 Oct 2019 13:10:50 -0700 (PDT) From: Jason Gunthorpe To: linux-mm@kvack.org, Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Date: Mon, 28 Oct 2019 17:10:28 -0300 Message-Id: <20191028201032.6352-12-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191028201032.6352-1-jgg@ziepe.ca> References: <20191028201032.6352-1-jgg@ziepe.ca> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 11/15] nouveau: use mmu_range_notifier instead of hmm_mirror X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , David Zhou , Mike Marciniszyn , Stefano Stabellini , Oleksandr Andrushchenko , linux-rdma@vger.kernel.org, nouveau@lists.freedesktop.org, Dennis Dalessandro , amd-gfx@lists.freedesktop.org, Christoph Hellwig , Jason Gunthorpe , dri-devel@lists.freedesktop.org, Alex Deucher , xen-devel@lists.xenproject.org, Boris Ostrovsky , Petr Cvek , =?UTF-8?q?Christian=20K=C3=B6nig?= , Ben Skeggs Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Jason Gunthorpe Remove the hmm_mirror object and use the mmu_range_notifier API instead for the range, and use the normal mmu_notifier API for the general invalidation callback. While here re-organize the pagefault path so the locking pattern is clear. nouveau is the only driver that uses a temporary range object and instead forwards nearly every invalidation range directly to the HW. While this is not how the mmu_range_notifier was intended to be used, the overheads on the pagefaulting path are similar to the existing hmm_mirror version. Particularly since the interval tree will be small. Cc: Ben Skeggs Cc: dri-devel@lists.freedesktop.org Cc: nouveau@lists.freedesktop.org Cc: Ralph Campbell Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/nouveau/nouveau_svm.c | 180 ++++++++++++++------------ 1 file changed, 100 insertions(+), 80 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouvea= u/nouveau_svm.c index 577f8811925a59..f27317fbe36f45 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -96,8 +96,6 @@ struct nouveau_svmm { } unmanaged; =20 struct mutex mutex; - - struct hmm_mirror mirror; }; =20 #define SVMM_DBG(s,f,a...) = \ @@ -293,23 +291,11 @@ static const struct mmu_notifier_ops nouveau_mn_ops = =3D { .free_notifier =3D nouveau_svmm_free_notifier, }; =20 -static int -nouveau_svmm_sync_cpu_device_pagetables(struct hmm_mirror *mirror, - const struct mmu_notifier_range *update) -{ - return 0; -} - -static const struct hmm_mirror_ops nouveau_svmm =3D { - .sync_cpu_device_pagetables =3D nouveau_svmm_sync_cpu_device_pagetables, -}; - void nouveau_svmm_fini(struct nouveau_svmm **psvmm) { struct nouveau_svmm *svmm =3D *psvmm; if (svmm) { - hmm_mirror_unregister(&svmm->mirror); mutex_lock(&svmm->mutex); svmm->vmm =3D NULL; mutex_unlock(&svmm->mutex); @@ -357,15 +343,10 @@ nouveau_svmm_init(struct drm_device *dev, void *data, goto out_free; =20 down_write(¤t->mm->mmap_sem); - svmm->mirror.ops =3D &nouveau_svmm; - ret =3D hmm_mirror_register(&svmm->mirror, current->mm); - if (ret) - goto out_mm_unlock; - svmm->notifier.ops =3D &nouveau_mn_ops; ret =3D __mmu_notifier_register(&svmm->notifier, current->mm); if (ret) - goto out_hmm_unregister; + goto out_mm_unlock; /* Note, ownership of svmm transfers to mmu_notifier */ =20 cli->svm.svmm =3D svmm; @@ -374,8 +355,6 @@ nouveau_svmm_init(struct drm_device *dev, void *data, mutex_unlock(&cli->mutex); return 0; =20 -out_hmm_unregister: - hmm_mirror_unregister(&svmm->mirror); out_mm_unlock: up_write(¤t->mm->mmap_sem); out_free: @@ -503,43 +482,91 @@ nouveau_svm_fault_cache(struct nouveau_svm *svm, fault->inst, fault->addr, fault->access); } =20 -static inline bool -nouveau_range_done(struct hmm_range *range) +struct svm_notifier { + struct mmu_range_notifier notifier; + struct nouveau_svmm *svmm; +}; + +static bool nouveau_svm_range_invalidate(struct mmu_range_notifier *mrn, + const struct mmu_notifier_range *range, + unsigned long cur_seq) { - bool ret =3D hmm_range_valid(range); + struct svm_notifier *sn =3D + container_of(mrn, struct svm_notifier, notifier); =20 - hmm_range_unregister(range); - return ret; + /* + * serializes the update to mrn->invalidate_seq done by caller and + * prevents invalidation of the PTE from progressing while HW is being + * programmed. This is very hacky and only works because the normal + * notifier that does invalidation is always called after the range + * notifier. + */ + if (mmu_notifier_range_blockable(range)) + mutex_lock(&sn->svmm->mutex); + else if (!mutex_trylock(&sn->svmm->mutex)) + return false; + mmu_range_set_seq(mrn, cur_seq); + mutex_unlock(&sn->svmm->mutex); + return true; } =20 -static int -nouveau_range_fault(struct nouveau_svmm *svmm, struct hmm_range *range) +static const struct mmu_range_notifier_ops nouveau_svm_mrn_ops =3D { + .invalidate =3D nouveau_svm_range_invalidate, +}; + +static int nouveau_range_fault(struct nouveau_svmm *svmm, + struct nouveau_drm *drm, void *data, u32 size, + u64 *pfns, + struct svm_notifier *notifier) { + unsigned long timeout =3D + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + /* Have HMM fault pages within the fault window to the GPU. */ + struct hmm_range range =3D { + .notifier =3D ¬ifier->notifier, + .start =3D notifier->notifier.interval_tree.start, + .end =3D notifier->notifier.interval_tree.last + 1, + .pfns =3D pfns, + .flags =3D nouveau_svm_pfn_flags, + .values =3D nouveau_svm_pfn_values, + .pfn_shift =3D NVIF_VMM_PFNMAP_V0_ADDR_SHIFT, + }; + struct mm_struct *mm =3D notifier->notifier.mm; long ret; =20 - range->default_flags =3D 0; - range->pfn_flags_mask =3D -1UL; + while (true) { + if (time_after(jiffies, timeout)) + return -EBUSY; =20 - ret =3D hmm_range_register(range, &svmm->mirror); - if (ret) { - up_read(&svmm->notifier.mm->mmap_sem); - return (int)ret; - } + range.notifier_seq =3D mmu_range_read_begin(range.notifier); + range.default_flags =3D 0; + range.pfn_flags_mask =3D -1UL; + down_read(&mm->mmap_sem); + ret =3D hmm_range_fault(&range, 0); + up_read(&mm->mmap_sem); + if (ret <=3D 0) { + if (ret =3D=3D 0 || ret =3D=3D -EBUSY) + continue; + return ret; + } =20 - if (!hmm_range_wait_until_valid(range, HMM_RANGE_DEFAULT_TIMEOUT)) { - up_read(&svmm->notifier.mm->mmap_sem); - return -EBUSY; + mutex_lock(&svmm->mutex); + if (mmu_range_read_retry(range.notifier, + range.notifier_seq)) { + mutex_unlock(&svmm->mutex); + continue; + } + break; } =20 - ret =3D hmm_range_fault(range, 0); - if (ret <=3D 0) { - if (ret =3D=3D 0) - ret =3D -EBUSY; - up_read(&svmm->notifier.mm->mmap_sem); - hmm_range_unregister(range); - return ret; - } - return 0; + nouveau_dmem_convert_pfn(drm, &range); + + svmm->vmm->vmm.object.client->super =3D true; + ret =3D nvif_object_ioctl(&svmm->vmm->vmm.object, data, size, NULL); + svmm->vmm->vmm.object.client->super =3D false; + mutex_unlock(&svmm->mutex); + + return ret; } =20 static int @@ -559,7 +586,6 @@ nouveau_svm_fault(struct nvif_notify *notify) } i; u64 phys[16]; } args; - struct hmm_range range; struct vm_area_struct *vma; u64 inst, start, limit; int fi, fn, pi, fill; @@ -615,6 +641,7 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.version =3D 0; =20 for (fi =3D 0; fn =3D fi + 1, fi < buffer->fault_nr; fi =3D fn) { + struct svm_notifier notifier; struct mm_struct *mm; =20 /* Cancel any faults from non-SVM channels. */ @@ -623,7 +650,6 @@ nouveau_svm_fault(struct nvif_notify *notify) continue; } SVMM_DBG(svmm, "addr %016llx", buffer->fault[fi]->addr); - mm =3D svmm->notifier.mm; =20 /* We try and group handling of faults within a small * window into a single update. @@ -637,6 +663,12 @@ nouveau_svm_fault(struct nvif_notify *notify) start =3D max_t(u64, start, svmm->unmanaged.limit); SVMM_DBG(svmm, "wndw %016llx-%016llx", start, limit); =20 + mm =3D svmm->notifier.mm; + if (!mmget_not_zero(mm)) { + nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); + continue; + } + /* Intersect fault window with the CPU VMA, cancelling * the fault if the address is invalid. */ @@ -645,16 +677,18 @@ nouveau_svm_fault(struct nvif_notify *notify) if (!vma) { SVMM_ERR(svmm, "wndw %016llx-%016llx", start, limit); up_read(&mm->mmap_sem); + mmput(mm); nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); continue; } start =3D max_t(u64, start, vma->vm_start); limit =3D min_t(u64, limit, vma->vm_end); + up_read(&mm->mmap_sem); SVMM_DBG(svmm, "wndw %016llx-%016llx", start, limit); =20 if (buffer->fault[fi]->addr !=3D start) { SVMM_ERR(svmm, "addr %016llx", buffer->fault[fi]->addr); - up_read(&mm->mmap_sem); + mmput(mm); nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); continue; } @@ -710,33 +744,19 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.addr, args.i.p.addr + args.i.p.size, fn - fi); =20 - /* Have HMM fault pages within the fault window to the GPU. */ - range.start =3D args.i.p.addr; - range.end =3D args.i.p.addr + args.i.p.size; - range.pfns =3D args.phys; - range.flags =3D nouveau_svm_pfn_flags; - range.values =3D nouveau_svm_pfn_values; - range.pfn_shift =3D NVIF_VMM_PFNMAP_V0_ADDR_SHIFT; -again: - ret =3D nouveau_range_fault(svmm, &range); - if (ret =3D=3D 0) { - mutex_lock(&svmm->mutex); - if (!nouveau_range_done(&range)) { - mutex_unlock(&svmm->mutex); - goto again; - } - - nouveau_dmem_convert_pfn(svm->drm, &range); - - svmm->vmm->vmm.object.client->super =3D true; - ret =3D nvif_object_ioctl(&svmm->vmm->vmm.object, - &args, sizeof(args.i) + - pi * sizeof(args.phys[0]), - NULL); - svmm->vmm->vmm.object.client->super =3D false; - mutex_unlock(&svmm->mutex); - up_read(&mm->mmap_sem); + notifier.svmm =3D svmm; + notifier.notifier.ops =3D &nouveau_svm_mrn_ops; + ret =3D mmu_range_notifier_insert(¬ifier.notifier, + args.i.p.addr, args.i.p.size, + svmm->notifier.mm); + if (!ret) { + ret =3D nouveau_range_fault( + svmm, svm->drm, &args, + sizeof(args.i) + pi * sizeof(args.phys[0]), + args.phys, ¬ifier); + mmu_range_notifier_remove(¬ifier.notifier); } + mmput(mm); =20 /* Cancel any faults in the window whose pages didn't manage * to keep their valid bit, or stay writeable when required. @@ -745,10 +765,10 @@ nouveau_svm_fault(struct nvif_notify *notify) */ while (fi < fn) { struct nouveau_svm_fault *fault =3D buffer->fault[fi++]; - pi =3D (fault->addr - range.start) >> PAGE_SHIFT; + pi =3D (fault->addr - args.i.p.addr) >> PAGE_SHIFT; if (ret || - !(range.pfns[pi] & NVIF_VMM_PFNMAP_V0_V) || - (!(range.pfns[pi] & NVIF_VMM_PFNMAP_V0_W) && + !(args.phys[pi] & NVIF_VMM_PFNMAP_V0_V) || + (!(args.phys[pi] & NVIF_VMM_PFNMAP_V0_W) && fault->access !=3D 0 && fault->access !=3D 3)) { nouveau_svm_fault_cancel_fault(svm, fault); continue; --=20 2.23.0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel