[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism

Joel Fernandes posted 7 patches 3 months, 3 weeks ago
[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by Joel Fernandes 3 months, 3 weeks ago
While not terribly complicated, a little bit of documentation will help
augment the code for this very important mechanism.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 Documentation/gpu/nova/core/pramin.rst | 113 +++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst       |   1 +
 2 files changed, 114 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/pramin.rst

diff --git a/Documentation/gpu/nova/core/pramin.rst b/Documentation/gpu/nova/core/pramin.rst
new file mode 100644
index 000000000000..19615e504db9
--- /dev/null
+++ b/Documentation/gpu/nova/core/pramin.rst
@@ -0,0 +1,113 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=========================
+PRAMIN aperture mechanism
+=========================
+
+.. note::
+   The following description is approximate and current as of the Ampere family.
+   It may change for future generations and is intended to assist in understanding
+   the driver code.
+
+Introduction
+============
+
+PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video RAM (VRAM) before
+the GPU's Memory Management Unit (MMU) and page tables are initialized. This 1MB sliding window,
+located at a fixed offset within BAR0, is essential for setting up page tables and other critical
+GPU data structures without relying on the GPU's MMU.
+
+Architecture Overview
+=====================
+
+Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)
+and provides a CPU-accessible window into VRAM through the PCIe interface::
+
+    +-----------------+    PCIe     +------------------------------+
+    |      CPU        |<----------->|           GPU                |
+    +-----------------+             |                              |
+                                    |  +----------------------+    |
+                                    |  |       PBUS           |    |
+                                    |  |  (Bus Controller)    |    |
+                                    |  |                      |    |
+                                    |  |  +--------------.<------------ (window always starts at
+                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)
+                                    |  |  |   Window     |    |    |
+                                    |  |  |   (1MB)      |    |    |
+                                    |  |  +--------------+    |    |
+                                    |  |         |            |    |
+                                    |  +---------|------------+    |
+                                    |            |                 |
+                                    |            v                 |
+                                    |  .----------------------.<------------ (Program PRAMIN to any
+                                    |  |       VRAM           |    |    64KB VRAM physical boundary)
+                                    |  |    (Several GBs)     |    |
+                                    |  |                      |    |
+                                    |  |  FB[0x000000000000]  |    |
+                                    |  |          ...         |    |
+                                    |  |  FB[0x7FFFFFFFFFF]   |    |
+                                    |  +----------------------+    |
+                                    +------------------------------+
+
+PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIO
+accesses to the BAR registers.
+
+PRAMIN Window Operation
+=======================
+
+The PRAMIN window provides a 1MB sliding aperture that can be repositioned over
+the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register.
+
+Window Control Mechanism
+-------------------------
+
+The window position is controlled via the PBUS BAR0_WINDOW register::
+
+    NV_PBUS_BAR0_WINDOW Register
+    +-----+-----+--------------------------------------+
+    |31-26|25-24|           23-0                       |
+    |     |TARG |         BASE_ADDR                    |
+    |     | ET  |        (bits 39:16 of VRAM address)  |
+    +-----+-----+--------------------------------------+
+
+    TARGET field values:
+    - 0x0: VID_MEM (Video Memory / VRAM)
+    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
+    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
+
+64KB Alignment Requirement
+---------------------------
+
+The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
+by the BASE_ADDR field representing bits [39:16] of the target address::
+
+    VRAM Address Calculation:
+    actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
+    Where:
+    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
+    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
+    Example Window Positioning:
+    +---------------------------------------------------------+
+    |                    VRAM Space                           |
+    |                                                         |
+    |  0x000000000  +-----------------+ <-- 64KB aligned      |
+    |               | PRAMIN Window   |                       |
+    |               |    (1MB)        |                       |
+    |  0x0000FFFFF  +-----------------+                       |
+    |                                                         |
+    |       |              ^                                  |
+    |       |              | Window can slide                 |
+    |       v              | to any 64KB boundary             |
+    |                                                         |
+    |  0x123400000  +-----------------+ <-- 64KB aligned      |
+    |               | PRAMIN Window   |                       |
+    |               |    (1MB)        |                       |
+    |  0x1234FFFFF  +-----------------+                       |
+    |                                                         |
+    |                       ...                               |
+    |                                                         |
+    |  0x7FFFF0000  +-----------------+ <-- 64KB aligned      |
+    |               | PRAMIN Window   |                       |
+    |               |    (1MB)        |                       |
+    |  0x7FFFFFFFF  +-----------------+                       |
+    +---------------------------------------------------------+
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 46302daace34..e77d3ee336a4 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/fwsec
    core/falcon
    core/msgq
+   core/pramin
-- 
2.34.1
Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by Bagas Sanjaya 3 months, 2 weeks ago
On Mon, Oct 20, 2025 at 02:55:36PM -0400, Joel Fernandes wrote:
> +The window position is controlled via the PBUS BAR0_WINDOW register::
> +
> +    NV_PBUS_BAR0_WINDOW Register
> +    +-----+-----+--------------------------------------+
> +    |31-26|25-24|           23-0                       |
> +    |     |TARG |         BASE_ADDR                    |
> +    |     | ET  |        (bits 39:16 of VRAM address)  |
> +    +-----+-----+--------------------------------------+

Shouldn't the TARGET field cell above be fitted (extended)?

> +
> +    TARGET field values:
> +    - 0x0: VID_MEM (Video Memory / VRAM)
> +    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
> +    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
> +
> +64KB Alignment Requirement
> +---------------------------
> +
> +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
> +by the BASE_ADDR field representing bits [39:16] of the target address::
> +
> +    VRAM Address Calculation:
> +    actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
> +    Where:
> +    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
> +    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
> +    Example Window Positioning:

Move the supplementary explanation out of respective literal code blocks,
like::

---- >8 ----
diff --git a/Documentation/gpu/nova/core/pramin.rst b/Documentation/gpu/nova/core/pramin.rst
index 19615e504db9d5..47abe539e32a81 100644
--- a/Documentation/gpu/nova/core/pramin.rst
+++ b/Documentation/gpu/nova/core/pramin.rst
@@ -70,23 +70,28 @@ The window position is controlled via the PBUS BAR0_WINDOW register::
     |     | ET  |        (bits 39:16 of VRAM address)  |
     +-----+-----+--------------------------------------+
 
-    TARGET field values:
-    - 0x0: VID_MEM (Video Memory / VRAM)
-    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
-    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
+where TARGET field values are:
+
+  - 0x0: VID_MEM (Video Memory / VRAM)
+  - 0x1: SYS_MEM_COHERENT (Coherent system memory)
+  - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
 
 64KB Alignment Requirement
 ---------------------------
 
 The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
-by the BASE_ADDR field representing bits [39:16] of the target address::
+by the BASE_ADDR field representing bits [39:16] of the target address. The
+VRAM address calculation is determined by::
 
-    VRAM Address Calculation:
     actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
-    Where:
-    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
-    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
-    Example Window Positioning:
+
+where:
+
+  - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
+  - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
+
+Example window positioning::
+
     +---------------------------------------------------------+
     |                    VRAM Space                           |
     |                                                         |

Thanks. 

-- 
An old man doll... just what I always wanted! - Clara
Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by John Hubbard 3 months, 3 weeks ago
On 10/20/25 11:55 AM, Joel Fernandes wrote:
> While not terribly complicated, a little bit of documentation will help
> augment the code for this very important mechanism.
> 
> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> ---
>  Documentation/gpu/nova/core/pramin.rst | 113 +++++++++++++++++++++++++
>  Documentation/gpu/nova/index.rst       |   1 +
>  2 files changed, 114 insertions(+)
>  create mode 100644 Documentation/gpu/nova/core/pramin.rst

Hi Joel,

Here's a more thorough review.

> 
> diff --git a/Documentation/gpu/nova/core/pramin.rst b/Documentation/gpu/nova/core/pramin.rst
> new file mode 100644
> index 000000000000..19615e504db9
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/pramin.rst
> @@ -0,0 +1,113 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +=========================
> +PRAMIN aperture mechanism
> +=========================
> +
> +.. note::
> +   The following description is approximate and current as of the Ampere family.
> +   It may change for future generations and is intended to assist in understanding
> +   the driver code.
> +
> +Introduction
> +============
> +
> +PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video RAM (VRAM) before
> +the GPU's Memory Management Unit (MMU) and page tables are initialized. This 1MB sliding window,
> +located at a fixed offset within BAR0, is essential for setting up page tables and other critical
> +GPU data structures without relying on the GPU's MMU.
> +
> +Architecture Overview
> +=====================
> +
> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)

How about:

The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe Bus Controller Unit)


> +and provides a CPU-accessible window into VRAM through the PCIe interface::
> +
> +    +-----------------+    PCIe     +------------------------------+
> +    |      CPU        |<----------->|           GPU                |
> +    +-----------------+             |                              |
> +                                    |  +----------------------+    |
> +                                    |  |       PBUS           |    |
> +                                    |  |  (Bus Controller)    |    |
> +                                    |  |                      |    |
> +                                    |  |  +--------------.<------------ (window always starts at
> +                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)
> +                                    |  |  |   Window     |    |    |
> +                                    |  |  |   (1MB)      |    |    |
> +                                    |  |  +--------------+    |    |
> +                                    |  |         |            |    |
> +                                    |  +---------|------------+    |
> +                                    |            |                 |
> +                                    |            v                 |
> +                                    |  .----------------------.<------------ (Program PRAMIN to any
> +                                    |  |       VRAM           |    |    64KB VRAM physical boundary)
> +                                    |  |    (Several GBs)     |    |
> +                                    |  |                      |    |
> +                                    |  |  FB[0x000000000000]  |    |
> +                                    |  |          ...         |    |
> +                                    |  |  FB[0x7FFFFFFFFFF]   |    |
> +                                    |  +----------------------+    |
> +                                    +------------------------------+
> +
> +PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIO

How about:

PBUS (PCIe Bus Controller) is responsible for, among other things, handling MMIO

> +accesses to the BAR registers.
> +
> +PRAMIN Window Operation
> +=======================
> +
> +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over
> +the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register.
> +
> +Window Control Mechanism
> +-------------------------
> +
> +The window position is controlled via the PBUS BAR0_WINDOW register::
> +
> +    NV_PBUS_BAR0_WINDOW Register
> +    +-----+-----+--------------------------------------+
> +    |31-26|25-24|           23-0                       |

Should we use ":" notation here?

31:26 | 25:24 | 23:0

> +    |     |TARG |         BASE_ADDR                    |
> +    |     | ET  |        (bits 39:16 of VRAM address)  |
> +    +-----+-----+--------------------------------------+
> +
> +    TARGET field values:
> +    - 0x0: VID_MEM (Video Memory / VRAM)
> +    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
> +    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)

These SYS_MEM_* items imply to the reader that PRAMIN can target
sysmem. However, pramin.rs in this series hard-codes this field
to VID_MEM:

let window_reg = regs::NV_PBUS_BAR0_WINDOW::default().set_window_addr(fb_offset);

It's not clear why that is done (I seem to recall a potential deadlock
in older chips, in similar situations, but that's probably stale info).

If, however, there is some limitation that prevents using SYS_MEM*, then
we should document it, probably here.

> +
> +64KB Alignment Requirement
> +---------------------------
> +
> +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
> +by the BASE_ADDR field representing bits [39:16] of the target address::
> +
> +    VRAM Address Calculation:
> +    actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
> +    Where:
> +    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
> +    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]

How about:

      - pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF]

> +    Example Window Positioning:
> +    +---------------------------------------------------------+
> +    |                    VRAM Space                           |
> +    |                                                         |
> +    |  0x000000000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x0000FFFFF  +-----------------+                       |
> +    |                                                         |
> +    |       |              ^                                  |
> +    |       |              | Window can slide                 |
> +    |       v              | to any 64KB boundary             |

Maybe:

    s/any 64KB boundary/any 64KB-aligned boundary/

> +    |                                                         |
> +    |  0x123400000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x1234FFFFF  +-----------------+                       |
> +    |                                                         |
> +    |                       ...                               |
> +    |                                                         |
> +    |  0x7FFFF0000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x7FFFFFFFF  +-----------------+                       |
> +    +---------------------------------------------------------+
> diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
> index 46302daace34..e77d3ee336a4 100644
> --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver.
>     core/fwsec
>     core/falcon
>     core/msgq
> +   core/pramin

thanks,
-- 
John Hubbard
Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by John Hubbard 3 months, 3 weeks ago
On 10/20/25 11:55 AM, Joel Fernandes wrote:
...> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)
> +and provides a CPU-accessible window into VRAM through the PCIe interface::
> +
> +    +-----------------+    PCIe     +------------------------------+
> +    |      CPU        |<----------->|           GPU                |
> +    +-----------------+             |                              |
> +                                    |  +----------------------+    |
> +                                    |  |       PBUS           |    |
> +                                    |  |  (Bus Controller)    |    |
> +                                    |  |                      |    |
> +                                    |  |  +--------------.<------------ (window always starts at
> +                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)

Quick question: does "window always starts at" actually mean "windows
is always initialized to" ? Or something else?


thanks,
-- 
John Hubbard

> +                                    |  |  |   Window     |    |    |
> +                                    |  |  |   (1MB)      |    |    |
> +                                    |  |  +--------------+    |    |
> +                                    |  |         |            |    |
> +                                    |  +---------|------------+    |
> +                                    |            |                 |
> +                                    |            v                 |
> +                                    |  .----------------------.<------------ (Program PRAMIN to any
> +                                    |  |       VRAM           |    |    64KB VRAM physical boundary)
> +                                    |  |    (Several GBs)     |    |
> +                                    |  |                      |    |
> +                                    |  |  FB[0x000000000000]  |    |
> +                                    |  |          ...         |    |
> +                                    |  |  FB[0x7FFFFFFFFFF]   |    |
> +                                    |  +----------------------+    |
> +                                    +------------------------------+
> +
> +PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIO
> +accesses to the BAR registers.
> +
> +PRAMIN Window Operation
> +=======================
> +
> +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over
> +the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register.
> +
> +Window Control Mechanism
> +-------------------------
> +
> +The window position is controlled via the PBUS BAR0_WINDOW register::
> +
> +    NV_PBUS_BAR0_WINDOW Register
> +    +-----+-----+--------------------------------------+
> +    |31-26|25-24|           23-0                       |
> +    |     |TARG |         BASE_ADDR                    |
> +    |     | ET  |        (bits 39:16 of VRAM address)  |
> +    +-----+-----+--------------------------------------+
> +
> +    TARGET field values:
> +    - 0x0: VID_MEM (Video Memory / VRAM)
> +    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
> +    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
> +
> +64KB Alignment Requirement
> +---------------------------
> +
> +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
> +by the BASE_ADDR field representing bits [39:16] of the target address::
> +
> +    VRAM Address Calculation:
> +    actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
> +    Where:
> +    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
> +    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
> +    Example Window Positioning:
> +    +---------------------------------------------------------+
> +    |                    VRAM Space                           |
> +    |                                                         |
> +    |  0x000000000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x0000FFFFF  +-----------------+                       |
> +    |                                                         |
> +    |       |              ^                                  |
> +    |       |              | Window can slide                 |
> +    |       v              | to any 64KB boundary             |
> +    |                                                         |
> +    |  0x123400000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x1234FFFFF  +-----------------+                       |
> +    |                                                         |
> +    |                       ...                               |
> +    |                                                         |
> +    |  0x7FFFF0000  +-----------------+ <-- 64KB aligned      |
> +    |               | PRAMIN Window   |                       |
> +    |               |    (1MB)        |                       |
> +    |  0x7FFFFFFFF  +-----------------+                       |
> +    +---------------------------------------------------------+
> diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
> index 46302daace34..e77d3ee336a4 100644
> --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver.
>     core/fwsec
>     core/falcon
>     core/msgq
> +   core/pramin
Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by Joel Fernandes 3 months, 3 weeks ago

> On Oct 20, 2025, at 3:36 PM, John Hubbard <jhubbard@nvidia.com> wrote:
> 
> On 10/20/25 11:55 AM, Joel Fernandes wrote:
> ...> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)
>> +and provides a CPU-accessible window into VRAM through the PCIe interface::
>> +
>> +    +-----------------+    PCIe     +------------------------------+
>> +    |      CPU        |<----------->|           GPU                |
>> +    +-----------------+             |                              |
>> +                                    |  +----------------------+    |
>> +                                    |  |       PBUS           |    |
>> +                                    |  |  (Bus Controller)    |    |
>> +                                    |  |                      |    |
>> +                                    |  |  +--------------.<------------ (window always starts at
>> +                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)
> 
> Quick question: does "window always starts at" actually mean "windows
> is always initialized to" ? Or something else?

It means start of the 1MB window (offset 0) is accessed from the BAR at that location.

Thanks.



> 
> 
> thanks,
> --
> John Hubbard
> 
>> +                                    |  |  |   Window     |    |    |
>> +                                    |  |  |   (1MB)      |    |    |
>> +                                    |  |  +--------------+    |    |
>> +                                    |  |         |            |    |
>> +                                    |  +---------|------------+    |
>> +                                    |            |                 |
>> +                                    |            v                 |
>> +                                    |  .----------------------.<------------ (Program PRAMIN to any
>> +                                    |  |       VRAM           |    |    64KB VRAM physical boundary)
>> +                                    |  |    (Several GBs)     |    |
>> +                                    |  |                      |    |
>> +                                    |  |  FB[0x000000000000]  |    |
>> +                                    |  |          ...         |    |
>> +                                    |  |  FB[0x7FFFFFFFFFF]   |    |
>> +                                    |  +----------------------+    |
>> +                                    +------------------------------+
>> +
>> +PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIO
>> +accesses to the BAR registers.
>> +
>> +PRAMIN Window Operation
>> +=======================
>> +
>> +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over
>> +the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register.
>> +
>> +Window Control Mechanism
>> +-------------------------
>> +
>> +The window position is controlled via the PBUS BAR0_WINDOW register::
>> +
>> +    NV_PBUS_BAR0_WINDOW Register
>> +    +-----+-----+--------------------------------------+
>> +    |31-26|25-24|           23-0                       |
>> +    |     |TARG |         BASE_ADDR                    |
>> +    |     | ET  |        (bits 39:16 of VRAM address)  |
>> +    +-----+-----+--------------------------------------+
>> +
>> +    TARGET field values:
>> +    - 0x0: VID_MEM (Video Memory / VRAM)
>> +    - 0x1: SYS_MEM_COHERENT (Coherent system memory)
>> +    - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
>> +
>> +64KB Alignment Requirement
>> +---------------------------
>> +
>> +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
>> +by the BASE_ADDR field representing bits [39:16] of the target address::
>> +
>> +    VRAM Address Calculation:
>> +    actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
>> +    Where:
>> +    - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
>> +    - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
>> +    Example Window Positioning:
>> +    +---------------------------------------------------------+
>> +    |                    VRAM Space                           |
>> +    |                                                         |
>> +    |  0x000000000  +-----------------+ <-- 64KB aligned      |
>> +    |               | PRAMIN Window   |                       |
>> +    |               |    (1MB)        |                       |
>> +    |  0x0000FFFFF  +-----------------+                       |
>> +    |                                                         |
>> +    |       |              ^                                  |
>> +    |       |              | Window can slide                 |
>> +    |       v              | to any 64KB boundary             |
>> +    |                                                         |
>> +    |  0x123400000  +-----------------+ <-- 64KB aligned      |
>> +    |               | PRAMIN Window   |                       |
>> +    |               |    (1MB)        |                       |
>> +    |  0x1234FFFFF  +-----------------+                       |
>> +    |                                                         |
>> +    |                       ...                               |
>> +    |                                                         |
>> +    |  0x7FFFF0000  +-----------------+ <-- 64KB aligned      |
>> +    |               | PRAMIN Window   |                       |
>> +    |               |    (1MB)        |                       |
>> +    |  0x7FFFFFFFF  +-----------------+                       |
>> +    +---------------------------------------------------------+
>> diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
>> index 46302daace34..e77d3ee336a4 100644
>> --- a/Documentation/gpu/nova/index.rst
>> +++ b/Documentation/gpu/nova/index.rst
>> @@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver.
>>    core/fwsec
>>    core/falcon
>>    core/msgq
>> +   core/pramin
> 
> 
Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by John Hubbard 3 months, 3 weeks ago
On 10/20/25 12:48 PM, Joel Fernandes wrote:
> 
> 
>> On Oct 20, 2025, at 3:36 PM, John Hubbard <jhubbard@nvidia.com> wrote:
>>
>> On 10/20/25 11:55 AM, Joel Fernandes wrote:
>> ...> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)
>>> +and provides a CPU-accessible window into VRAM through the PCIe interface::
>>> +
>>> +    +-----------------+    PCIe     +------------------------------+
>>> +    |      CPU        |<----------->|           GPU                |
>>> +    +-----------------+             |                              |
>>> +                                    |  +----------------------+    |
>>> +                                    |  |       PBUS           |    |
>>> +                                    |  |  (Bus Controller)    |    |
>>> +                                    |  |                      |    |
>>> +                                    |  |  +--------------.<------------ (window always starts at
>>> +                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)
>>
>> Quick question: does "window always starts at" actually mean "windows
>> is always initialized to" ? Or something else?
> 
> It means start of the 1MB window (offset 0) is accessed from the BAR at that location.
> 
OK, yes. May I suggest this slightly tweaked wording:

   (window into VRAM starts at BAR0 + 0x700000)

? 

This avoids "always" (because HW may change someday), and also
gives a subtly stronger hint about how this is laid out.



thanks,
-- 
John Hubbard

Re: [PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
Posted by Joel Fernandes 3 months, 3 weeks ago

On 10/20/2025 4:42 PM, John Hubbard wrote:
> On 10/20/25 12:48 PM, Joel Fernandes wrote:
>>
>>
>>> On Oct 20, 2025, at 3:36 PM, John Hubbard <jhubbard@nvidia.com> wrote:
>>>
>>> On 10/20/25 11:55 AM, Joel Fernandes wrote:
>>> ...> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)
>>>> +and provides a CPU-accessible window into VRAM through the PCIe interface::
>>>> +
>>>> +    +-----------------+    PCIe     +------------------------------+
>>>> +    |      CPU        |<----------->|           GPU                |
>>>> +    +-----------------+             |                              |
>>>> +                                    |  +----------------------+    |
>>>> +                                    |  |       PBUS           |    |
>>>> +                                    |  |  (Bus Controller)    |    |
>>>> +                                    |  |                      |    |
>>>> +                                    |  |  +--------------.<------------ (window always starts at
>>>> +                                    |  |  |   PRAMIN     |    |    |     BAR0 + 0x700000)
>>>
>>> Quick question: does "window always starts at" actually mean "windows
>>> is always initialized to" ? Or something else?
>>
>> It means start of the 1MB window (offset 0) is accessed from the BAR at that location.
>>
> OK, yes. May I suggest this slightly tweaked wording:
> 
>    (window into VRAM starts at BAR0 + 0x700000)
> 
> ? 
> 
> This avoids "always" (because HW may change someday), and also
> gives a subtly stronger hint about how this is laid out.
> 

Sure, I will change to this wording for next posting. Thanks,

 - Joel