| 1 | Hi, | 1 | Hi, |
|---|---|---|---|
| 2 | 2 | ||
| 3 | In v4, the approach to extending the BAR was changed to follow the | 3 | Another revision after the feedback from Ilpo. |
| 4 | suggestion from Christian. | 4 | Few tweaks here and there, the biggest change is the removal of the |
| 5 | The control is now entirely on the driver side, with PCI subsystem | 5 | loop in pci_iov_vf_bar_get_sizes(). |
| 6 | checking whether the resource fits during VF enabling. | ||
| 7 | I also added helpers to move between IOV / BAR # (suggested by Ilpo), | ||
| 8 | since the ifdefs started to look mighty ugly after addressing some of | ||
| 9 | the review feedback. | ||
| 10 | 6 | ||
| 11 | v3 can be found here: | 7 | v6 can be found here: |
| 12 | https://lore.kernel.org/dri-devel/20241010103203.382898-1-michal.winiarski@intel.com/ | 8 | https://lore.kernel.org/linux-pci/20250320110854.3866284-1-michal.winiarski@intel.com/ |
| 13 | 9 | ||
| 14 | For regular BAR, drivers can use pci_resize_resource to resize it to the | 10 | For regular BAR, drivers can use pci_resize_resource to resize it to the |
| 15 | desired size provided that it is supported by the hardware, which the | 11 | desired size provided that it is supported by the hardware, which the |
| 16 | driver can query using pci_rebar_get_possible_sizes. | 12 | driver can query using pci_rebar_get_possible_sizes. |
| 17 | This series expands the API to work with IOV BAR as well. | 13 | This series expands the API to work with IOV BAR as well. |
| ... | ... | ||
| 20 | resource boundary). | 16 | resource boundary). |
| 21 | 17 | ||
| 22 | Thanks, | 18 | Thanks, |
| 23 | -Michał | 19 | -Michał |
| 24 | 20 | ||
| 25 | v1 -> v2: | 21 | v6 -> v7: |
| 26 | - Add pci_iov_resource_extend() and usage in Xe driver | 22 | - Eliminate the loop in pci_iov_vf_bar_get_sizes() (Ilpo) |
| 27 | - Reduce the number of ifdefs (Christian) | 23 | - Use helper variable for indexes (Ilpo) |
| 28 | - Drop patch 2/2 from v1 (Christian) | 24 | - Kerneldoc formatting (Ilpo) |
| 29 | - Add a helper to avoid upsetting static analysis tools (Krzysztof) | 25 | - Refer to latest PCI spec (Ilpo) |
| 26 | - Commit message wording (Ilpo) | ||
| 27 | |||
| 28 | v5 -> v6: | ||
| 29 | - Rebased on latest pci/next | ||
| 30 | - Cache the VF resizable BAR capability position to avoid multiple | ||
| 31 | lookups (Ilpo) | ||
| 32 | - Use pci_resource_n helper (Ilpo) | ||
| 33 | |||
| 34 | v4 -> v5: | ||
| 35 | - Rename pci_resource_to/from_iov helpers and add WARN if called without | ||
| 36 | CONFIG_PCI_IOV (Ilpo) | ||
| 37 | - Reword kerneldoc for pci_iov_vf_bar_get_sizes (Bjorn) | ||
| 38 | - Reword commit message for VF BAR size check, extract the additional | ||
| 39 | size check to separate conditional (Bjorn) | ||
| 40 | |||
| 41 | v3 -> v4: | ||
| 42 | - Change the approach to extending the BAR (Christian) | ||
| 43 | - Tidy the commit messages, use 80 line limit where necessary (Bjorn) | ||
| 44 | - Add kerneldocs to exported functions (Bjorn) | ||
| 45 | - Add pci_resource_to_iov() / pci_resource_from_iov() helpers (Ilpo) | ||
| 46 | - Use FIELD_GET(), tidy whitespace (Ilpo) | ||
| 30 | 47 | ||
| 31 | v2 -> v3: | 48 | v2 -> v3: |
| 32 | - Extract introducing pci_resource_is_iov to separate commit and | 49 | - Extract introducing pci_resource_is_iov to separate commit and |
| 33 | use it elsewhere in PCI subsystem (Christian) | 50 | use it elsewhere in PCI subsystem (Christian) |
| 34 | - Extract restoring VF rebar state to separate commit (Christian) | 51 | - Extract restoring VF rebar state to separate commit (Christian) |
| 35 | - Reorganize memory decoding check (Christian) | 52 | - Reorganize memory decoding check (Christian) |
| 36 | - Don't use dev_WARN (Ilpo) | 53 | - Don't use dev_WARN (Ilpo) |
| 37 | - Fix build without CONFIG_PCI_IOV (CI) | 54 | - Fix build without CONFIG_PCI_IOV (CI) |
| 38 | 55 | ||
| 39 | v3 -> v4: | 56 | v1 -> v2: |
| 40 | - Change the approach to extending the BAR (Christian) | 57 | - Add pci_iov_resource_extend() and usage in Xe driver |
| 41 | - Tidy the commit messages, use 80 line limit where necessary (Bjorn) | 58 | - Reduce the number of ifdefs (Christian) |
| 42 | - Add kerneldocs to exported functions (Bjorn) | 59 | - Drop patch 2/2 from v1 (Christian) |
| 43 | - Add pci_resource_to_iov() / pci_resource_from_iov() helpers (Ilpo) | 60 | - Add a helper to avoid upsetting static analysis tools (Krzysztof) |
| 44 | - Use FIELD_GET(), tidy whitespace (Ilpo) | ||
| 45 | 61 | ||
| 46 | Michał Winiarski (7): | 62 | Michał Winiarski (6): |
| 47 | PCI/IOV: Restore VF resizable BAR state after reset | 63 | PCI/IOV: Restore VF resizable BAR state after reset |
| 48 | PCI: Add a helper to identify IOV resources | 64 | PCI: Add a helper to convert between VF BAR number and IOV resource |
| 49 | PCI: Add a helper to convert between standard and IOV resources | ||
| 50 | PCI: Allow IOV resources to be resized in pci_resize_resource() | 65 | PCI: Allow IOV resources to be resized in pci_resize_resource() |
| 51 | PCI/IOV: Check that VF BAR fits within the reservation | 66 | PCI/IOV: Check that VF BAR fits within the reservation |
| 52 | PCI: Allow drivers to control VF BAR size | 67 | PCI: Allow drivers to control VF BAR size |
| 53 | drm/xe/pf: Set VF LMEM BAR size | 68 | drm/xe/pf: Set VF LMEM BAR size |
| 54 | 69 | ||
| 55 | drivers/gpu/drm/xe/regs/xe_bars.h | 1 + | 70 | drivers/gpu/drm/xe/regs/xe_bars.h | 1 + |
| 56 | drivers/gpu/drm/xe/xe_pci_sriov.c | 22 +++++ | 71 | drivers/gpu/drm/xe/xe_pci_sriov.c | 22 +++++ |
| 57 | drivers/pci/iov.c | 158 +++++++++++++++++++++++++++--- | 72 | drivers/pci/iov.c | 149 +++++++++++++++++++++++++++--- |
| 58 | drivers/pci/pci.c | 10 +- | 73 | drivers/pci/pci.c | 10 +- |
| 59 | drivers/pci/pci.h | 45 ++++++++- | 74 | drivers/pci/pci.h | 29 ++++++ |
| 60 | drivers/pci/setup-bus.c | 9 +- | 75 | drivers/pci/setup-bus.c | 3 +- |
| 61 | drivers/pci/setup-res.c | 40 ++++++-- | 76 | drivers/pci/setup-res.c | 35 ++++++- |
| 62 | include/linux/pci.h | 6 ++ | 77 | include/linux/pci.h | 6 ++ |
| 63 | include/uapi/linux/pci_regs.h | 1 + | 78 | include/uapi/linux/pci_regs.h | 9 ++ |
| 64 | 9 files changed, 261 insertions(+), 31 deletions(-) | 79 | 9 files changed, 246 insertions(+), 18 deletions(-) |
| 65 | 80 | ||
| 66 | -- | 81 | -- |
| 67 | 2.47.0 | 82 | 2.49.0 |
| 68 | 83 | diff view generated by jsdifflib |
| 1 | Similar to regular resizable BAR, VF BAR can also be resized, e.g. by | 1 | Similar to regular resizable BAR, VF BAR can also be resized, e.g. by |
|---|---|---|---|
| 2 | the system firmware or the PCI subsystem itself. | 2 | the system firmware or the PCI subsystem itself. |
| 3 | 3 | ||
| 4 | The capability layout is the same as PCI_EXT_CAP_ID_REBAR. | ||
| 5 | |||
| 4 | Add the capability ID and restore it as a part of IOV state. | 6 | Add the capability ID and restore it as a part of IOV state. |
| 5 | 7 | ||
| 6 | See PCIe r4.0, sec 9.3.7.4. | 8 | See PCIe r6.2, sec 7.8.7. |
| 7 | 9 | ||
| 8 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 10 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 9 | Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> | 11 | Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> |
| 12 | Reviewed-by: Christian König <christian.koenig@amd.com> | ||
| 10 | --- | 13 | --- |
| 11 | drivers/pci/iov.c | 29 ++++++++++++++++++++++++++++- | 14 | drivers/pci/iov.c | 30 +++++++++++++++++++++++++++++- |
| 12 | include/uapi/linux/pci_regs.h | 1 + | 15 | drivers/pci/pci.h | 1 + |
| 13 | 2 files changed, 29 insertions(+), 1 deletion(-) | 16 | include/uapi/linux/pci_regs.h | 9 +++++++++ |
| 17 | 3 files changed, 39 insertions(+), 1 deletion(-) | ||
| 14 | 18 | ||
| 15 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c | 19 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c |
| 16 | index XXXXXXX..XXXXXXX 100644 | 20 | index XXXXXXX..XXXXXXX 100644 |
| 17 | --- a/drivers/pci/iov.c | 21 | --- a/drivers/pci/iov.c |
| 18 | +++ b/drivers/pci/iov.c | 22 | +++ b/drivers/pci/iov.c |
| ... | ... | ||
| 22 | 26 | ||
| 23 | +#include <linux/bitfield.h> | 27 | +#include <linux/bitfield.h> |
| 24 | #include <linux/pci.h> | 28 | #include <linux/pci.h> |
| 25 | #include <linux/slab.h> | 29 | #include <linux/slab.h> |
| 26 | #include <linux/export.h> | 30 | #include <linux/export.h> |
| 31 | @@ -XXX,XX +XXX,XX @@ static int sriov_init(struct pci_dev *dev, int pos) | ||
| 32 | pci_read_config_byte(dev, pos + PCI_SRIOV_FUNC_LINK, &iov->link); | ||
| 33 | if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) | ||
| 34 | iov->link = PCI_DEVFN(PCI_SLOT(dev->devfn), iov->link); | ||
| 35 | + iov->vf_rebar_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VF_REBAR); | ||
| 36 | |||
| 37 | if (pdev) | ||
| 38 | iov->dev = pci_dev_get(pdev); | ||
| 27 | @@ -XXX,XX +XXX,XX @@ static void sriov_release(struct pci_dev *dev) | 39 | @@ -XXX,XX +XXX,XX @@ static void sriov_release(struct pci_dev *dev) |
| 28 | dev->sriov = NULL; | 40 | dev->sriov = NULL; |
| 29 | } | 41 | } |
| 30 | 42 | ||
| 31 | +static void sriov_restore_vf_rebar_state(struct pci_dev *dev) | 43 | +static void sriov_restore_vf_rebar_state(struct pci_dev *dev) |
| 32 | +{ | 44 | +{ |
| 33 | + unsigned int pos, nbars, i; | 45 | + unsigned int pos, nbars, i; |
| 34 | + u32 ctrl; | 46 | + u32 ctrl; |
| 35 | + | 47 | + |
| 36 | + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VF_REBAR); | 48 | + pos = dev->sriov->vf_rebar_cap; |
| 37 | + if (!pos) | 49 | + if (!pos) |
| 38 | + return; | 50 | + return; |
| 39 | + | 51 | + |
| 40 | + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); | 52 | + pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl); |
| 41 | + nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl); | 53 | + nbars = FIELD_GET(PCI_VF_REBAR_CTRL_NBAR_MASK, ctrl); |
| 42 | + | 54 | + |
| 43 | + for (i = 0; i < nbars; i++, pos += 8) { | 55 | + for (i = 0; i < nbars; i++, pos += 8) { |
| 44 | + int bar_idx, size; | 56 | + int bar_idx, size; |
| 45 | + | 57 | + |
| 46 | + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); | 58 | + pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl); |
| 47 | + bar_idx = FIELD_GET(PCI_REBAR_CTRL_BAR_IDX, ctrl); | 59 | + bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl); |
| 48 | + size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]); | 60 | + size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]); |
| 49 | + ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE; | 61 | + ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE; |
| 50 | + ctrl |= FIELD_PREP(PCI_REBAR_CTRL_BAR_SIZE, size); | 62 | + ctrl |= FIELD_PREP(PCI_VF_REBAR_CTRL_BAR_SIZE, size); |
| 51 | + pci_write_config_dword(dev, pos + PCI_REBAR_CTRL, ctrl); | 63 | + pci_write_config_dword(dev, pos + PCI_VF_REBAR_CTRL, ctrl); |
| 52 | + } | 64 | + } |
| 53 | +} | 65 | +} |
| 54 | + | 66 | + |
| 55 | static void sriov_restore_state(struct pci_dev *dev) | 67 | static void sriov_restore_state(struct pci_dev *dev) |
| 56 | { | 68 | { |
| ... | ... | ||
| 65 | sriov_restore_state(dev); | 77 | sriov_restore_state(dev); |
| 66 | + } | 78 | + } |
| 67 | } | 79 | } |
| 68 | 80 | ||
| 69 | /** | 81 | /** |
| 82 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h | ||
| 83 | index XXXXXXX..XXXXXXX 100644 | ||
| 84 | --- a/drivers/pci/pci.h | ||
| 85 | +++ b/drivers/pci/pci.h | ||
| 86 | @@ -XXX,XX +XXX,XX @@ struct pci_sriov { | ||
| 87 | u16 subsystem_vendor; /* VF subsystem vendor */ | ||
| 88 | u16 subsystem_device; /* VF subsystem device */ | ||
| 89 | resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ | ||
| 90 | + u16 vf_rebar_cap; /* VF Resizable BAR capability offset */ | ||
| 91 | bool drivers_autoprobe; /* Auto probing of VFs by driver */ | ||
| 92 | }; | ||
| 93 | |||
| 70 | diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h | 94 | diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h |
| 71 | index XXXXXXX..XXXXXXX 100644 | 95 | index XXXXXXX..XXXXXXX 100644 |
| 72 | --- a/include/uapi/linux/pci_regs.h | 96 | --- a/include/uapi/linux/pci_regs.h |
| 73 | +++ b/include/uapi/linux/pci_regs.h | 97 | +++ b/include/uapi/linux/pci_regs.h |
| 74 | @@ -XXX,XX +XXX,XX @@ | 98 | @@ -XXX,XX +XXX,XX @@ |
| ... | ... | ||
| 77 | #define PCI_EXT_CAP_ID_DVSEC 0x23 /* Designated Vendor-Specific */ | 101 | #define PCI_EXT_CAP_ID_DVSEC 0x23 /* Designated Vendor-Specific */ |
| 78 | +#define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ | 102 | +#define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ |
| 79 | #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ | 103 | #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ |
| 80 | #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ | 104 | #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ |
| 81 | #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */ | 105 | #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */ |
| 106 | @@ -XXX,XX +XXX,XX @@ | ||
| 107 | #define PCI_DVSEC_HEADER2 0x8 /* Designated Vendor-Specific Header2 */ | ||
| 108 | #define PCI_DVSEC_HEADER2_ID(x) ((x) & 0xffff) | ||
| 109 | |||
| 110 | +/* VF Resizable BARs, same layout as PCI_REBAR */ | ||
| 111 | +#define PCI_VF_REBAR_CAP PCI_REBAR_CAP | ||
| 112 | +#define PCI_VF_REBAR_CAP_SIZES PCI_REBAR_CAP_SIZES | ||
| 113 | +#define PCI_VF_REBAR_CTRL PCI_REBAR_CTRL | ||
| 114 | +#define PCI_VF_REBAR_CTRL_BAR_IDX PCI_REBAR_CTRL_BAR_IDX | ||
| 115 | +#define PCI_VF_REBAR_CTRL_NBAR_MASK PCI_REBAR_CTRL_NBAR_MASK | ||
| 116 | +#define PCI_VF_REBAR_CTRL_BAR_SIZE PCI_REBAR_CTRL_BAR_SIZE | ||
| 117 | + | ||
| 118 | /* Data Link Feature */ | ||
| 119 | #define PCI_DLF_CAP 0x04 /* Capabilities Register */ | ||
| 120 | #define PCI_DLF_EXCHANGE_ENABLE 0x80000000 /* Data Link Feature Exchange Enable */ | ||
| 82 | -- | 121 | -- |
| 83 | 2.47.0 | 122 | 2.49.0 |
| 84 | 123 | diff view generated by jsdifflib |
Deleted patch | |||
|---|---|---|---|
| 1 | There are multiple places where special handling is required for IOV | ||
| 2 | resources. | ||
| 3 | 1 | ||
| 4 | Extract it to pci_resource_is_iov() helper and drop a few ifdefs. | ||
| 5 | |||
| 6 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | ||
| 7 | Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> | ||
| 8 | --- | ||
| 9 | drivers/pci/pci.h | 19 +++++++++++++++---- | ||
| 10 | drivers/pci/setup-bus.c | 7 +++---- | ||
| 11 | drivers/pci/setup-res.c | 4 +--- | ||
| 12 | 3 files changed, 19 insertions(+), 11 deletions(-) | ||
| 13 | |||
| 14 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h | ||
| 15 | index XXXXXXX..XXXXXXX 100644 | ||
| 16 | --- a/drivers/pci/pci.h | ||
| 17 | +++ b/drivers/pci/pci.h | ||
| 18 | @@ -XXX,XX +XXX,XX @@ void pci_iov_update_resource(struct pci_dev *dev, int resno); | ||
| 19 | resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno); | ||
| 20 | void pci_restore_iov_state(struct pci_dev *dev); | ||
| 21 | int pci_iov_bus_range(struct pci_bus *bus); | ||
| 22 | +static inline bool pci_resource_is_iov(int resno) | ||
| 23 | +{ | ||
| 24 | + return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END; | ||
| 25 | +} | ||
| 26 | extern const struct attribute_group sriov_pf_dev_attr_group; | ||
| 27 | extern const struct attribute_group sriov_vf_dev_attr_group; | ||
| 28 | #else | ||
| 29 | @@ -XXX,XX +XXX,XX @@ static inline int pci_iov_init(struct pci_dev *dev) | ||
| 30 | } | ||
| 31 | static inline void pci_iov_release(struct pci_dev *dev) { } | ||
| 32 | static inline void pci_iov_remove(struct pci_dev *dev) { } | ||
| 33 | +static inline void pci_iov_update_resource(struct pci_dev *dev, int resno) { } | ||
| 34 | +static inline resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, | ||
| 35 | + int resno) | ||
| 36 | +{ | ||
| 37 | + return 0; | ||
| 38 | +} | ||
| 39 | static inline void pci_restore_iov_state(struct pci_dev *dev) { } | ||
| 40 | static inline int pci_iov_bus_range(struct pci_bus *bus) | ||
| 41 | { | ||
| 42 | return 0; | ||
| 43 | } | ||
| 44 | - | ||
| 45 | +static inline bool pci_resource_is_iov(int resno) | ||
| 46 | +{ | ||
| 47 | + return false; | ||
| 48 | +} | ||
| 49 | #endif /* CONFIG_PCI_IOV */ | ||
| 50 | |||
| 51 | #ifdef CONFIG_PCIE_PTM | ||
| 52 | @@ -XXX,XX +XXX,XX @@ unsigned long pci_cardbus_resource_alignment(struct resource *); | ||
| 53 | static inline resource_size_t pci_resource_alignment(struct pci_dev *dev, | ||
| 54 | struct resource *res) | ||
| 55 | { | ||
| 56 | -#ifdef CONFIG_PCI_IOV | ||
| 57 | int resno = res - dev->resource; | ||
| 58 | |||
| 59 | - if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END) | ||
| 60 | + if (pci_resource_is_iov(resno)) | ||
| 61 | return pci_sriov_resource_alignment(dev, resno); | ||
| 62 | -#endif | ||
| 63 | if (dev->class >> 8 == PCI_CLASS_BRIDGE_CARDBUS) | ||
| 64 | return pci_cardbus_resource_alignment(res); | ||
| 65 | return resource_alignment(res); | ||
| 66 | diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c | ||
| 67 | index XXXXXXX..XXXXXXX 100644 | ||
| 68 | --- a/drivers/pci/setup-bus.c | ||
| 69 | +++ b/drivers/pci/setup-bus.c | ||
| 70 | @@ -XXX,XX +XXX,XX @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, | ||
| 71 | (r->flags & mask) != type3)) | ||
| 72 | continue; | ||
| 73 | r_size = resource_size(r); | ||
| 74 | -#ifdef CONFIG_PCI_IOV | ||
| 75 | + | ||
| 76 | /* Put SRIOV requested res to the optional list */ | ||
| 77 | - if (realloc_head && i >= PCI_IOV_RESOURCES && | ||
| 78 | - i <= PCI_IOV_RESOURCE_END) { | ||
| 79 | + if (realloc_head && pci_resource_is_iov(i)) { | ||
| 80 | add_align = max(pci_resource_alignment(dev, r), add_align); | ||
| 81 | r->end = r->start - 1; | ||
| 82 | add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */); | ||
| 83 | children_add_size += r_size; | ||
| 84 | continue; | ||
| 85 | } | ||
| 86 | -#endif | ||
| 87 | + | ||
| 88 | /* | ||
| 89 | * aligns[0] is for 1MB (since bridge memory | ||
| 90 | * windows are always at least 1MB aligned), so | ||
| 91 | diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c | ||
| 92 | index XXXXXXX..XXXXXXX 100644 | ||
| 93 | --- a/drivers/pci/setup-res.c | ||
| 94 | +++ b/drivers/pci/setup-res.c | ||
| 95 | @@ -XXX,XX +XXX,XX @@ void pci_update_resource(struct pci_dev *dev, int resno) | ||
| 96 | { | ||
| 97 | if (resno <= PCI_ROM_RESOURCE) | ||
| 98 | pci_std_update_resource(dev, resno); | ||
| 99 | -#ifdef CONFIG_PCI_IOV | ||
| 100 | - else if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END) | ||
| 101 | + else if (pci_resource_is_iov(resno)) | ||
| 102 | pci_iov_update_resource(dev, resno); | ||
| 103 | -#endif | ||
| 104 | } | ||
| 105 | |||
| 106 | int pci_claim_resource(struct pci_dev *dev, int resource) | ||
| 107 | -- | ||
| 108 | 2.47.0 | ||
| 109 | diff view generated by jsdifflib |
| 1 | There are multiple places where conversions between IOV resources and | 1 | There are multiple places where conversions between IOV resources and |
|---|---|---|---|
| 2 | standard resources are done. | 2 | corresponding VF BAR numbers are done. |
| 3 | 3 | ||
| 4 | Extract the logic to pci_resource_to_iov() and pci_resource_from_iov() | 4 | Extract the logic to pci_resource_num_from_vf_bar() and |
| 5 | helpers. | 5 | pci_resource_num_to_vf_bar() helpers. |
| 6 | 6 | ||
| 7 | Suggested-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> | 7 | Suggested-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> |
| 8 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 8 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 9 | Acked-by: Christian König <christian.koenig@amd.com> | ||
| 9 | --- | 10 | --- |
| 10 | drivers/pci/iov.c | 20 ++++++++++---------- | 11 | drivers/pci/iov.c | 26 ++++++++++++++++---------- |
| 11 | drivers/pci/pci.h | 18 ++++++++++++++++++ | 12 | drivers/pci/pci.h | 19 +++++++++++++++++++ |
| 12 | drivers/pci/setup-bus.c | 2 +- | 13 | drivers/pci/setup-bus.c | 3 ++- |
| 13 | 3 files changed, 29 insertions(+), 11 deletions(-) | 14 | 3 files changed, 37 insertions(+), 11 deletions(-) |
| 14 | 15 | ||
| 15 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c | 16 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c |
| 16 | index XXXXXXX..XXXXXXX 100644 | 17 | index XXXXXXX..XXXXXXX 100644 |
| 17 | --- a/drivers/pci/iov.c | 18 | --- a/drivers/pci/iov.c |
| 18 | +++ b/drivers/pci/iov.c | 19 | +++ b/drivers/pci/iov.c |
| 19 | @@ -XXX,XX +XXX,XX @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) | 20 | @@ -XXX,XX +XXX,XX @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) |
| 20 | if (!dev->is_physfn) | 21 | if (!dev->is_physfn) |
| 21 | return 0; | 22 | return 0; |
| 22 | 23 | ||
| 23 | - return dev->sriov->barsz[resno - PCI_IOV_RESOURCES]; | 24 | - return dev->sriov->barsz[resno - PCI_IOV_RESOURCES]; |
| 24 | + return dev->sriov->barsz[pci_resource_from_iov(resno)]; | 25 | + return dev->sriov->barsz[pci_resource_num_to_vf_bar(resno)]; |
| 25 | } | 26 | } |
| 26 | 27 | ||
| 27 | static void pci_read_vf_config_common(struct pci_dev *virtfn) | 28 | static void pci_read_vf_config_common(struct pci_dev *virtfn) |
| 28 | @@ -XXX,XX +XXX,XX @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) | 29 | @@ -XXX,XX +XXX,XX @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) |
| 29 | virtfn->multifunction = 0; | 30 | virtfn->multifunction = 0; |
| 30 | 31 | ||
| 31 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 32 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 32 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; | 33 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; |
| 33 | + res = &dev->resource[pci_resource_to_iov(i)]; | 34 | + int idx = pci_resource_num_from_vf_bar(i); |
| 35 | + | ||
| 36 | + res = &dev->resource[idx]; | ||
| 34 | if (!res->parent) | 37 | if (!res->parent) |
| 35 | continue; | 38 | continue; |
| 36 | virtfn->resource[i].name = pci_name(virtfn); | 39 | virtfn->resource[i].name = pci_name(virtfn); |
| 37 | virtfn->resource[i].flags = res->flags; | 40 | virtfn->resource[i].flags = res->flags; |
| 38 | - size = pci_iov_resource_size(dev, i + PCI_IOV_RESOURCES); | 41 | - size = pci_iov_resource_size(dev, i + PCI_IOV_RESOURCES); |
| 39 | + size = pci_iov_resource_size(dev, pci_resource_to_iov(i)); | 42 | + size = pci_iov_resource_size(dev, idx); |
| 40 | virtfn->resource[i].start = res->start + size * id; | 43 | resource_set_range(&virtfn->resource[i], |
| 41 | virtfn->resource[i].end = virtfn->resource[i].start + size - 1; | 44 | res->start + size * id, size); |
| 42 | rc = request_resource(res, &virtfn->resource[i]); | 45 | rc = request_resource(res, &virtfn->resource[i]); |
| 43 | @@ -XXX,XX +XXX,XX @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) | 46 | @@ -XXX,XX +XXX,XX @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) |
| 44 | 47 | ||
| 45 | nres = 0; | 48 | nres = 0; |
| 46 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 49 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 47 | - bars |= (1 << (i + PCI_IOV_RESOURCES)); | 50 | - bars |= (1 << (i + PCI_IOV_RESOURCES)); |
| 48 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; | 51 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; |
| 49 | + bars |= (1 << pci_resource_to_iov(i)); | 52 | + int idx = pci_resource_num_from_vf_bar(i); |
| 50 | + res = &dev->resource[pci_resource_to_iov(i)]; | 53 | + |
| 54 | + bars |= (1 << idx); | ||
| 55 | + res = &dev->resource[idx]; | ||
| 51 | if (res->parent) | 56 | if (res->parent) |
| 52 | nres++; | 57 | nres++; |
| 53 | } | 58 | } |
| 54 | @@ -XXX,XX +XXX,XX @@ static int sriov_init(struct pci_dev *dev, int pos) | 59 | @@ -XXX,XX +XXX,XX @@ static int sriov_init(struct pci_dev *dev, int pos) |
| 55 | 60 | ||
| 56 | nres = 0; | 61 | nres = 0; |
| 57 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 62 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 58 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; | 63 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; |
| 59 | - res_name = pci_resource_name(dev, i + PCI_IOV_RESOURCES); | 64 | - res_name = pci_resource_name(dev, i + PCI_IOV_RESOURCES); |
| 60 | + res = &dev->resource[pci_resource_to_iov(i)]; | 65 | + int idx = pci_resource_num_from_vf_bar(i); |
| 61 | + res_name = pci_resource_name(dev, pci_resource_to_iov(i)); | 66 | + |
| 67 | + res = &dev->resource[idx]; | ||
| 68 | + res_name = pci_resource_name(dev, idx); | ||
| 62 | 69 | ||
| 63 | /* | 70 | /* |
| 64 | * If it is already FIXED, don't change it, something | 71 | * If it is already FIXED, don't change it, something |
| 65 | @@ -XXX,XX +XXX,XX @@ static int sriov_init(struct pci_dev *dev, int pos) | 72 | @@ -XXX,XX +XXX,XX @@ static int sriov_init(struct pci_dev *dev, int pos) |
| 66 | dev->is_physfn = 0; | 73 | dev->is_physfn = 0; |
| 67 | failed: | 74 | failed: |
| 68 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 75 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 69 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; | 76 | - res = &dev->resource[i + PCI_IOV_RESOURCES]; |
| 70 | + res = &dev->resource[pci_resource_to_iov(i)]; | 77 | + res = &dev->resource[pci_resource_num_from_vf_bar(i)]; |
| 71 | res->flags = 0; | 78 | res->flags = 0; |
| 72 | } | 79 | } |
| 73 | 80 | ||
| 74 | @@ -XXX,XX +XXX,XX @@ static void sriov_restore_state(struct pci_dev *dev) | 81 | @@ -XXX,XX +XXX,XX @@ static void sriov_restore_state(struct pci_dev *dev) |
| 75 | pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, ctrl); | 82 | pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, ctrl); |
| 76 | 83 | ||
| 77 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) | 84 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) |
| 78 | - pci_update_resource(dev, i + PCI_IOV_RESOURCES); | 85 | - pci_update_resource(dev, i + PCI_IOV_RESOURCES); |
| 79 | + pci_update_resource(dev, pci_resource_to_iov(i)); | 86 | + pci_update_resource(dev, pci_resource_num_from_vf_bar(i)); |
| 80 | 87 | ||
| 81 | pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); | 88 | pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); |
| 82 | pci_iov_set_numvfs(dev, iov->num_VFs); | 89 | pci_iov_set_numvfs(dev, iov->num_VFs); |
| 83 | @@ -XXX,XX +XXX,XX @@ void pci_iov_update_resource(struct pci_dev *dev, int resno) | 90 | @@ -XXX,XX +XXX,XX @@ void pci_iov_update_resource(struct pci_dev *dev, int resno) |
| 84 | { | 91 | { |
| 85 | struct pci_sriov *iov = dev->is_physfn ? dev->sriov : NULL; | 92 | struct pci_sriov *iov = dev->is_physfn ? dev->sriov : NULL; |
| 86 | struct resource *res = dev->resource + resno; | 93 | struct resource *res = pci_resource_n(dev, resno); |
| 87 | - int vf_bar = resno - PCI_IOV_RESOURCES; | 94 | - int vf_bar = resno - PCI_IOV_RESOURCES; |
| 88 | + int vf_bar = pci_resource_from_iov(resno); | 95 | + int vf_bar = pci_resource_num_to_vf_bar(resno); |
| 89 | struct pci_bus_region region; | 96 | struct pci_bus_region region; |
| 90 | u16 cmd; | 97 | u16 cmd; |
| 91 | u32 new; | 98 | u32 new; |
| 92 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h | 99 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h |
| 93 | index XXXXXXX..XXXXXXX 100644 | 100 | index XXXXXXX..XXXXXXX 100644 |
| 94 | --- a/drivers/pci/pci.h | 101 | --- a/drivers/pci/pci.h |
| 95 | +++ b/drivers/pci/pci.h | 102 | +++ b/drivers/pci/pci.h |
| 96 | @@ -XXX,XX +XXX,XX @@ static inline bool pci_resource_is_iov(int resno) | 103 | @@ -XXX,XX +XXX,XX @@ static inline bool pci_resource_is_iov(int resno) |
| 97 | { | 104 | { |
| 98 | return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END; | 105 | return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END; |
| 99 | } | 106 | } |
| 100 | +static inline int pci_resource_to_iov(int resno) | 107 | +static inline int pci_resource_num_from_vf_bar(int resno) |
| 101 | +{ | 108 | +{ |
| 102 | + return resno + PCI_IOV_RESOURCES; | 109 | + return resno + PCI_IOV_RESOURCES; |
| 103 | +} | 110 | +} |
| 104 | + | 111 | + |
| 105 | +static inline int pci_resource_from_iov(int resno) | 112 | +static inline int pci_resource_num_to_vf_bar(int resno) |
| 106 | +{ | 113 | +{ |
| 107 | + return resno - PCI_IOV_RESOURCES; | 114 | + return resno - PCI_IOV_RESOURCES; |
| 108 | +} | 115 | +} |
| 109 | extern const struct attribute_group sriov_pf_dev_attr_group; | 116 | extern const struct attribute_group sriov_pf_dev_attr_group; |
| 110 | extern const struct attribute_group sriov_vf_dev_attr_group; | 117 | extern const struct attribute_group sriov_vf_dev_attr_group; |
| 111 | #else | 118 | #else |
| 112 | @@ -XXX,XX +XXX,XX @@ static inline bool pci_resource_is_iov(int resno) | 119 | @@ -XXX,XX +XXX,XX @@ static inline bool pci_resource_is_iov(int resno) |
| 113 | { | 120 | { |
| 114 | return false; | 121 | return false; |
| 115 | } | 122 | } |
| 116 | +static inline int pci_resource_to_iov(int resno) | 123 | +static inline int pci_resource_num_from_vf_bar(int resno) |
| 117 | +{ | 124 | +{ |
| 125 | + WARN_ON_ONCE(1); | ||
| 118 | + return -ENODEV; | 126 | + return -ENODEV; |
| 119 | +} | 127 | +} |
| 120 | + | 128 | +static inline int pci_resource_num_to_vf_bar(int resno) |
| 121 | +static inline int pci_resource_from_iov(int resno) | ||
| 122 | +{ | 129 | +{ |
| 130 | + WARN_ON_ONCE(1); | ||
| 123 | + return -ENODEV; | 131 | + return -ENODEV; |
| 124 | +} | 132 | +} |
| 125 | #endif /* CONFIG_PCI_IOV */ | 133 | #endif /* CONFIG_PCI_IOV */ |
| 126 | 134 | ||
| 127 | #ifdef CONFIG_PCIE_PTM | 135 | #ifdef CONFIG_PCIE_TPH |
| 128 | diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c | 136 | diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c |
| 129 | index XXXXXXX..XXXXXXX 100644 | 137 | index XXXXXXX..XXXXXXX 100644 |
| 130 | --- a/drivers/pci/setup-bus.c | 138 | --- a/drivers/pci/setup-bus.c |
| 131 | +++ b/drivers/pci/setup-bus.c | 139 | +++ b/drivers/pci/setup-bus.c |
| 132 | @@ -XXX,XX +XXX,XX @@ static int iov_resources_unassigned(struct pci_dev *dev, void *data) | 140 | @@ -XXX,XX +XXX,XX @@ static int iov_resources_unassigned(struct pci_dev *dev, void *data) |
| 133 | bool *unassigned = data; | 141 | bool *unassigned = data; |
| 134 | 142 | ||
| 135 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 143 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 136 | - struct resource *r = &dev->resource[i + PCI_IOV_RESOURCES]; | 144 | - struct resource *r = &dev->resource[i + PCI_IOV_RESOURCES]; |
| 137 | + struct resource *r = &dev->resource[pci_resource_to_iov(i)]; | 145 | + int idx = pci_resource_num_from_vf_bar(i); |
| 146 | + struct resource *r = &dev->resource[idx]; | ||
| 138 | struct pci_bus_region region; | 147 | struct pci_bus_region region; |
| 139 | 148 | ||
| 140 | /* Not assigned or rejected by kernel? */ | 149 | /* Not assigned or rejected by kernel? */ |
| 141 | -- | 150 | -- |
| 142 | 2.47.0 | 151 | 2.49.0 |
| 143 | 152 | diff view generated by jsdifflib |
| 1 | Similar to regular resizable BAR, VF BAR can also be resized. | 1 | Similar to regular resizable BAR, VF BAR can also be resized. |
|---|---|---|---|
| 2 | 2 | ||
| 3 | The structures are very similar, which means we can reuse most of the | 3 | The capability layout is the same as PCI_EXT_CAP_ID_REBAR, which means |
| 4 | implementation. | 4 | we can reuse most of the implementation, the only difference being |
| 5 | resource size calculation (which is multiplied by total VFs) and memory | ||
| 6 | decoding (which is controlled by a separate VF MSE field in SR-IOV cap). | ||
| 5 | 7 | ||
| 6 | Extend the pci_resize_resource() function to accept IOV resources. | 8 | Extend the pci_resize_resource() function to accept IOV resources. |
| 7 | 9 | ||
| 8 | See PCIe r4.0, sec 9.3.7.4. | 10 | See PCIe r6.2, sec 7.8.7. |
| 9 | 11 | ||
| 10 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 12 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 11 | --- | 13 | --- |
| 12 | drivers/pci/iov.c | 21 +++++++++++++++++++++ | 14 | drivers/pci/iov.c | 21 +++++++++++++++++++++ |
| 13 | drivers/pci/pci.c | 10 +++++++++- | 15 | drivers/pci/pci.c | 10 +++++++++- |
| 14 | drivers/pci/pci.h | 10 +++++++++- | 16 | drivers/pci/pci.h | 9 +++++++++ |
| 15 | drivers/pci/setup-res.c | 36 +++++++++++++++++++++++++++++++----- | 17 | drivers/pci/setup-res.c | 35 ++++++++++++++++++++++++++++++----- |
| 16 | 4 files changed, 70 insertions(+), 7 deletions(-) | 18 | 4 files changed, 69 insertions(+), 6 deletions(-) |
| 17 | 19 | ||
| 18 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c | 20 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c |
| 19 | index XXXXXXX..XXXXXXX 100644 | 21 | index XXXXXXX..XXXXXXX 100644 |
| 20 | --- a/drivers/pci/iov.c | 22 | --- a/drivers/pci/iov.c |
| 21 | +++ b/drivers/pci/iov.c | 23 | +++ b/drivers/pci/iov.c |
| 22 | @@ -XXX,XX +XXX,XX @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) | 24 | @@ -XXX,XX +XXX,XX @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) |
| 23 | return dev->sriov->barsz[pci_resource_from_iov(resno)]; | 25 | return dev->sriov->barsz[pci_resource_num_to_vf_bar(resno)]; |
| 24 | } | 26 | } |
| 25 | 27 | ||
| 26 | +void pci_iov_resource_set_size(struct pci_dev *dev, int resno, | 28 | +void pci_iov_resource_set_size(struct pci_dev *dev, int resno, |
| 27 | + resource_size_t size) | 29 | + resource_size_t size) |
| 28 | +{ | 30 | +{ |
| 29 | + if (!pci_resource_is_iov(resno)) { | 31 | + if (!pci_resource_is_iov(resno)) { |
| 30 | + pci_warn(dev, "%s is not an IOV resource\n", | 32 | + pci_warn(dev, "%s is not an IOV resource\n", |
| 31 | + pci_resource_name(dev, resno)); | 33 | + pci_resource_name(dev, resno)); |
| 32 | + return; | 34 | + return; |
| 33 | + } | 35 | + } |
| 34 | + | 36 | + |
| 35 | + dev->sriov->barsz[pci_resource_from_iov(resno)] = size; | 37 | + dev->sriov->barsz[pci_resource_num_to_vf_bar(resno)] = size; |
| 36 | +} | 38 | +} |
| 37 | + | 39 | + |
| 38 | +bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev) | 40 | +bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev) |
| 39 | +{ | 41 | +{ |
| 40 | + u16 cmd; | 42 | + u16 cmd; |
| ... | ... | ||
| 50 | diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c | 52 | diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c |
| 51 | index XXXXXXX..XXXXXXX 100644 | 53 | index XXXXXXX..XXXXXXX 100644 |
| 52 | --- a/drivers/pci/pci.c | 54 | --- a/drivers/pci/pci.c |
| 53 | +++ b/drivers/pci/pci.c | 55 | +++ b/drivers/pci/pci.c |
| 54 | @@ -XXX,XX +XXX,XX @@ static int pci_rebar_find_pos(struct pci_dev *pdev, int bar) | 56 | @@ -XXX,XX +XXX,XX @@ static int pci_rebar_find_pos(struct pci_dev *pdev, int bar) |
| 55 | { | ||
| 56 | unsigned int pos, nbars, i; | 57 | unsigned int pos, nbars, i; |
| 57 | u32 ctrl; | 58 | u32 ctrl; |
| 58 | + int cap; | 59 | |
| 59 | 60 | - pos = pdev->rebar_cap; | |
| 60 | - pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR); | ||
| 61 | + if (pci_resource_is_iov(bar)) { | 61 | + if (pci_resource_is_iov(bar)) { |
| 62 | + cap = PCI_EXT_CAP_ID_VF_REBAR; | 62 | + if (!pdev->physfn) |
| 63 | + bar = pci_resource_from_iov(bar); | 63 | + return -ENOTSUPP; |
| 64 | + pos = pdev->sriov->vf_rebar_cap; | ||
| 65 | + bar = pci_resource_num_to_vf_bar(bar); | ||
| 64 | + } else { | 66 | + } else { |
| 65 | + cap = PCI_EXT_CAP_ID_REBAR; | 67 | + pos = pdev->rebar_cap; |
| 66 | + } | 68 | + } |
| 67 | + | 69 | + |
| 68 | + pos = pci_find_ext_capability(pdev, cap); | ||
| 69 | if (!pos) | 70 | if (!pos) |
| 70 | return -ENOTSUPP; | 71 | return -ENOTSUPP; |
| 71 | 72 | ||
| 72 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h | 73 | diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h |
| 73 | index XXXXXXX..XXXXXXX 100644 | 74 | index XXXXXXX..XXXXXXX 100644 |
| ... | ... | ||
| 94 | + return false; | 95 | + return false; |
| 95 | +} | 96 | +} |
| 96 | static inline bool pci_resource_is_iov(int resno) | 97 | static inline bool pci_resource_is_iov(int resno) |
| 97 | { | 98 | { |
| 98 | return false; | 99 | return false; |
| 99 | @@ -XXX,XX +XXX,XX @@ static inline int pci_resource_to_iov(int resno) | ||
| 100 | { | ||
| 101 | return -ENODEV; | ||
| 102 | } | ||
| 103 | - | ||
| 104 | static inline int pci_resource_from_iov(int resno) | ||
| 105 | { | ||
| 106 | return -ENODEV; | ||
| 107 | diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c | 100 | diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c |
| 108 | index XXXXXXX..XXXXXXX 100644 | 101 | index XXXXXXX..XXXXXXX 100644 |
| 109 | --- a/drivers/pci/setup-res.c | 102 | --- a/drivers/pci/setup-res.c |
| 110 | +++ b/drivers/pci/setup-res.c | 103 | +++ b/drivers/pci/setup-res.c |
| 111 | @@ -XXX,XX +XXX,XX @@ void pci_release_resource(struct pci_dev *dev, int resno) | 104 | @@ -XXX,XX +XXX,XX @@ void pci_release_resource(struct pci_dev *dev, int resno) |
| ... | ... | ||
| 127 | + | 120 | + |
| 128 | +static void pci_resize_resource_set_size(struct pci_dev *dev, int resno, | 121 | +static void pci_resize_resource_set_size(struct pci_dev *dev, int resno, |
| 129 | + int size) | 122 | + int size) |
| 130 | +{ | 123 | +{ |
| 131 | + resource_size_t res_size = pci_rebar_size_to_bytes(size); | 124 | + resource_size_t res_size = pci_rebar_size_to_bytes(size); |
| 132 | + struct resource *res = dev->resource + resno; | 125 | + struct resource *res = pci_resource_n(dev, resno); |
| 133 | + | 126 | + |
| 134 | + if (!pci_resource_is_iov(resno)) { | 127 | + if (!pci_resource_is_iov(resno)) { |
| 135 | + res->end = res->start + res_size - 1; | 128 | + resource_set_size(res, res_size); |
| 136 | + } else { | 129 | + } else { |
| 137 | + res->end = res->start + | 130 | + resource_set_size(res, res_size * pci_sriov_get_totalvfs(dev)); |
| 138 | + res_size * pci_sriov_get_totalvfs(dev) - 1; | ||
| 139 | + pci_iov_resource_set_size(dev, resno, res_size); | 131 | + pci_iov_resource_set_size(dev, resno, res_size); |
| 140 | + } | 132 | + } |
| 141 | +} | 133 | +} |
| 142 | + | 134 | + |
| 143 | int pci_resize_resource(struct pci_dev *dev, int resno, int size) | 135 | int pci_resize_resource(struct pci_dev *dev, int resno, int size) |
| 144 | { | 136 | { |
| 145 | struct resource *res = dev->resource + resno; | 137 | struct resource *res = pci_resource_n(dev, resno); |
| 146 | struct pci_host_bridge *host; | 138 | struct pci_host_bridge *host; |
| 147 | int old, ret; | 139 | int old, ret; |
| 148 | u32 sizes; | 140 | u32 sizes; |
| 149 | - u16 cmd; | 141 | - u16 cmd; |
| 150 | 142 | ||
| ... | ... | ||
| 162 | sizes = pci_rebar_get_possible_sizes(dev, resno); | 154 | sizes = pci_rebar_get_possible_sizes(dev, resno); |
| 163 | @@ -XXX,XX +XXX,XX @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) | 155 | @@ -XXX,XX +XXX,XX @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) |
| 164 | if (ret) | 156 | if (ret) |
| 165 | return ret; | 157 | return ret; |
| 166 | 158 | ||
| 167 | - res->end = res->start + pci_rebar_size_to_bytes(size) - 1; | 159 | - resource_set_size(res, pci_rebar_size_to_bytes(size)); |
| 168 | + pci_resize_resource_set_size(dev, resno, size); | 160 | + pci_resize_resource_set_size(dev, resno, size); |
| 169 | 161 | ||
| 170 | /* Check if the new config works by trying to assign everything. */ | 162 | /* Check if the new config works by trying to assign everything. */ |
| 171 | if (dev->bus->self) { | 163 | if (dev->bus->self) { |
| 172 | @@ -XXX,XX +XXX,XX @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) | 164 | @@ -XXX,XX +XXX,XX @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) |
| 173 | 165 | ||
| 174 | error_resize: | 166 | error_resize: |
| 175 | pci_rebar_set_size(dev, resno, old); | 167 | pci_rebar_set_size(dev, resno, old); |
| 176 | - res->end = res->start + pci_rebar_size_to_bytes(old) - 1; | 168 | - resource_set_size(res, pci_rebar_size_to_bytes(old)); |
| 177 | + pci_resize_resource_set_size(dev, resno, old); | 169 | + pci_resize_resource_set_size(dev, resno, old); |
| 178 | return ret; | 170 | return ret; |
| 179 | } | 171 | } |
| 180 | EXPORT_SYMBOL(pci_resize_resource); | 172 | EXPORT_SYMBOL(pci_resize_resource); |
| 181 | -- | 173 | -- |
| 182 | 2.47.0 | 174 | 2.49.0 |
| 183 | 175 | diff view generated by jsdifflib |
| 1 | VF MMIO resource reservation, either created by system firmware and | 1 | When the resource representing VF MMIO BAR reservation is created, its |
|---|---|---|---|
| 2 | inherited by Linux PCI subsystem or created by the subsystem itself, | 2 | size is always large enough to accommodate the BAR of all SR-IOV Virtual |
| 3 | should contain enough space to fit the BAR of all SR-IOV Virtual | 3 | Functions that can potentially be created (total VFs). If for whatever |
| 4 | Functions that can potentially be created (total VFs supported by the | 4 | reason it's not possible to accommodate all VFs - the resource is not |
| 5 | device). | 5 | assigned and no VFs can be created. |
| 6 | 6 | ||
| 7 | However, that assumption only holds in an environment where VF BAR size | 7 | An upcoming change will allow VF BAR size to be modified by drivers at |
| 8 | can't be modified. | 8 | a later point in time, which means that the check for resource |
| 9 | assignment is no longer sufficient. | ||
| 9 | 10 | ||
| 10 | Add an additional check that verifies that VF BAR for all enabled VFs | 11 | Add an additional check that verifies that VF BAR for all enabled VFs |
| 11 | fits within the underlying reservation resource. | 12 | fits within the underlying reservation resource. |
| 12 | 13 | ||
| 13 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 14 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 14 | --- | 15 | --- |
| 15 | drivers/pci/iov.c | 8 ++++++-- | 16 | drivers/pci/iov.c | 3 +++ |
| 16 | 1 file changed, 6 insertions(+), 2 deletions(-) | 17 | 1 file changed, 3 insertions(+) |
| 17 | 18 | ||
| 18 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c | 19 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c |
| 19 | index XXXXXXX..XXXXXXX 100644 | 20 | index XXXXXXX..XXXXXXX 100644 |
| 20 | --- a/drivers/pci/iov.c | 21 | --- a/drivers/pci/iov.c |
| 21 | +++ b/drivers/pci/iov.c | 22 | +++ b/drivers/pci/iov.c |
| 22 | @@ -XXX,XX +XXX,XX @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) | 23 | @@ -XXX,XX +XXX,XX @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) |
| 23 | |||
| 24 | nres = 0; | 24 | nres = 0; |
| 25 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { | 25 | for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { |
| 26 | + int vf_bar_sz = pci_iov_resource_size(dev, | 26 | int idx = pci_resource_num_from_vf_bar(i); |
| 27 | + pci_resource_to_iov(i)); | 27 | + resource_size_t vf_bar_sz = pci_iov_resource_size(dev, idx); |
| 28 | bars |= (1 << pci_resource_to_iov(i)); | 28 | |
| 29 | res = &dev->resource[pci_resource_to_iov(i)]; | 29 | bars |= (1 << idx); |
| 30 | - if (res->parent) | 30 | res = &dev->resource[idx]; |
| 31 | - nres++; | 31 | + if (vf_bar_sz * nr_virtfn > resource_size(res)) |
| 32 | + if (!res->parent || vf_bar_sz * nr_virtfn > resource_size(res)) | ||
| 33 | + continue; | 32 | + continue; |
| 34 | + | 33 | if (res->parent) |
| 35 | + nres++; | 34 | nres++; |
| 36 | } | 35 | } |
| 37 | if (nres != iov->nres) { | ||
| 38 | pci_err(dev, "not enough MMIO resources for SR-IOV\n"); | ||
| 39 | -- | 36 | -- |
| 40 | 2.47.0 | 37 | 2.49.0 |
| 41 | 38 | diff view generated by jsdifflib |
| ... | ... | ||
|---|---|---|---|
| 7 | allow up to num_vfs to be successfully enabled with the current | 7 | allow up to num_vfs to be successfully enabled with the current |
| 8 | underlying reservation size. | 8 | underlying reservation size. |
| 9 | 9 | ||
| 10 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 10 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 11 | --- | 11 | --- |
| 12 | drivers/pci/iov.c | 80 +++++++++++++++++++++++++++++++++++++++++++++ | 12 | drivers/pci/iov.c | 69 +++++++++++++++++++++++++++++++++++++++++++++ |
| 13 | include/linux/pci.h | 6 ++++ | 13 | include/linux/pci.h | 6 ++++ |
| 14 | 2 files changed, 86 insertions(+) | 14 | 2 files changed, 75 insertions(+) |
| 15 | 15 | ||
| 16 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c | 16 | diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c |
| 17 | index XXXXXXX..XXXXXXX 100644 | 17 | index XXXXXXX..XXXXXXX 100644 |
| 18 | --- a/drivers/pci/iov.c | 18 | --- a/drivers/pci/iov.c |
| 19 | +++ b/drivers/pci/iov.c | 19 | +++ b/drivers/pci/iov.c |
| ... | ... | ||
| 24 | + | 24 | + |
| 25 | +/** | 25 | +/** |
| 26 | + * pci_iov_vf_bar_set_size - set a new size for a VF BAR | 26 | + * pci_iov_vf_bar_set_size - set a new size for a VF BAR |
| 27 | + * @dev: the PCI device | 27 | + * @dev: the PCI device |
| 28 | + * @resno: the resource number | 28 | + * @resno: the resource number |
| 29 | + * @size: new size as defined in the spec (0=1MB, 19=512GB) | 29 | + * @size: new size as defined in the spec (0=1MB, 31=128TB) |
| 30 | + * | 30 | + * |
| 31 | + * Set the new size of a VF BAR that supports VF resizable BAR capability. | 31 | + * Set the new size of a VF BAR that supports VF resizable BAR capability. |
| 32 | + * Unlike pci_resize_resource(), this does not cause the resource that | 32 | + * Unlike pci_resize_resource(), this does not cause the resource that |
| 33 | + * reserves the MMIO space (originally up to total_VFs) to be resized, which | 33 | + * reserves the MMIO space (originally up to total_VFs) to be resized, which |
| 34 | + * means that following calls to pci_enable_sriov() can fail if the resources | 34 | + * means that following calls to pci_enable_sriov() can fail if the resources |
| 35 | + * no longer fit. | 35 | + * no longer fit. |
| 36 | + * | 36 | + * |
| 37 | + * Returns 0 on success, or negative on failure. | 37 | + * Return: 0 on success, or negative on failure. |
| 38 | + * | ||
| 39 | + */ | 38 | + */ |
| 40 | +int pci_iov_vf_bar_set_size(struct pci_dev *dev, int resno, int size) | 39 | +int pci_iov_vf_bar_set_size(struct pci_dev *dev, int resno, int size) |
| 41 | +{ | 40 | +{ |
| 42 | + int ret; | 41 | + int ret; |
| 43 | + u32 sizes; | 42 | + u32 sizes; |
| ... | ... | ||
| 64 | + return 0; | 63 | + return 0; |
| 65 | +} | 64 | +} |
| 66 | +EXPORT_SYMBOL_GPL(pci_iov_vf_bar_set_size); | 65 | +EXPORT_SYMBOL_GPL(pci_iov_vf_bar_set_size); |
| 67 | + | 66 | + |
| 68 | +/** | 67 | +/** |
| 69 | + * pci_iov_vf_bar_get_sizes - get VF BAR sizes that allow to create up to num_vfs | 68 | + * pci_iov_vf_bar_get_sizes - get VF BAR sizes allowing to create up to num_vfs |
| 70 | + * @dev: the PCI device | 69 | + * @dev: the PCI device |
| 71 | + * @resno: the resource number | 70 | + * @resno: the resource number |
| 72 | + * @num_vfs: number of VFs | 71 | + * @num_vfs: number of VFs |
| 73 | + * | 72 | + * |
| 74 | + * Get the sizes of a VF resizable BAR that can fit up to num_vfs within the | 73 | + * Get the sizes of a VF resizable BAR that can accommodate @num_vfs within |
| 75 | + * resource that reserves the MMIO space (originally up to total_VFs) the as | 74 | + * the currently assigned size of the resource @resno. |
| 76 | + * bitmask defined in the spec (bit 0=1MB, bit 19=512GB). | ||
| 77 | + * | 75 | + * |
| 78 | + * Returns 0 if BAR isn't resizable. | 76 | + * Return: A bitmask of sizes in format defined in the spec (bit 0=1MB, |
| 79 | + * | 77 | + * bit 31=128TB). |
| 80 | + */ | 78 | + */ |
| 81 | +u32 pci_iov_vf_bar_get_sizes(struct pci_dev *dev, int resno, int num_vfs) | 79 | +u32 pci_iov_vf_bar_get_sizes(struct pci_dev *dev, int resno, int num_vfs) |
| 82 | +{ | 80 | +{ |
| 83 | + resource_size_t size; | 81 | + resource_size_t vf_len = pci_resource_len(dev, resno); |
| 84 | + u32 sizes; | 82 | + u32 sizes; |
| 85 | + int i; | ||
| 86 | + | 83 | + |
| 87 | + sizes = pci_rebar_get_possible_sizes(dev, resno); | 84 | + if (!num_vfs) |
| 88 | + if (!sizes) | ||
| 89 | + return 0; | 85 | + return 0; |
| 90 | + | 86 | + |
| 91 | + while (sizes > 0) { | 87 | + do_div(vf_len, num_vfs); |
| 92 | + i = __fls(sizes); | 88 | + sizes = (roundup_pow_of_two(vf_len + 1) - 1) >> ilog2(SZ_1M); |
| 93 | + size = pci_rebar_size_to_bytes(i); | ||
| 94 | + | 89 | + |
| 95 | + if (size * num_vfs <= pci_resource_len(dev, resno)) | 90 | + return sizes & pci_rebar_get_possible_sizes(dev, resno); |
| 96 | + break; | ||
| 97 | + | ||
| 98 | + sizes &= ~BIT(i); | ||
| 99 | + } | ||
| 100 | + | ||
| 101 | + return sizes; | ||
| 102 | +} | 91 | +} |
| 103 | +EXPORT_SYMBOL_GPL(pci_iov_vf_bar_get_sizes); | 92 | +EXPORT_SYMBOL_GPL(pci_iov_vf_bar_get_sizes); |
| 104 | diff --git a/include/linux/pci.h b/include/linux/pci.h | 93 | diff --git a/include/linux/pci.h b/include/linux/pci.h |
| 105 | index XXXXXXX..XXXXXXX 100644 | 94 | index XXXXXXX..XXXXXXX 100644 |
| 106 | --- a/include/linux/pci.h | 95 | --- a/include/linux/pci.h |
| ... | ... | ||
| 124 | +{ return 0; } | 113 | +{ return 0; } |
| 125 | static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { } | 114 | static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { } |
| 126 | #endif | 115 | #endif |
| 127 | 116 | ||
| 128 | -- | 117 | -- |
| 129 | 2.47.0 | 118 | 2.49.0 |
| 130 | 119 | diff view generated by jsdifflib |
| 1 | LMEM is partitioned between multiple VFs and we expect that the more | 1 | LMEM is partitioned between multiple VFs and we expect that the more |
|---|---|---|---|
| 2 | VFs we have, the less LMEM is assigned to each VF. | 2 | VFs we have, the less LMEM is assigned to each VF. |
| 3 | This means that we can achieve full LMEM BAR access without the need to | 3 | This means that we can achieve full LMEM BAR access without the need to |
| 4 | attempt full VF LMEM BAR resize via pci_resize_resource(). | 4 | attempt full VF LMEM BAR resize via pci_resize_resource(). |
| 5 | 5 | ||
| 6 | Always set the largest possible BAR size that allows to fit the number | 6 | Always try to set the largest possible BAR size that allows to fit the |
| 7 | of enabled VFs. | 7 | number of enabled VFs and inform the user in case the resize attempt is |
| 8 | not successful. | ||
| 8 | 9 | ||
| 9 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> | 10 | Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> |
| 11 | Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> | ||
| 10 | --- | 12 | --- |
| 11 | drivers/gpu/drm/xe/regs/xe_bars.h | 1 + | 13 | drivers/gpu/drm/xe/regs/xe_bars.h | 1 + |
| 12 | drivers/gpu/drm/xe/xe_pci_sriov.c | 22 ++++++++++++++++++++++ | 14 | drivers/gpu/drm/xe/xe_pci_sriov.c | 22 ++++++++++++++++++++++ |
| 13 | 2 files changed, 23 insertions(+) | 15 | 2 files changed, 23 insertions(+) |
| 14 | 16 | ||
| ... | ... | ||
| 36 | + | 38 | + |
| 37 | +#include "regs/xe_bars.h" | 39 | +#include "regs/xe_bars.h" |
| 38 | #include "xe_assert.h" | 40 | #include "xe_assert.h" |
| 39 | #include "xe_device.h" | 41 | #include "xe_device.h" |
| 40 | #include "xe_gt_sriov_pf_config.h" | 42 | #include "xe_gt_sriov_pf_config.h" |
| 41 | @@ -XXX,XX +XXX,XX @@ static void pf_reset_vfs(struct xe_device *xe, unsigned int num_vfs) | 43 | @@ -XXX,XX +XXX,XX @@ static void pf_link_vfs(struct xe_device *xe, int num_vfs) |
| 42 | xe_gt_sriov_pf_control_trigger_flr(gt, n); | 44 | } |
| 43 | } | 45 | } |
| 44 | 46 | ||
| 45 | +static int resize_vf_vram_bar(struct xe_device *xe, int num_vfs) | 47 | +static int resize_vf_vram_bar(struct xe_device *xe, int num_vfs) |
| 46 | +{ | 48 | +{ |
| 47 | + struct pci_dev *pdev = to_pci_dev(xe->drm.dev); | 49 | + struct pci_dev *pdev = to_pci_dev(xe->drm.dev); |
| ... | ... | ||
| 69 | + | 71 | + |
| 70 | err = pci_enable_sriov(pdev, num_vfs); | 72 | err = pci_enable_sriov(pdev, num_vfs); |
| 71 | if (err < 0) | 73 | if (err < 0) |
| 72 | goto failed; | 74 | goto failed; |
| 73 | -- | 75 | -- |
| 74 | 2.47.0 | 76 | 2.49.0 |
| 75 | 77 | diff view generated by jsdifflib |