> -----Original Message----- > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > Sent: Wednesday, February 15, 2023 5:43 PM > > Hello Wentong, > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote: > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a > > companion chip designed to provide secure and low power vision > > capability to IA platforms. IVSC is available in existing commercial > > platforms from multiple OEMs. > > > > The primary use case of IVSC is to bring in context awareness. IVSC > > interfaces directly with the platform main camera sensor via a CSI-2 > > link and processes the image data with the embedded AI engine. The > > detected events are sent over I2C to ISH (Intel Sensor Hub) for > > additional data fusion from multiple sensors. The fusion results are > > used to implement advanced use cases like: > > - Face detection to unlock screen > > - Detect user presence to manage backlight setting or waking up > > system > > Do you have plan to support these features in the ivsc driver in the future ? Not sure, but the first step is to upstream this driver. > > > Since the Image Processing Unit(IPU) used on the host processor needs > > to configure the CSI-2 link in normal camera usages, the CSI-2 link > > and camera sensor can only be used in mutually-exclusive ways by host > > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera > > sensor. The IPU driver can take ownership of the CSI-2 link and camera > > sensor using interfaces provided by this IVSC driver. > > > > Switching ownership requires an interface with two different hardware > > modules inside IVSC. The software interface to these modules is via > > Intel MEI (The Intel Management Engine) commands. These two hardware > > modules have two different MEI UUIDs to enumerate. These hardware > modules are: > > - ACE (Algorithm Context Engine): This module is for algorithm > > computing when IVSC owns camera sensor. Also ACE module controls > > camera sensor's ownership. This hardware module is used to set ownership of > camera sensor. > > - CSI (Camera Serial Interface): This module is used to route camera > > sensor data either to IVSC or to host for IPU driver and application. > > > > IVSC also provides a privacy mode. When privacy mode is turned on, > > camera sensor can't be used. This means that both ACE and host IPU > > can't get image data. And when this mode is turned on, host IPU driver > > is informed via a registered callback, so that user can be notified. > > How does the privacy mode work, and how can the user trust that the closed- > source IVSC and IME firmwares will honour the privacy settings ? No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side. > > > In summary, to acquire ownership of camera by IPU driver, first ACE > > module needs to be informed of ownership and then to setup MIPI CSI-2 > > link for the camera sensor and IPU. > > > > Implementation: > > There are two different drivers to handle ACE and CSI hardware modules > > inside IVSC. > > - mei_csi: MEI client driver to send commands and receive > > notifications from CSI module. > > - mei_ace: MEI client driver to send commands and get status from ACE > > module. > > Interface is exposed via ivsc.h to acquire and release camera sensor > > and > > CSI-2 link. > > Do I understand correctly, from your diagram below, that the communication > between the IME and IVSC goes through SPI ? > > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. > > ---------------------------------------------------------------------- > > ------- > > | Host Processor | > > | | > > | ----------------- ----------------- --------------- | > > | | | | | | | I2C | > > | | IPU | | ISH | |camera driver|--| | > > | | | | | | | | | > > | ----------------- ----------------- --------------- | | > > | | | | | | > > | | | --------------- | | > > | | | | | | | > > | | | | IVSC driver | | | > > | | | | | | | > > | | | --------------- | | > > | | | | | | > > ----------------|-----------------------|----------------------|---------|--- > > | CSI | I2C |SPI | > > | | | | > > ----------------|-----------------------|---------------- | | > > | IVSC | | | | > > | | | | | > > | ----------------- ----------------- | | | > > | | | | | | | | > > | | CSI | | ACE | |------| | > > | | | | | | | > > | ----------------- ----------------- | | > > | | | I2C | | > > ----------------|-----------------------|---------------- | > > | CSI | | > > | | | > > -------------------------------- | > > | | I2C | > > | camera sensor |-----------------------------| > > | | > > -------------------------------- > > > > Wentong Wu (3): > > media: pci: intel: ivsc: Add CSI submodule > > media: pci: intel: ivsc: Add ACE submodule > > media: pci: intel: ivsc: Add acquire/release API for ivsc > > > > drivers/media/pci/Kconfig | 1 + > > drivers/media/pci/intel/Makefile | 2 + > > drivers/media/pci/intel/ivsc/Kconfig | 12 + > > drivers/media/pci/intel/ivsc/Makefile | 7 + > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ > > drivers/media/pci/intel/ivsc/mei_ace.c | 472 > > +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | > > 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ > > drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ > > include/linux/ivsc.h | 74 ++++ > > 10 files changed, 1090 insertions(+) > > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h > > create mode 100644 include/linux/ivsc.h > > -- > Regards, > > Laurent Pinchart
Hello Wentong, On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote: > On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote: > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote: > > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a > > > companion chip designed to provide secure and low power vision > > > capability to IA platforms. IVSC is available in existing commercial > > > platforms from multiple OEMs. > > > > > > The primary use case of IVSC is to bring in context awareness. IVSC > > > interfaces directly with the platform main camera sensor via a CSI-2 > > > link and processes the image data with the embedded AI engine. The > > > detected events are sent over I2C to ISH (Intel Sensor Hub) for > > > additional data fusion from multiple sensors. The fusion results are > > > used to implement advanced use cases like: > > > - Face detection to unlock screen > > > - Detect user presence to manage backlight setting or waking up > > > system > > > > Do you have plan to support these features in the ivsc driver in the future ? > > Not sure, but the first step is to upstream this driver. Sure, no problem. > > > Since the Image Processing Unit(IPU) used on the host processor needs > > > to configure the CSI-2 link in normal camera usages, the CSI-2 link > > > and camera sensor can only be used in mutually-exclusive ways by host > > > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera > > > sensor. The IPU driver can take ownership of the CSI-2 link and camera > > > sensor using interfaces provided by this IVSC driver. > > > > > > Switching ownership requires an interface with two different hardware > > > modules inside IVSC. The software interface to these modules is via > > > Intel MEI (The Intel Management Engine) commands. These two hardware > > > modules have two different MEI UUIDs to enumerate. These hardware > > > modules are: > > > - ACE (Algorithm Context Engine): This module is for algorithm > > > computing when IVSC owns camera sensor. Also ACE module controls > > > camera sensor's ownership. This hardware module is used to set ownership of > > > camera sensor. > > > - CSI (Camera Serial Interface): This module is used to route camera > > > sensor data either to IVSC or to host for IPU driver and application. > > > > > > IVSC also provides a privacy mode. When privacy mode is turned on, > > > camera sensor can't be used. This means that both ACE and host IPU > > > can't get image data. And when this mode is turned on, host IPU driver > > > is informed via a registered callback, so that user can be notified. > > > > How does the privacy mode work, and how can the user trust that the closed- > > source IVSC and IME firmwares will honour the privacy settings ? > > No camera data will be allowed to go through IVSC, and then there will > be no data on IVSC CSI transmitter side. But how can I be sure that the IVSC will not use the camera behind my back, if it's all controlled through a closed-source firmware ? > > > In summary, to acquire ownership of camera by IPU driver, first ACE > > > module needs to be informed of ownership and then to setup MIPI CSI-2 > > > link for the camera sensor and IPU. > > > > > > Implementation: > > > There are two different drivers to handle ACE and CSI hardware modules > > > inside IVSC. > > > - mei_csi: MEI client driver to send commands and receive notifications from CSI module. > > > - mei_ace: MEI client driver to send commands and get status from ACE module. > > > Interface is exposed via ivsc.h to acquire and release camera sensor and > > > CSI-2 link. > > > > Do I understand correctly, from your diagram below, that the communication > > between the IME and IVSC goes through SPI ? > > > > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. > > > ----------------------------------------------------------------------------- > > > | Host Processor | > > > | | > > > | ----------------- ----------------- --------------- | > > > | | | | | | | I2C | > > > | | IPU | | ISH | |camera driver|--| | > > > | | | | | | | | | > > > | ----------------- ----------------- --------------- | | > > > | | | | | | > > > | | | --------------- | | > > > | | | | | | | > > > | | | | IVSC driver | | | > > > | | | | | | | > > > | | | --------------- | | > > > | | | | | | > > > ----------------|-----------------------|----------------------|---------|--- > > > | CSI | I2C |SPI | > > > | | | | > > > ----------------|-----------------------|---------------- | | > > > | IVSC | | | | > > > | | | | | > > > | ----------------- ----------------- | | | > > > | | | | | | | | > > > | | CSI | | ACE | |------| | > > > | | | | | | | > > > | ----------------- ----------------- | | > > > | | | I2C | | > > > ----------------|-----------------------|---------------- | > > > | CSI | | > > > | | | > > > -------------------------------- | > > > | | I2C | > > > | camera sensor |-----------------------------| > > > | | > > > -------------------------------- > > > > > > Wentong Wu (3): > > > media: pci: intel: ivsc: Add CSI submodule > > > media: pci: intel: ivsc: Add ACE submodule > > > media: pci: intel: ivsc: Add acquire/release API for ivsc > > > > > > drivers/media/pci/Kconfig | 1 + > > > drivers/media/pci/intel/Makefile | 2 + > > > drivers/media/pci/intel/ivsc/Kconfig | 12 + > > > drivers/media/pci/intel/ivsc/Makefile | 7 + > > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ > > > drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ > > > drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++ > > > drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ > > > drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ > > > include/linux/ivsc.h | 74 ++++ > > > 10 files changed, 1090 insertions(+) > > > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig > > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile > > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h > > > create mode 100644 include/linux/ivsc.h -- Regards, Laurent Pinchart
> -----Original Message----- > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > Sent: Friday, February 17, 2023 7:12 PM > > Hello Wentong, > > On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote: > > On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote: > > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote: > > > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", > > > > is a companion chip designed to provide secure and low power > > > > vision capability to IA platforms. IVSC is available in existing > > > > commercial platforms from multiple OEMs. > > > > > > > > The primary use case of IVSC is to bring in context awareness. > > > > IVSC interfaces directly with the platform main camera sensor via > > > > a CSI-2 link and processes the image data with the embedded AI > > > > engine. The detected events are sent over I2C to ISH (Intel Sensor > > > > Hub) for additional data fusion from multiple sensors. The fusion > > > > results are used to implement advanced use cases like: > > > > - Face detection to unlock screen > > > > - Detect user presence to manage backlight setting or waking up > > > > system > > > > > > Do you have plan to support these features in the ivsc driver in the future ? > > > > Not sure, but the first step is to upstream this driver. > > Sure, no problem. > > > > > Since the Image Processing Unit(IPU) used on the host processor > > > > needs to configure the CSI-2 link in normal camera usages, the > > > > CSI-2 link and camera sensor can only be used in > > > > mutually-exclusive ways by host IPU and IVSC. By default the IVSC > > > > owns the CSI-2 link and camera sensor. The IPU driver can take > > > > ownership of the CSI-2 link and camera sensor using interfaces provided by > this IVSC driver. > > > > > > > > Switching ownership requires an interface with two different > > > > hardware modules inside IVSC. The software interface to these > > > > modules is via Intel MEI (The Intel Management Engine) commands. > > > > These two hardware modules have two different MEI UUIDs to > > > > enumerate. These hardware modules are: > > > > - ACE (Algorithm Context Engine): This module is for algorithm > > > > computing when IVSC owns camera sensor. Also ACE module controls > > > > camera sensor's ownership. This hardware module is used to set > > > > ownership of camera sensor. > > > > - CSI (Camera Serial Interface): This module is used to route > > > > camera sensor data either to IVSC or to host for IPU driver and application. > > > > > > > > IVSC also provides a privacy mode. When privacy mode is turned on, > > > > camera sensor can't be used. This means that both ACE and host IPU > > > > can't get image data. And when this mode is turned on, host IPU > > > > driver is informed via a registered callback, so that user can be notified. > > > > > > How does the privacy mode work, and how can the user trust that the > > > closed- source IVSC and IME firmwares will honour the privacy settings ? > > > > No camera data will be allowed to go through IVSC, and then there will > > be no data on IVSC CSI transmitter side. > > But how can I be sure that the IVSC will not use the camera behind my back, if > it's all controlled through a closed-source firmware ? Actually I don't know how to answer your question, but this is guaranteed though we have no plan to open-source the firmware. > > > > > In summary, to acquire ownership of camera by IPU driver, first > > > > ACE module needs to be informed of ownership and then to setup > > > > MIPI CSI-2 link for the camera sensor and IPU. > > > > > > > > Implementation: > > > > There are two different drivers to handle ACE and CSI hardware > > > > modules inside IVSC. > > > > - mei_csi: MEI client driver to send commands and receive notifications > from CSI module. > > > > - mei_ace: MEI client driver to send commands and get status from ACE > module. > > > > Interface is exposed via ivsc.h to acquire and release camera > > > > sensor and > > > > CSI-2 link. > > > > > > Do I understand correctly, from your diagram below, that the > > > communication between the IME and IVSC goes through SPI ? > > > > > > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. > > > > ------------------------------------------------------------------ > > > > ----------- > > > > | Host Processor | > > > > | | > > > > | ----------------- ----------------- --------------- | > > > > | | | | | | | I2C | > > > > | | IPU | | ISH | |camera driver|--| | > > > > | | | | | | | | | > > > > | ----------------- ----------------- --------------- | | > > > > | | | | | | > > > > | | | --------------- | | > > > > | | | | | | | > > > > | | | | IVSC driver | | | > > > > | | | | | | | > > > > | | | --------------- | | > > > > | | | | | | > > > > ----------------|-----------------------|----------------------|---------|--- > > > > | CSI | I2C |SPI | > > > > | | | | > > > > ----------------|-----------------------|---------------- | | > > > > | IVSC | | | | > > > > | | | | | > > > > | ----------------- ----------------- | | | > > > > | | | | | | | | > > > > | | CSI | | ACE | |------| | > > > > | | | | | | | > > > > | ----------------- ----------------- | | > > > > | | | I2C | | > > > > ----------------|-----------------------|---------------- | > > > > | CSI | | > > > > | | | > > > > -------------------------------- | > > > > | | I2C | > > > > | camera sensor |-----------------------------| > > > > | | > > > > -------------------------------- > > > > > > > > Wentong Wu (3): > > > > media: pci: intel: ivsc: Add CSI submodule > > > > media: pci: intel: ivsc: Add ACE submodule > > > > media: pci: intel: ivsc: Add acquire/release API for ivsc > > > > > > > > drivers/media/pci/Kconfig | 1 + > > > > drivers/media/pci/intel/Makefile | 2 + > > > > drivers/media/pci/intel/ivsc/Kconfig | 12 + > > > > drivers/media/pci/intel/ivsc/Makefile | 7 + > > > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ > > > > drivers/media/pci/intel/ivsc/mei_ace.c | 472 > > > > +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h > > > > |> 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 > > > > ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ > > > > include/linux/ivsc.h | 74 ++++ > > > > 10 files changed, 1090 insertions(+) create mode 100644 > > > > drivers/media/pci/intel/ivsc/Kconfig > > > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile > > > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c > > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c > > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h > > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c > > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h > > > > create mode 100644 include/linux/ivsc.h > > -- > Regards, > > Laurent Pinchart
© 2016 - 2025 Red Hat, Inc.