> -----Original Message-----
> From: Bingbu Cao <bingbu.cao@linux.intel.com>
> Sent: Friday, February 17, 2023 9:44 AM
> 
> Hi, Sakari,
> 
> On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > Hi Bingbu, Wentong,
> >
> > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> >>
> >> Hi, Wentong,
> >>
> >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> >>> Hello Wentong,
> >>>
> >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> >>>> is a companion chip designed to provide secure and low power vision
> >>>> capability to IA platforms. IVSC is available in existing
> >>>> commercial platforms from multiple OEMs.
> >>>>
> >>>> The primary use case of IVSC is to bring in context awareness. IVSC
> >>>> interfaces directly with the platform main camera sensor via a
> >>>> CSI-2 link and processes the image data with the embedded AI
> >>>> engine. The detected events are sent over I2C to ISH (Intel Sensor
> >>>> Hub) for additional data fusion from multiple sensors. The fusion
> >>>> results are used to implement advanced use cases like:
> >>>>  - Face detection to unlock screen
> >>>>  - Detect user presence to manage backlight setting or waking up
> >>>> system
> >>>
> >>> Do you have plan to support these features in the ivsc driver in the
> >>> future ?
> >>>
> >>>> Since the Image Processing Unit(IPU) used on the host processor
> >>>> needs to configure the CSI-2 link in normal camera usages, the
> >>>> CSI-2 link and camera sensor can only be used in mutually-exclusive
> >>>> ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link
> >>>> and camera sensor. The IPU driver can take ownership of the CSI-2
> >>>> link and camera sensor using interfaces provided by this IVSC driver.
> >>>>
> >>>> Switching ownership requires an interface with two different
> >>>> hardware modules inside IVSC. The software interface to these
> >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> >>>> These two hardware modules have two different MEI UUIDs to enumerate.
> These hardware modules are:
> >>>>  - ACE (Algorithm Context Engine): This module is for algorithm
> >>>> computing when IVSC owns camera sensor. Also ACE module controls
> >>>> camera sensor's ownership. This hardware module is used to set ownership
> of camera sensor.
> >>>>  - CSI (Camera Serial Interface): This module is used to route
> >>>> camera sensor data either to IVSC or to host for IPU driver and application.
> >>>>
> >>>> IVSC also provides a privacy mode. When privacy mode is turned on,
> >>>> camera sensor can't be used. This means that both ACE and host IPU
> >>>> can't get image data. And when this mode is turned on, host IPU
> >>>> driver is informed via a registered callback, so that user can be notified.
> >>>
> >>> How does the privacy mode work, and how can the user trust that the
> >>> closed-source IVSC and IME firmwares will honour the privacy settings ?
> 
> As I know, without IVSC, once user enable the privacy mode, the Intel
> Converged Security Engine will configure the IPU camera mask (security register),
> which will mask the specific CSI2 port and produce dummy imaging data. For the
> case with IVSC, there is no final solution on Linux so far I think.
> 
> Wentong, is IVSC trying to cut off the stream and then notify user and IPU?

yes

> 
> >>>
> >>
> >> Continue with question from Laurent,
> >>
> >> How IVSC handle the privacy request from user? Is there some
> >> notification mechanism to user-space?

IVSC has already defined privacy callback for host IPU/camera driver.

> > I'd have concern if IVSC driver
> >> need private callback to request back-end(e.g. ISP driver) to handle stream
> cutting.
> >
> > How does the privacy mode work, does it just pass zeroes (or other
> > dummy
> > data) towards the host or nothing?

No data on CSI transmitter side

> >
> > A V4L2 control can be used for the purpose of passing the information
> > to the user space at least.

I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.

BR,
Wentong  

> >
> 
> --
> Best regards,
> Bingbu Cao

Hi Wentong,

On Fri, Feb 17, 2023 at 06:28:32AM +0000, Wu, Wentong wrote:
> 
> 
> > -----Original Message-----
> > From: Bingbu Cao <bingbu.cao@linux.intel.com>
> > Sent: Friday, February 17, 2023 9:44 AM
> > 
> > Hi, Sakari,
> > 
> > On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > > Hi Bingbu, Wentong,
> > >
> > > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> > >>
> > >> Hi, Wentong,
> > >>
> > >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > >>> Hello Wentong,
> > >>>
> > >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> > >>>> is a companion chip designed to provide secure and low power vision
> > >>>> capability to IA platforms. IVSC is available in existing
> > >>>> commercial platforms from multiple OEMs.
> > >>>>
> > >>>> The primary use case of IVSC is to bring in context awareness. IVSC
> > >>>> interfaces directly with the platform main camera sensor via a
> > >>>> CSI-2 link and processes the image data with the embedded AI
> > >>>> engine. The detected events are sent over I2C to ISH (Intel Sensor
> > >>>> Hub) for additional data fusion from multiple sensors. The fusion
> > >>>> results are used to implement advanced use cases like:
> > >>>>  - Face detection to unlock screen
> > >>>>  - Detect user presence to manage backlight setting or waking up
> > >>>> system
> > >>>
> > >>> Do you have plan to support these features in the ivsc driver in the
> > >>> future ?
> > >>>
> > >>>> Since the Image Processing Unit(IPU) used on the host processor
> > >>>> needs to configure the CSI-2 link in normal camera usages, the
> > >>>> CSI-2 link and camera sensor can only be used in mutually-exclusive
> > >>>> ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link
> > >>>> and camera sensor. The IPU driver can take ownership of the CSI-2
> > >>>> link and camera sensor using interfaces provided by this IVSC driver.
> > >>>>
> > >>>> Switching ownership requires an interface with two different
> > >>>> hardware modules inside IVSC. The software interface to these
> > >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> > >>>> These two hardware modules have two different MEI UUIDs to enumerate.
> > These hardware modules are:
> > >>>>  - ACE (Algorithm Context Engine): This module is for algorithm
> > >>>> computing when IVSC owns camera sensor. Also ACE module controls
> > >>>> camera sensor's ownership. This hardware module is used to set ownership
> > of camera sensor.
> > >>>>  - CSI (Camera Serial Interface): This module is used to route
> > >>>> camera sensor data either to IVSC or to host for IPU driver and application.
> > >>>>
> > >>>> IVSC also provides a privacy mode. When privacy mode is turned on,
> > >>>> camera sensor can't be used. This means that both ACE and host IPU
> > >>>> can't get image data. And when this mode is turned on, host IPU
> > >>>> driver is informed via a registered callback, so that user can be notified.
> > >>>
> > >>> How does the privacy mode work, and how can the user trust that the
> > >>> closed-source IVSC and IME firmwares will honour the privacy settings ?
> > 
> > As I know, without IVSC, once user enable the privacy mode, the Intel
> > Converged Security Engine will configure the IPU camera mask (security register),
> > which will mask the specific CSI2 port and produce dummy imaging data. For the
> > case with IVSC, there is no final solution on Linux so far I think.
> > 
> > Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
> 
> yes

Does the CSI-2 transmitter on IVCS go to some LP mode during this time, or
does the receiver need to initialise the bus again when the stream resuems?

> 
> > 
> > >>>
> > >>
> > >> Continue with question from Laurent,
> > >>
> > >> How IVSC handle the privacy request from user? Is there some
> > >> notification mechanism to user-space?
> 
> IVSC has already defined privacy callback for host IPU/camera driver.
> 
> > > I'd have concern if IVSC driver
> > >> need private callback to request back-end(e.g. ISP driver) to handle stream
> > cutting.
> > >
> > > How does the privacy mode work, does it just pass zeroes (or other
> > > dummy
> > > data) towards the host or nothing?
> 
> No data on CSI transmitter side

Can it stop in the middle of the frame? Or is it guaranteed to produce full
frames (assuming the sensor does)?

> 
> > >
> > > A V4L2 control can be used for the purpose of passing the information
> > > to the user space at least.
> 
> I will take some time to review V4L2 sub-device and control mechanism,
> and then update the driver.

-- 
Kind regards,

Sakari Ailus

Hi Sakari,

> -----Original Message-----
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> Sent: Friday, February 17, 2023 6:50 PM
> 
> Hi Wentong,
> 
> On Fri, Feb 17, 2023 at 06:28:32AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Bingbu Cao <bingbu.cao@linux.intel.com>
> > > Sent: Friday, February 17, 2023 9:44 AM
> > >
> > > Hi, Sakari,
> > >
> > > On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > > > Hi Bingbu, Wentong,
> > > >
> > > > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> > > >>
> > > >> Hi, Wentong,
> > > >>
> > > >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > > >>> Hello Wentong,
> > > >>>
> > > >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> > > >>>> Falls", is a companion chip designed to provide secure and low
> > > >>>> power vision capability to IA platforms. IVSC is available in
> > > >>>> existing commercial platforms from multiple OEMs.
> > > >>>>
> > > >>>> The primary use case of IVSC is to bring in context awareness.
> > > >>>> IVSC interfaces directly with the platform main camera sensor
> > > >>>> via a
> > > >>>> CSI-2 link and processes the image data with the embedded AI
> > > >>>> engine. The detected events are sent over I2C to ISH (Intel
> > > >>>> Sensor
> > > >>>> Hub) for additional data fusion from multiple sensors. The
> > > >>>> fusion results are used to implement advanced use cases like:
> > > >>>>  - Face detection to unlock screen
> > > >>>>  - Detect user presence to manage backlight setting or waking
> > > >>>> up system
> > > >>>
> > > >>> Do you have plan to support these features in the ivsc driver in
> > > >>> the future ?
> > > >>>
> > > >>>> Since the Image Processing Unit(IPU) used on the host processor
> > > >>>> needs to configure the CSI-2 link in normal camera usages, the
> > > >>>> CSI-2 link and camera sensor can only be used in
> > > >>>> mutually-exclusive ways by host IPU and IVSC. By default the
> > > >>>> IVSC owns the CSI-2 link and camera sensor. The IPU driver can
> > > >>>> take ownership of the CSI-2 link and camera sensor using interfaces
> provided by this IVSC driver.
> > > >>>>
> > > >>>> Switching ownership requires an interface with two different
> > > >>>> hardware modules inside IVSC. The software interface to these
> > > >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> > > >>>> These two hardware modules have two different MEI UUIDs to
> enumerate.
> > > These hardware modules are:
> > > >>>>  - ACE (Algorithm Context Engine): This module is for algorithm
> > > >>>> computing when IVSC owns camera sensor. Also ACE module
> > > >>>> controls camera sensor's ownership. This hardware module is
> > > >>>> used to set ownership
> > > of camera sensor.
> > > >>>>  - CSI (Camera Serial Interface): This module is used to route
> > > >>>> camera sensor data either to IVSC or to host for IPU driver and
> application.
> > > >>>>
> > > >>>> IVSC also provides a privacy mode. When privacy mode is turned
> > > >>>> on, camera sensor can't be used. This means that both ACE and
> > > >>>> host IPU can't get image data. And when this mode is turned on,
> > > >>>> host IPU driver is informed via a registered callback, so that user can be
> notified.
> > > >>>
> > > >>> How does the privacy mode work, and how can the user trust that
> > > >>> the closed-source IVSC and IME firmwares will honour the privacy
> settings ?
> > >
> > > As I know, without IVSC, once user enable the privacy mode, the
> > > Intel Converged Security Engine will configure the IPU camera mask
> > > (security register), which will mask the specific CSI2 port and
> > > produce dummy imaging data. For the case with IVSC, there is no final
> solution on Linux so far I think.
> > >
> > > Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
> >
> > yes
> 
> Does the CSI-2 transmitter on IVCS go to some LP mode during this time, 

Yes, The low power mode is following the D-Phy spec.

> or does
> the receiver need to initialise the bus again when the stream resuems?

No, it's the same link.

> 
> >
> > >
> > > >>>
> > > >>
> > > >> Continue with question from Laurent,
> > > >>
> > > >> How IVSC handle the privacy request from user? Is there some
> > > >> notification mechanism to user-space?
> >
> > IVSC has already defined privacy callback for host IPU/camera driver.
> >
> > > > I'd have concern if IVSC driver
> > > >> need private callback to request back-end(e.g. ISP driver) to
> > > >> handle stream
> > > cutting.
> > > >
> > > > How does the privacy mode work, does it just pass zeroes (or other
> > > > dummy
> > > > data) towards the host or nothing?
> >
> > No data on CSI transmitter side
> 
> Can it stop in the middle of the frame?

No,

 Or is it guaranteed to produce full frames
> (assuming the sensor does)?

Yes, full frame will be guaranteed as camera sensor does.

> 
> >
> > > >
> > > > A V4L2 control can be used for the purpose of passing the
> > > > information to the user space at least.
> >
> > I will take some time to review V4L2 sub-device and control mechanism,
> > and then update the driver.
> 
> --
> Kind regards,
> 
> Sakari Ailus