Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

danieldegrasse
Copy link
Collaborator

@danieldegrasse danieldegrasse commented May 15, 2024

Add SMARTDMA video driver. This driver uses the SMARTDMA engine as a
parallel camera interface, which can read QVGA frames from a camera
device. Due to SRAM constraints, the video driver divides the camera
stream into multiple horizontal video buffers as it streams them back to
an application.

Note that this PR also contains two additional major changes:

  • an addition to the video API to enable streaming partial frames within one video buffer
  • a rework of the SMARTDMA driver interface to better support custom firmware use cases like the camera in this PR

This PR depends on #72826 for camera support

PR has been tested on the following boards:

  • frdm_mcxn947//cpu0: samples/subsys/video/capture
  • mimxrt595_evk//cm33: samples/subsys/display/lvgl (using G1120B0MIPI shield)

@zephyrbot
Copy link
Collaborator

zephyrbot commented May 15, 2024

The following west manifest projects have been modified in this Pull Request:

Name Old Revision New Revision Diff
hal_nxp zephyrproject-rtos/hal_nxp@abc6697 (master) zephyrproject-rtos/hal_nxp#389 zephyrproject-rtos/hal_nxp#389/files

Note: This message is automatically posted and updated by the Manifest GitHub Action.

@zephyrbot zephyrbot added manifest manifest-hal_nxp DNM This PR should not be merged (Do Not Merge) labels May 15, 2024
Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA
resolution in YUV and RGB mode.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add ov7670 camera driver to build test for video drivers

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Video sources on some smaller embedded systems cannot feasibly stream a
full frame into RAM (as there is not enough space). On these systems, a
scheme can be used where each video buffer corresponds to a new
horizontal "slice" of the full frame, which the system can then process.

In order to support this use case, add "vbufs_per_frame" field to video
capabilities structure. All video source drivers are also updated in
this PR to set the vbufs_per_frame field to 1, as this is the expected
default value when a single video buffer corresponds to one frame.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Use vbufs_per_frame field to determine buffer size allocated within the
video capture sample

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
The SMARTDMA is a programmable DMA engine, and supports custom firmware
in order to run complex DMA operations. Update the driver to increase
the flexibility users have when configuring the SMARTDMA with
custom firmware, and remove the RT500 display firmware specific
definitions and functionality from the driver.

This display setup is now handled from the MIPI DSI driver, since the
firmware used for this case is specific to the MIPI DSI IP.

This change also requires an update to the RT500 devicetree, as the
register definition for the SMARTDMA has changed, so the base address
must as well.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add SMARTDMA video driver. This driver uses the SMARTDMA engine as a
parallel camera interface, which can read QVGA frames from a camera
device. Due to SRAM constraints, the video driver divides the camera
stream into multiple horizontal video buffers as it streams them back to
an application.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add definition for SMARTDMA device to the MCXN94x devicetree.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add support for LPI2C7. This peripheral is used for interfacing with
cameras connected to the J9 header.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add support for SMARTDMA camera engine, and a OV7670 parallel camera
definition for the frdm_mcnx947 board. This support has been tested with
the video capture sample.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add support for using the SMARTDMA engine on the FRDM-MCXN947 board with
the video capture sample.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
@josuah
Copy link
Contributor

josuah commented May 20, 2024

  • an addition to the video API to enable streaming partial frames within one video buffer

Some news about it there, with a proposed API: #66994 (comment)

danieldegrasse added a commit to nxp-upstream/zephyr that referenced this pull request May 20, 2024
Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA
resolution in YUV and RGB mode.

Support was verified on the FRDM-MCXN947, using the SmartDMA camera
engine, which is enabled in the following PR:
zephyrproject-rtos#72827

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
@josuah
Copy link
Contributor

josuah commented May 23, 2024

here (need to click on "load diff") is a new API that seems underway:

	if (drv_data->fifo_length == 0) {
		vbuf->flags = VIDEO_BUF_EOF;
	} else {
		vbuf->flags = VIDEO_BUF_FRAG;
		k_work_submit_to_queue(&ac_work_q, &drv_data->buf_work);
	}

@danieldegrasse
Copy link
Collaborator Author

here (need to click on "load diff") is a new API that seems underway:

No issues with this API, I suppose we would like to make this PR dependent on that API merging? I could also cherry pick the commits adding the API from that PR (@ArduCAM would you be ok with that approach?)

@ArduCAM
Copy link

ArduCAM commented May 25, 2024

here (need to click on "load diff") is a new API that seems underway:

No issues with this API, I suppose we would like to make this PR dependent on that API merging? I could also cherry pick the commits adding the API from that PR (@ArduCAM would you be ok with that approach?)

Yes, have marked resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DNM This PR should not be merged (Do Not Merge) manifest manifest-hal_nxp
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants