Dongsoo Kim ([email protected])[email protected] Heungjun...

58
Dongsoo Kim ([email protected] ) Heungjun Kim([email protected] ) Samsung Electronics

Transcript of Dongsoo Kim ([email protected])[email protected] Heungjun...

Page 1: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

Dongsoo Kim ([email protected])Heungjun Kim([email protected])

Samsung Electronics

Page 2: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

2

Introducing mobile camera devices- SoC camera subsystem

- Mobile camera module devices

Framework for digital camera- Video4Linux2 and camera device

- New APIs for high-end mobile camera devices

- Dual camera framework for 3G handset

Porting issues in SoC camera subsystem

Page 3: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

3

AP : Application Processor

Camera interface : interface device which is syncing image data from external camera device and gives some commands to them

Camera module : external camera device packaged with lens, sensor and ISP on it

ISP : Image Signal Processor. Also supports lots of additional functions

V4L2 : Video for Linux 2

Page 4: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

4

This document contains some animations.

Slideshow mode is recommended

Page 5: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

5

Exposure control- Aperture (Iris)

- Shutter

- ISO Lens control- Focus

- Zoom

Strobe support- Various program

mode : slow sync, red-eye reduction and on….

Page 6: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

6

Introducing mobile camera devices

Page 7: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

7

History of camera phone

First Camera Phone in 2000 with 3.5MP

1.3MP in 20032MP in 20043MP in20045MP in20047MP in 200510MP in 200612MP in 2009

Page 8: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

8

Old and low-end mobile camera modules

Feature Low end camera

Exposure

Aperture Not supported

Shutter Just change frame rate

ISO Low sensitivity & High noise level

Lens

Focus Pan focus

Zoom Digital zoom

Strobe Just LED light most of time

Page 9: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

9

Brand new and high-end mobile camera modulesFeature High end camera

Exposure

Aperture Supported with lens packaging

Shutter Supported with lens packaging

ISO High sensitivity & Low noise level

Lens

Focus Movable focus point (Auto/manual)

Zoom Optical zoom & Digital zoom

Strobe XENON flash is also supported

Page 10: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

10

What is SoC camera subsystem?

ARM SoC solution

ARMCore

Peripheral1(Display)

Peripheral2 (MMC)

Peripheral3 (USB)Camera interface

Peripherals

External

Camera module

External

Camera module

Page 11: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

11

Camera interface

Buffer handlingBuffer

handling

Resolution

handling

Resolution

handling

Data format handling

(YUV/RAW/JPEG…)

Data format handling

(YUV/RAW/JPEG…)

Camera module

I2CI2C

Exposure

control

Exposure

control

Lenscontr

ol

Lenscontr

ol

Strobe

control

Strobe

control

White balanc

e control

White balanc

e control

Effect contr

ol

Effect contr

ol

Facedetec

t

Facedetec

t

Zoomcontr

ol

Zoomcontr

ol

.

.

.

.

.

.

CommandinterfaceCommandinterface

RAW DATA

Image data in expected formatRequest for data

Page 12: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

12

Various ways of preview (Live view) in SoC camera subsystems

CameraModule

Memory

CameraInterfa

ce

ARMcore

Frame buffer

OKCanc

el

User Applicati

onCOPY!

DATA

DA

TA

Page 13: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

13

Various ways of preview (Live view) in SoC camera subsystems

CameraModule

Memory

CameraInterfa

ce

ARMcore

Frame buffer

OKCanc

el

User Applicati

on

COPY!

DMA Through

DMA

DATA

DA

TA

Saveas file

Page 14: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

14

How to capture still shots in SoC camera subsystem- Your camera module has a dedicated JPEG encoder

on it

Memory

CameraInterfa

ce

ARMcore

User Applicati

onCOPY!

DATAMEMORY

Camera module

RAWDATA

PROCESS

JPEGPROCES

S

DATA

DA

TA

Save as JPEG file

Page 15: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

15

How to capture still shots in SoC camera subsystem- No JPEG encoder in your camera module

CameraModule

Memory

CameraInterfa

ce

ARMcore

User Applicati

onCOPY!

DATA

DA

TA

RAWDATA

Codec S/W

Save as JPEG file

Page 16: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

16

How to take motion pictures in SoC camera subsystem

CameraModule

Memory

CameraInterfa

ce

ARMcore

Frame buffer

OKCanc

el

User Applicati

on

COPY!

DATA

DA

TA

COPY!

Codec S/W

Save as…

Page 17: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

17

In Kernel

V4L2 (AKA Video For Linux 2)- The second version of the Video For Linux API

- Kernel interface for analog radio and video capture and output drivers

In user space

Multimedia middleware

Gstreamer- Pipeline based multimedia framework written in the C programming

language with the type system based on GObject

OpenMAX- Cross-platform set of C-language programming interfaces that provides

abstractions for routines especially useful for audio, video, and still images

Page 18: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

18

Is V4L2 enough for mobile

digital camera modules?

No way

Page 19: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

19

Framework for digital camera

Page 20: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

20

Camera subsystem in Linux kernel aspect

Camera interfaceCamera interface

Buffer handlingBuffer

handlingResolution handling

Resolution handling

Data format handling

(YUV/RAW/JPEG…)

Data format handling

(YUV/RAW/JPEG…)

CommandinterfaceCommandinterface

V4L2 driverV4L2 driver

Camera module

Exposure

control

Exposure

control

Lenscontro

l

Lenscontro

l

Strobecontro

l

Strobecontro

l

White balance control

White balance control

Effect contro

l

Effect contro

l

FacedetectFace

detect

Zoomcontro

l

Zoomcontro

l

.

.

.

.

.

.

V4L2 driverV4L2 driver

videobufvideobuf-core

videobuf-sg-dma

videodevV4l2-dev

V4l2-ioctl

V4l2-subdev(new)

V4l2-device

V4l2-int-

device(old)

V4L2

Page 21: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

21

How does it work with V4L2?

Open deviceOpen

device

VIDIOC_QUERYCAP

VIDIOC_QUERYCAP

Application or Middleware

VIDIOC_S_INPUT

VIDIOC_S_INPUT

VIDIOC_REQBUF

VIDIOC_REQBUF

VIDIOC_QBUF

VIDIOC_QBUF

VIDIOC_STREAMON

VIDIOC_STREAMON

VIDIOC_QBUFVIDIOC_QBUF VIDIOC_DQBUF

VIDIOC_DQBUF

VIDIOC_QUERYBUF

VIDIOC_QUERYBUF

Copy memory to userspace

Page 22: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

22

New V4L2 APIs for digital camera

Exposure control

Page 23: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

23

Need for detailed control in exposure- We’ve got camera devices with mechanical shutter

- Iris could be handled also

- ISO controlled output is somehow useful

Why not using enhanced exposure control like a regular digital camera?

Page 24: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

24

What we’ve got in mainline V4L2 right now?

#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)enum  v4l2_exposure_auto_type {        V4L2_EXPOSURE_AUTO = 0,        V4L2_EXPOSURE_MANUAL = 1,        V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,        V4L2_EXPOSURE_APERTURE_PRIORITY = 3};#define V4L2_CID_EXPOSURE_ABSOLUTE  (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)

No way to control iris

No way to control exposure metering method

Page 25: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

25

Additional exposure control

enum v4l2_exposure_auto_type {V4L2_EXPOSURE_AUTO = 0,V4L2_EXPOSURE_MANUAL = 1,V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,

V4L2_EXPOSURE_APERTURE_PRIORITY = 3,+ /* Additional features for digital camera */+ V4L2_EXPOSURE_ISO_PRIORITY =4,};#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)

. . .

+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

Page 26: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

26

Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products

+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)+#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21)

Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU

{    .id = V4L2_CID_CAM_APERTURE,    .type = V4L2_CTRL_TYPE_MENU,    .name = "Aperture",    .minimum = 0,    .maximum = 4,    .step = 1,    .default_value = 0,},

static const char *camera_iris_stages[] = {/* This module supports 5 Iris stages  * on it's own but F number depends  * how you package Lens module */

"2.8“, "4“, "5.6“, "8“, "11“, NULL};

Page 27: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

27

New exposure metering control

+/* Exposure Methods */+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)+enum v4l2_photometry_mode {+ V4L2_PHOTOMETRY_MULTISEG = 0, + V4L2_PHOTOMETRY_CWA = 1,+ V4L2_PHOTOMETRY_SPOT = 2,+ V4L2_PHOTOMETRY_AFSPOT = 3, +};

Page 28: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

28

New V4L2 APIs for digital camera

Lens control

Page 29: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

29

Need for enhanced Focus mode API

- Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam

- High-end camera modules support for various focus mode presets like

•AF Macro

•AF-S / AF-C

For compatibilityFocus control API should be defined

Page 30: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

30

Focus Mode & Enumeration values

#define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10)#define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11)+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)

+/* Focus Methods */+enum v4l2_focus_mode {+ V4L2_FOCUS_AUTO = 0,+ V4L2_FOCUS_MANUAL = 1,+ V4L2_FOCUS_MACRO = 2,+ V4L2_FOCUS_CONTINUOUS = 3,+};

Page 31: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

31

New V4L2 APIs for digital camera

Object recognition

Page 32: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

32

Object detection

Gee!It recognized my

face!Hello everyone

Page 33: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

33

What do we set?- Detect mode

- Type of object

- How many objects to detect

- Triggered action

What do we get?- How many objects are detected

- Detected object’s type

- Detected object’s coordinates

Page 34: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

34

New API for object recognition- New capability

#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000

+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

- New IOCTL

Page 35: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

35

+/* Object detection and triggered actions */+enum v4l2_recog_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recog_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++enum v4l2_recog_pattern {+ V4L2_RECOG_PATTERN_FACE = 0; /* Face */+ V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */+ V4L2_RECOG_PATTERN_CHAR = 2; /* Character */+}

New API for object detection

Page 36: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

36

+struct v4l2_recog_rect {+ enum v4l2_recog_pattern p; /* What pattern detected */+ struct v4l2_rect o; /* detected area */+ __u32 reserved[4];+}++struct v4l2_recognition_data {+ __u32 detect_cnt; /* detected object counter */+ struct v4l2_recog_rect obj; /* detected area */+ __u32 reserved[4];+};++struct v4l2_recognition {+ enum v4l2_recog_mode mode;+ enum v4l2_recog_pattern pattern; /* What pattern to detect */+ __u32 obj_num; /* How many object to detect */+ __u32 detect_idx; /* select detected object */+ struct v4l2_recog_data data; /* read only */+ enum v4l2_recognition_action action;+ __u32 reserved[4];+};

New API for object detection

Page 37: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

37

What do we set?- Detect mode

- How many objects to detect

- Triggered action

What do we get?- How many objects are detected

- Detected object’s coordinates

Page 38: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

38

New API for object recognition- New capability

#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000

+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

- New IOCTL

Page 39: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

39

+/* Object detection and triggered actions */+enum v4l2_recognition_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recognition_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++struct v4l2_recognition_data {+ __u8 detect_cnt; /* detected object counter */+ struct v4l2_rect o; /* detected area */+};++struct v4l2_recognition {+ enum v4l2_recognition_mode mode;+ __u8 obj_num; /* How many object to detect */+ __u8 detect_idx; /* select detected object */+ struct v4l2_recognition_data data; /* read only */+ enum v4l2_recognition_action action;+};

New API for object detection

Page 40: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

40

New V4L2 APIs for digital camera

Dual camera

Page 41: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

41

What is dual camera?

Dual camera applications- Video telephony

- Camera applications• Taking self portrait shots

Flip side camera: for still shots

Front side camera: for special purpose

Page 42: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

42

H/W restrictions in Application Processor’s camera peripheral interface

Megacamera

VGAcamera

Datapath

Page 43: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

43

H/W restrictions in Application Processor’s camera peripheral interface

“SINGLE” camera interfaceon Application Processor !“SINGLE” camera interfaceon Application Processor !

Only one camera could be using data pins at a time

Only one camera could be using data pins at a time

Each camera is different inEach camera is different in

Working resolution

(SYNC)

Working resolution

(SYNC)Working MCLKWorking MCLK Working PCLKWorking PCLK

Only one camera could be handled at a timeOnly one camera could be handled at a time

Page 44: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

44

We already have API for dual camera- Enumerate video inputs

-  Query or select the current video input

VIDIOC_ENUMINPUTVIDIOC_ENUMINPUT

VIDIOC_G_INPUTVIDIOC_G_INPUT

VIDIOC_S_INPUTVIDIOC_S_INPUT

Page 45: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

45

What do we need to consider?

•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on

•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on

•Context of camera interface•Working information of the camera module getting switched : Clock information and so on

•Context of camera interface•Working information of the camera module getting switched : Clock information and so on

•Let user space application get noticed about camera input devices

•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES

•Let user space application get noticed about camera input devices

•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES

Page 46: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

46

How to switch active camera

Condition 2Shared things

Camera power enable pinCamera data pinsMCLK

Condition 3Dedicated things

Camera enable pinsCamera reset pins

Condition 1How to enable camera module:

Give power sourceEnable expecting cameraGive MCLK CAM

POWERMCLK DATA PINS

MegaEnable

MegaReset

VGAEnable

VGAReset

Mega camera VGA camera

CPU

Page 47: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

47

STREAMOFF

Camera (A)

initialize

Camera (A)

initialize

Start preview

Start preview

Switch to camera (B)

Camera (B)

initialize

Camera (B)

initialize

Camera interface reset

& setting

Start preview

Start preview

Reset camera interface’s

current setting

Make a proper MCLK for camera

(B)

Setup resolution,

color-space, DMA

Stop previewStop DMA

Cam power off

Camera (B) Enable pin on

Camera (A) enable pin off

Campower on

Stop previe

w

Stop previe

w

STREAMON

G_INPUT/

S_INPUT

G_INPUT/

S_INPUT

Page 48: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

48

Porting issues in SoC camera subsystem

Page 49: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

49

I2C protocol has start and stop condition

Some camera modules generate unexpected noise until their power up sequence- Makes I2C bus in bus busy state

Page 50: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

50

How to prevent this malfunctioning issue

Use a “noise free on power up” camera module

Use a level shifter to block and ignore noise from camera module

Change functionality of I2C pins while turning on camera module

OR

OR

Page 51: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

51

All about between camera interface and camera module- Expected resolution from camera interface

- Served resolution from camera module

- Symptoms depend on camera interface

Expecting resolution is bigger than served resolution- A “select time out” error in user space application or

- Preview image is floating in some direction

Serving resolution is bigger than expected resolution- A “select time out” error in user space application or

- Preview angle looks like to be zoomed in

Page 52: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

52

Color order mismatched between camera interface and camera module

Page 53: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

53

Preview freezing while checking lens response- Using while() loop while it checks response, preview will

freeze until focusing job gets finished

- If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.

Page 54: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

54

Solving preview freezing while checking lens statusstruct platform_isp {

const struct platform_data *pdata;struct v4l2_int_device *v4l2_int_device;struct i2c_client *i2c_client;int reschk;

. . . . . . . . .}

static int ioctl_s_ctrl(struct v4l2_int_device *s,struct v4l2_control *vc)

{ struct platform_isp *isp = s->priv; struct i2c_client *client = isp->i2c_client; switch (vc->id) { case V4L2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V4L2_CID_FOCUS_MACRO) { err = i2c_command_for_isp( client, focus_macro ); isp->reschk = 1; } break; }}

If isp->reschk is Set, The kernel thread is active.

Page 55: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

55

Solving preview freezing while checking lens statusint pseudo_reschk_thread (void *data){ int err = 0;……… do { switch (isp->reschk) { case 1: err = i2c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i2c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break;…… } } while(1); return 0;}

This function return when the ISP ismatched the focus(+ return) or timeout(- return).

This is able to re-check to the next check time.

Platform Data Vairable for isp, or static variable.

Page 56: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

56

JPEG has no synchronization protocol when it gets transferred

All about camera interface and camera module’s data synchronization

- When camera interface supports JPEG sync• Follow user manual and make camera module to follow that

protocol

- When camera interface does not support JPEG sync• Make synchronization protocol: make JPEG data to be

synchronized with VSYNC

Page 57: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

57

Q&A

Page 58: Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com.

58

Thank you