© Copyright Khronos Group, 2002 - Page 1

© Copyright Khronos Group, 2002 -

BOF MeetingBOF MeetingSIGGRAPH 2002SIGGRAPH 2002

[email protected]

Khronos Secretary


Today’s AgendaToday’s AgendaKhronos - Technical and Organizational OverviewKhronos - Technical and Organizational Overview• What is Khronos? What are our goals?

- Neil Trevett, Senior Vice President, 3Dlabs - Khronos Secretary

• OpenML™ 1.0 Technical Overview- Jean-Luc Dery, Technical Leader, Advanced Systems, Discreet

• OpenGL® ES - Embedded API initiative- Neil Trevett, Senior Vice President, 3Dlabs - Khronos Secretary

• Khronos IP Policy – an Innovative Approach- Shawn Underwood, Director of Marketing, SGI

• How do I get involved in driving Khronos Standards?- Neil Trevett, Senior Vice President, 3Dlabs - Khronos Secretary

• Questions and Answers – we want your feedback!


Driving OPEN Standards for CREATING

and DEPLOYINGDYNAMIC Media


Khronos’ Evolving OrganizationKhronos’ Evolving Organization

Khronos 1.0Original Promoters cooperated under Khronos Participation agreement 1.0

to produce OpenML 1.0 specification

Khronos 2.0New Khronos Participation Agreement 2.0

Recruiting member companies nowEnables multiple working groups

Media API for Dynamic Media Authoring.Specification available now

Deploying and evolving OpenML OpenGL ES Small footprint

OpenGL profiles for Embedded Systems


Khronos Goal #1:Khronos Goal #1:Simplify Dynamic Media Simplify Dynamic Media AUTHORINGAUTHORING

Publicly Released at Siggraph 2001Publicly Released at Siggraph 2001

The result…The result…


Dynamic Media ProgrammingDynamic Media ProgrammingSynergistic integration of OpenML and OpenGL Synergistic integration of OpenML and OpenGL • What OpenGL is to GRAPHICS - 2D, 3D and image processing…

…OpenML is to MEDIA - video and audio processing• OpenML and OpenGL together create a unified programming environment

- With seamless integration of graphics and media

• Tightly integrated graphics, video and audio = DYNAMIC MEDIA

Workstation3D graphics

Professional video and audio authoring


?

OpenML Digital Media OpenML Digital Media Programming EnvironmentProgramming Environment

ApplicationsApplications

WindowWindow SystemSystem OpenGLOpenGL

®® MLdc ML

2D Graphics 3D Graphics Displays Video Audio

Synchronization

OpenGL Extensions


Technical Strategy for OpenML 1.0Technical Strategy for OpenML 1.0Move PROVEN technologies into Open StandardsMove PROVEN technologies into Open Standards

UST/MSCSynchronization

#4: synchronization of diverse media streams (graphics, audio, video)

SGI’s dmSDKDigital Media Handling

#1: Configure and control the Input/Output and processing of video and digital audio ML

OpenGL ExtensionsIntegrating graphics and video

#3: Enable OpenGL hardware to add more value to the media authoring workflow

SGI’s Xdc extension to X11Display Control

#2: Configure and control display devices MLdc

Problem Solution


Khronos Goal #2:Khronos Goal #2:

EmbeddedEmbedded Devices and Devices and AppliancesAppliances Create Standards for Dynamic Media on Create Standards for Dynamic Media on


OpenGL® ES


Well Defined OpenGL ProfilesWell Defined OpenGL ProfilesAddressing broad application requirementsAddressing broad application requirements• Khronos activities are endorsed and supported by the OpenGL ARB

2D

3D

OpenGL ES

ARB

Feedback and Ratification

Workstation FocusMedia and Embedded Focus

Small footprint OpenGL profiles


OpenML 1.0 Technical OpenML 1.0 Technical OverviewOverview

Jean-Luc DeryTechnical Leader, Advanced Systems

Discreet, a division of Autodesk


OverviewOverviewWhat is OpenML?What is OpenML?• Open and cross-platform standard• Programming environment used for capturing, processing, synchronizing

and playing digitial media content (video and audio)• Composed of four components:

- Graphics video backend control (MLdc)- Video/Audio IO (ML)- OpenGL extensions- Synchronization (UST/MSC)

ApplicationApplication

MLdcMLdc

Video channelsVideo channels Graphics devicesGraphics devicesDigital media devicesDigital media devices

OpenGL + ExtensionsOpenGL + ExtensionsMLML


OpenML’s “MLdc” APIOpenML’s “MLdc” APIDisplay control Display control interfaceinterface• Enables an application to control the video back-end of the graphics

devices that produce video output streams• Enables to set up channels in the display screen(s)

- Channels that are driven from sections of physical frame buffers- Channels depends on the display device

Frame Buffers

Progressive ScanComputer MonitorProgressive ScanComputer Monitor

Progressive Scan HDTV Monitor

Progressive Scan HDTV Monitor

Interlaced Video Monitor

Interlaced Video Monitor

ChannelsScreens


OpenML’s “MLdc” APIOpenML’s “MLdc” API (ctn’d) (ctn’d)MLdc provides MLdc provides controlcontrol for: for:• Screens and channels • Quering monitor information• Video output channel position• Video format query and settings• Vertical retrace rate • Scan mode (interlaced vs. progressive)• Gamma ramp control• Video synchonization sources• Genlock state notification


OpenML’s “ML” API OpenML’s “ML” API Media device control interfaceMedia device control interface• Low level API that allows to configure and control video/audio device

input, output and processing features• Allows to setup processing pipelines between application and I/O devices

- Asynchronous communication between application and media devices- Device control and capability queries- Synchronization mechanism- Processing capability for digital media streams- Optimized buffering mechanism for smooth and efficient delivery of media streams

OpenInventorOpenInventor QuickTimeQuickTime

OpenGLOpenGL MLML

Low levelLow level(ioctl interface)(ioctl interface)

3D Media


OpenML’s “ML” APIOpenML’s “ML” API (ctn’d) (ctn’d)Application Application and and memorymemory centric modelcentric model

Application memory Application memory

Videoinput/output

device

Videoinput/output

device

GraphicsGraphics

1. send2. receive

3. drawPixels


OpenML’s OpenML’s OpenGL ExtensionsOpenGL ExtensionsVideo imaging and 3D graphics integrationVideo imaging and 3D graphics integration• OpenML is built around OpenGL• OpenML and OpenGL: A tight and synergistic integration• Extends OGL with video primitives and DCC rendering features• Defines a set of OpenGL extensions that OpenML compliant hardware

must implement


OpenML’s OpenGL ExtensionsOpenML’s OpenGL Extensions (ctn’d)(ctn’d)Current extensionsCurrent extensions• Pbuffers are the fundamental method of integrating graphics and video

- Enables buffering and off-screen processing of video data- The primary data link between OpenML and OpenGL

• OpenGL imaging extensions are used to process video streams- Filtering, color correction, blending, image enhancement, color space conversion

• Synchronization control- Primitives to control UST/MSC synchronization from OpenGL- Functions added: WaitForMSC, SwapBuffersMSC, GetSyncValues

• Interlace support- Based on two previous extensions: SGIX_interlace, INGR_interlace_read

Enables reading and writing of the frame buffer while skipping every other line- Enables processing (e.g., convolution) on interlaced video streams


OpenML’s OpenGL ExtensionsOpenML’s OpenGL Extensions (ctn’d)(ctn’d)Current extensionsCurrent extensions• Texture color mask management to enable advanced compositing

- Enables independent loading of RGB images and Alpha masks in texture memory- Recently promoted from an SGI extension to an ARB extension- Works for textures like the OpenGL color mask works for the color buffer

• Chroma resampling to support pixel formats such as YCrCb- Upsampling and downsampling when converting between YCrCb and RGB- Sampling options are: Replicate, Zero fill, Average


OpenML SynchronizationOpenML SynchronizationMMedia edia device device synchronizationsynchronization interface interface• Critical part of authoring environment• Allows to synchronize video, audio and graphics streams making sure

that all media streams start at the same time and stay in sync• Non-trivial problem to solve

- Devices from various manufacturers - Device latencies are unpredictable- OS scheduling latencies Audio Device AAudio Device A

Video Device AVideo Device A

Audio Device BAudio Device B

Gfx Device BGfx Device B

Gfx Device AGfx Device A

ApplicationApplication Do all the streams come out in sync?


OpenML SynchronizationOpenML Synchronization (ctn’d) (ctn’d)The UST/MSC/SBC SolutionThe UST/MSC/SBC Solution• UST - Unadjusted System Time – time reference for s/w and devices

- 64-bit monotonic counter with nanosecond resolution, microsecond system-wide accuracy

• MSC - Media Stream Counter - incremented for each channel sample- Video - per frame, Graphics - per retrace, audio - per audio sample, e.g. 44.1KHz

• SBC – Swap Buffer Count - incremented on graphics buffer swap- Not necessarily in sync with retrace

UST

Graphics devices

Digital audio/video devicesSBC

OpenGLOpenGL MLML

MSC

Output graphicframes/ samples

Video/AudioIO samples


OpenML SynchronizationOpenML Synchronization (ctn’d) (ctn’d)Robust synchronization – even on non real-time OSRobust synchronization – even on non real-time OS• Every media sample is time-stamped

- Timestamp can be accessed through OpenGL and ML APIs for temporal analysis and control

• Application can synchronize output events to UST- Enabling precise timing and synchronization in non-real-time operating systems

• Applications can detect dropped samples- And take corrective action


Real-world ExamplesReal-world Examples

• Currently shipping Discreet applications on PC, Mac and SGI- Special effects and compositing- Off-line and on-line editing- Broadcast graphics- 3D modeling and animation

• Scenarios examples- Video preview- Media playback- Video capture- Real-time video effects

• Expertise learned from these products was used to define the OpenML standard


Benefits of OpenMLBenefits of OpenMLTo Developers…To Developers…• OpenML will allow multimedia application developers to…

- Spend more time and energy on application development and less time on mundane details such as device control, synchronization and buffering issues

- Avoid dealing with different APIs for video, graphics and audio for each operating environment- Develop and deploy efficient cross-platform applications that interoperate a wide selection of

systems and peripherals

• OpenML will allow hardware vendors to…- Gain access to new market opportunities by delivering on the promise of interoperability- Leverage the efforts of a larger base of application developers- Add significantly more value to the rich media creation workflow


Future Developments - OpenML 1.1Future Developments - OpenML 1.1Meeting Meeting the ongoing needs of media authoringthe ongoing needs of media authoring• Tight integration into the evolving OpenGL 1.1 framework• Closer integration of ML and OpenGL video streaming• Higher quality rendering• OpenGL resource management• Conformance tests and performance benchmarks• POSSIBILITIES include:

- Extended color range >8 bit per component- Improved compositing capabilities – separate alpha and matte channels- Improved texture and memory management – including render to texture, interlaced textures- Improved efficiency – including asynchronous operations- Improved live video handling - streaming of video to/from Pbuffers

We need your input now – particularly application We need your input now – particularly application developers!developers!


Any Questions on OpenML?Any Questions on OpenML?


Want to use OpenML Want to use OpenML 1.0?1.0?ApplicationApplication Developers Developers

• No licenseNo license needed to use OpenML needed to use OpenML

• Specification, free of charge, at www.khronos.orgSpecification, free of charge, at www.khronos.org

OpenML Library OpenML Library ImplementersImplementers • Royalty-free Royalty-free AdoptersAdopters AgreementAgreement

• Spec and agreement at www.khronos.orgSpec and agreement at www.khronos.org


Encouraging OpenML DeploymentEncouraging OpenML DeploymentOpenML Implementation Working GroupOpenML Implementation Working Group• Create sample implementation

- Starting from SGI’s donated dmSDK source

• Define DDI level interfaces- To enable new devices to be added easily

• Cooperative, possibly open source, development- Source trees managed by Khronos

• Define and implement conformance tests- Maintain specification integrity and ensure portability


OpenGL ES OverviewOpenGL ES Overview

Neil TrevettVice President, 3Dlabs


Khronos Goal #2:Khronos Goal #2:

EmbeddedEmbedded Devices and Devices and AppliancesAppliances Create Standards for Dynamic Media on Create Standards for Dynamic Media on



Inverse Screen Size LawInverse Screen Size LawThe need for advanced graphicsThe need for advanced graphics• Smaller screens need more advanced graphics processing per pixel

- To make best use of restricted screen real-estate

• Advanced graphics techniques provided by state-of-the-art APIs- Anti-aliasing, multi-level compositing, rich media, 3D

Interactive3D Gaming Content

Video Processing and synchronization Perspective windows

maximize screen real-estate

Anti-aliased text to makesmall text extremely legible

Translucent Menus don’t obscure lower layers


The Power of a Standard APIThe Power of a Standard APITo enable hardware and software communitiesTo enable hardware and software communities• But there are no acceptable cross-platform embedded graphics APIs

- Particularly with advanced graphics capabilities

• We are missing this fundamental prerequisite for market growth- The graphics hardware community should take the imitative in driving the platform forward- Just as on the PC

Standard Graphics

API

Embedded Hardware

Community

Embedded Software

Community

Requirements

Capabilities

Opportunities to optimize cost, power and performance for standardized functionality

Can use advanced graphics capabilities consistently across

multiple platforms


Cross Platform Solution EssentialCross Platform Solution EssentialMany platforms and initiatives need these APIsMany platforms and initiatives need these APIs

WindML® – expanding multimedia capabilities of VxWorks®

Looking to further expand 3D and rich media delivery capabilities

Embedded Linux – need for low-footprint APIsNo standardized 2D API other than X Windows – not low footprint

Increasing use of OpenGL for 3D – but not low footprint

Java Community – Java2D, Java Media Framework, Java Advanced ImagingJava3D is a high-level API – need low-footprint 3D API for embedded applications

ARINC – standards for advanced cockpit displaysNeed standardized 2D & 3D APIs

Severe certification demands drive need for minimum API size

Leading Symbian OS for mobile consumer devicesNeeds minimal API size and increasing functionality

Web3D ConsortiumExtensible standards for 3D delivery over the internet, driving 3D in MPEG-4

BUT needs target small footprint 2D/3D API for higher-level standards


OpenGL ESOpenGL ESA proposed solution for advanced embedded A proposed solution for advanced embedded graphicsgraphics• Create well-defined subset profiles of OpenGL

- A series of increasingly compact profiles - Selecting functionality needed for key target markets

• Proper subsets ensure synergy with, and migration path to, full OpenGL- No new technologies are needed – just agreement on subsets of an existing API

Eliminate redundancyDuplicated functionalityUnnecessary queriesData-type variants

Eliminate workstation functionality e.g. 3D texturesDifficult/costly to implement with little benefit in embedded space

Select functionality needed for key target

markets


Poorly defined standard API requirements.Discussions with

BroadcastCL group- to avoid divergence

Strong support from ARINC and aerospace companies including Boeing, Airbus, Rockwell, Honeywell, Smiths, Lockheed…

At a meeting in April 2002, Symbian and the cell phone industry selected Khronos’

“Embedded OpenGL” as the graphics API for cell phones with accelerated 3D in 2004

Phones

Web pads

Cameras

PDAs

Hand-heldAppliances

Line-poweredAppliances

STBs

InternetTerminals

Consoles

Industry MomentumIndustry Momentum

Safety Critical

Avionics

Automotive

New Visual Processing

APIs


Why OpenGL?Why OpenGL?The obvious starting pointThe obvious starting point• OpenGL is already the most widely available cross-platform graphics API

- Workstations to embedded appliances, PCs to Macs, Windows to Linux• Open Standard – not controlled by any one company

- Specification controlled by the OpenGL Architecture Review Board (ARB)• Low-level API

- Enables lightest weight interface between software and hardware • Well-proven specification of advanced graphics techniques

- No need to re-invent the wheel• Successfully implemented by many graphics companies

- Low-cost chips and cores, software-only, floating point, fixed point• Leveraged development

- An ecology of development tools already based around OpenGL• Strong future roadmap with OpenGL 2.0

- Enabling access to the new generation of programmable hardware• Through Khronos – will be royalty free to the embedded industry

- Khronos has built in a royalty free framework into its participation agreement - Agreed by SGI – MAJOR ACHIEVENT, MAJOR OPPORTUNITY


OpenGL ES PhilosophyOpenGL ES PhilosophyTo enable rapid consensusTo enable rapid consensus• Be strongly requirements-based

- Handheld 3D gaming and vehicle displays are currently most urgent needs

• Small number of profiles- A large number will be come unwieldy

• Take large-scale profiling decisions – eliminate large functionality blocks- Don’t endlessly tweak individual function calls and parameters

- It often makes little difference to size- Increases ISV confusion to have many detailed differences

• Enable software, fixed point implementations- As well as hardware accelerated implementations

• Define utility libraries to add additional functionality – e.g. display lists- Avoid burdening core profiles with functionality that can be implemented well in software


Some Profile PossibilitiesSome Profile PossibilitiesAddressing broad application requirementsAddressing broad application requirements• Based on 3Dlabs strawman

- And early OpenGL ES ad hoc working group meetings

• Games 3D - Minimum footprint 3D - including texture-mapping- Good gaming platform

• Safety Critical 3D- Absolute minimum 3D for easy certification

• Core 2D- High-quality 2D geometry, Lines, polygons and text

• Core Image- High quality affine transforms

Core 2D

Core 3D

Core Image

Safety Critical 3D


Application ExampleApplication ExampleProposed profiles - solution for Avionics DisplaysProposed profiles - solution for Avionics Displays• Cockpit Display - typical requirements

- Anti-aliased 2D geometric graphics- No windowing system- Minimum possible code size to minimize DO-178 certification costs- SOLUTION – Core 2D

• Enhanced raster requirements- Display of raster maps- Rotation required for “direction of flight up” mode- SOLUTION – Core 2D and Core Image

• Enhanced 3D required- For display of “out of window” 3D terrain- SOLUTION – Safety Critical 3D


Aggressive Timelines for OpenGL ESAggressive Timelines for OpenGL ESEmbedded industry won’t waitEmbedded industry won’t wait• Need ratifiable specs by the end of the year• First ad hoc working group meetings at Siggraph

JulyKhronos Participation agreement available

here at Siggraph

AugustAd hoc Working Group

Meetings

4Q02First draft

specification

1Q03“OpenGL ES 1.0” ratified by Khronos


OpenGL 2.0“Pure” OpenGL 2.0OpenGL 2.0“Pure” OpenGL 2.0

OpenGL ES 2.0OpenGL ES 2.0

How OpenGL Standards Fit How OpenGL Standards Fit TogetherTogether

OpenGL 1.3OpenGL 1.3

OpenGL 1.xOpenGL 1.x

OpenGL 2.x“Pure” OpenGL 2.xOpenGL 2.x“Pure” OpenGL 2.x

OpenGL 2.x includes full 2.0 functionality – so all applications run unmodified

OpenGL 1.3GL2 extensionsOpenGL 1.3GL2 extensions

OpenGL 1.xGL2 extensionsOpenGL 1.xGL2 extensions

Applications can incrementally make use of new GL2 extensions

OpenGL ES 1.0OpenGL ES 1.0

Non-legacy applications can use recommended “Pure OpenGL” features and benefit from a simple, compact, efficient and flexible API


“embedded OpenGL”

Scalable, small footprint2D & 3D graphics

“embedded OpenML”

Small footprint video & audio processing

SYNERGYSYNERGYBetween Between GraphicsGraphics and and MediaMedia Processing ProcessingBetween Between AuthoringAuthoring & & DeploymentDeployment


Participation Participation FrameworkFramework



New Participation FrameworkNew Participation FrameworkKhronos membership is open or interested Khronos membership is open or interested companiescompanies• There are four levels at which a company can participate

- In OpenML and OpenGL® ES development

• Application developers- May use drivers implementing Khronos standards with no license or royalties

• Adopters- May sign a royalty free license to implement Khronos-based specifications

• Contributors- May apply for open membership in the Khronos Group at a nominal annual fee, with full

working group participation and voting rights

• Promoters- Must be voted in by other Promoting Members and act as the “Board of Directors” to drive the

direction of the Khronos Group and standardization activities, with final specification ratification voting rights.


How Khronos Membership Works How Khronos Membership Works

Board of PromotersBoard of PromotersPromoters establish and participate in

working groups for spec generation, marketing, reference implementations,

conformance tests, etc.

Contributor Membership Contributor Membership

Specification work undertaken in Working Group. Generates specs are passed back to the

Promoters for final ratification

Working

Group

Working

Group

Working

Group

Working

Group

Working

Group

Working

Group

Contributors participate in working groups and have full voting

rights within those groups

Ratified Khronos

GroupSpecifications

Ratified Khronos

GroupSpecifications

Application Developers

Application Developers AdoptersAdopters

Adopters may sign a royalty free license to implement Khronos-based specifications

Application developers may use drivers

implementing Khronos standards with no

license or royalties


Promoting Member Companies Promoting Member Companies Dues of $15,000 per yearDues of $15,000 per year• Standards-Definition Benefits

- Vote among Promoters to define objectives, priorities and operations of Khronos- Vote among Promoters on final ratification of specifications - Vote in the working groups

• Marketing Benefits - Company logo on trade show signage, and other marketing material - Quoted in Khronos Group press releases, whenever relevant - Speaking opportunities at major industry events on behalf of the Khronos Group - Opportunity to send staff to Khronos trade show booths and events - Logo and corporate description on the Khronos.org “Promoters” page- Listing the Khronos.org “Products” page


Contributing Member CompaniesContributing Member CompaniesDues of $2,900 per yearDues of $2,900 per year• Standards-Definition Benefits

- Vote in the working groups

• Marketing Benefits - Quoted in Khronos Group press releases, whenever relevant - Opportunity to send staff to Khronos trade show booths and events - Logo and corporate description on the Khronos.org “Contributors” page


Khronos IP FrameworkKhronos IP Framework

Shawn UnderwoodMarketing Director, SGI


Structure of Working GroupsStructure of Working GroupsA working group per taskA working group per task• OpenML 1.1• OpenGL ES 1.0• Implementation working groups• Marketing working group

Working GroupWorking Group




Board of PromotersBoard of Promoters


The brave, new world of IP….The brave, new world of IP….Khronos is taking a new approach to standards & IPKhronos is taking a new approach to standards & IP• Goal of Khronos is royalty-free specification• Contributions to Khronos are royalty-free• Members must actively participate• Avoid hostage situations

- Formal contributions- IP disclosure requirements

• Allow withdrawal to keep IP in-house


Highlights of AgreementHighlights of AgreementWith regards to IPWith regards to IP• Contributors only responsible for tracking activity in their working group• Intellectual property is contributed formally• Working group members must disclose IP that is impacted by

recommendations to the board of Promoters• Withdrawal from working group protects disclosed IP unless formally

contributed• Members may be compelled to contribute IP if it is not disclosed as such


Next StepsNext Steps



Participation TimelineParticipation Timeline

Contributor Participation Agreements Available on

web-site.

1-1 Discussions with Potential Promoters

Board of promoters formed.First Promoters Meeting.Working Groups formed

Working groups meetings start – signed up Promoters and Contributors can participate

Siggraph2002

September6th


Next StepsNext StepsGo to www.khronos.orgGo to www.khronos.org• To become a Contributor

- Download and review Contributor Agreement from www.khronos.org- Execute and send, with dues, to Khronos to become a member

• If you want to be considered as a potential Promoter- Talk to Neil, Shawn or Elizabeth

• Working Groups activities on schedules on the web and mailing lists- We will inform you of first working group meetings

• Any questions – email the Khronos Manager- [email protected]

www.khronos.org


The Khronos Group encourages your participation to help grow the dynamic media market!

SummarySummaryKhronos, OpenML and OpenGL ESKhronos, OpenML and OpenGL ES• Khronos has delivered its first specification - OpenML 1.0

- On schedule from one year ago at NAB 2000

• OpenML is to dynamic media as OpenGL is to 3D graphics- Will reduce costs and increase productivity for anyone creating rich media

• Khronos is now commencing work on OpenML 1.1- Further refinements, performance benchmarks and conformance tests

• Khronos is initiating OpenGL ES initiative- Produce profiled versions of OpenGL for embedded appliances and devices

• Strong synergy between authoring and playback environments- Call for involvement by any interested companies


Any Questions?Any Questions?

IRIX and OpenGL are registered trademarks of Silicon Graphics, Inc., and OpenML is a trademark of Silicon Graphics, Inc., used with permission by the Khronos Special Interest Group. Sun, and Solaris are trademarks or registered trademarks of Sun Microsystems Inc. in

the United States and other countries. Windows is a trademark of Microsoft Corp. Discreet is a trademark of Autodesk, Inc. / Discreet Logic. All other product names, trademarks, and/or company names are used solely for identification and belong to their respective owners.

© Copyright Khronos Group, 2002 - Page 1

Documents

Transcript of © Copyright Khronos Group, 2002 - Page 1